How to check whether AI research is making things up

A fluent report with a bibliography can still be wrong sentence by sentence. The check is the passage behind the claim, not a link at the end.

SpanSearch Team

AI research reports now look the part. The structure is tidy, the tone is sure, and there is a bibliography at the end. After you read one, it is hard to tell whether the work was actually checked, or whether the model joined a few impressions into a sentence.

Most checking in the market stops at "did it include a link."

Open it and you get an article, or a page title. The report never marks which passage the sentence depends on. You scroll, search, and guess whether this is the spot. A report with a dozen claims means reading a dozen pages again. When people are busy, they look for whether the link is there, and treat "sources are listed" as "this sentence is supported."

A worse case: the link is real, the page opens, and the page does not say what the report says. The model joined two sources, or paired a real title with a conclusion it filled in. You will not catch that kind of invention from the bibliography alone.

So the check should not start at the bibliography. It is one claim, and the passage the claim says it rests on. If they match, the sentence can stay. If they do not, that sentence is made up, however fluent the report is.

What we did

A report starts making things up when the writing begins too early. Links and snippets arrive, and a paragraph gets written from the impression. The page was never kept as complete material, and the arguments and the evidence were never gathered first.

We put the writing last. From the data that was collected, we first select the complete material that is relevant to this research. From that material we gather the evidence and the arguments: which passage says what, and which claim comes from those passages. The article is written from those arguments and that evidence.

A claim uses the passage that was gathered for it. Finished reports are in the library, under documents. Open the source next to a claim, and see whether the passage it points to is the same thing as the sentence. If it does not match, change the sentence or delete it. The source stays on that sentence, so after the edit you can check it against the same passage again.

A misread, or two passages joined into one sentence, can still happen. The difference is that the sentence is written from evidence already gathered, and when it does not match you can go back to that passage and change it.