Only 4 in 10 statistics quoted by AI assistants trace back to a matching study
Trakkr, an AI search tracking company, asked ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews 200 questions that call for a number, then followed 1,285 of the cited statistics back through up to five pages. It reports that 40% reached a study saying the same thing, 8% reached a study where the number or its context had changed, 40% ended at a source it could not check, and 11% led nowhere. The cited page usually did contain the figure (82% of the time), but only 21% of cited pages were the original research; more than half were company blogs. ChatGPT did best at 57% sourced; the other four ranged from 38% to 46%. This is a single-day test through each company's API, judged by an AI model and corrected with 150 hand checks.
On Hacker News (60 comments, 20 top-level), 3 top-level comments dismiss the post as an advert or AI-written, 4 point out that people repeat made-up statistics too, and 3 say source tracing is exactly what AI should be doing. No consensus.
Why it matters: If you put AI-supplied numbers into proposals, decks or client reports, click through to the original study and check the year, sample and who was asked before you use it. Sorted take: treat any AI statistic as a lead, not a source, even when every assistant agrees.
Trakkr Research ↗Discussion on Hacker News ↗AI-assisted, reviewed by Sorted AI
▲ 0
0 comments