Renews.
← AI

Study: 44% of LLM Research Claims Expired

Stanford researchers found that nearly half of widely cited claims about large language models have expired due to rapid model development.

AI·1 source
Photo: Monoar_CGI_Artist / Pixabay

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 different models. The study found that 44 percent of these claims have already expired. Among findings with an estimated expiration date, the average lifespan was 15.7 months. After two years, 39 percent of the findings had expired.

GPT-3.5 originally fabricated 55 percent of its scientific references in a 2023 study, while GPT-6.1 Sol fabricated 0 percent in the same test. Claude Sonnet 5.5 came in at 1 percent for fabricating references. However, Qwen3.7-Flash, a 2026 model, still fabricated 69 percent of its references. In reasoning tasks, Claude Sonnet 5.5 fails in about 10 percent of two-hop questions where it correctly answers both individual questions, compared to ChatGPT failing 43 percent in the original paper.

Andrej Karpathy, an OpenAI co-founder who has since joined Anthropic, recently said that currently available models are just a preview of future capabilities.

Sources

Every sentence of this story was checked against facts quoted from these articles. How we make the news

Spotted an error? Tell us

Sign in

Keep the stories you like in your favorites.

Forgot your password?

No account?