Often we discuss AI, and I often think: these are expired debates. I worry that we miss how fast this technology develops. We are talking about a technology that develops extremely fast. We have to be careful not to talk about old claims that have long stopped being true, and so miss what the technology can really do.
A study from the Stanford researchers Owen Queen and James Zou gives us some interesting findings for this debate. They retested 50 findings about large language models on 20 models released between July 2024 and September 2026. 44% of the findings have expired in current models, 34% have weakened, 16% persist and 6% got stronger. For the expired findings, the time from publication to expiration was about 16 months on average.
One interesting finding tackles a common thesis that AI hallucinates scientific references. Back in 2023, ChatGPT with GPT-3.5 invented 55% of the works it cited in short literature reviews. Queen and Zou sent the same prompts to the models of 2026: Claude Sonnet 5.5 invented 1% of its references, GPT-6.1 Sol none. Other famous failures went the same way: counting the letters in “strawberry” or facts lost in the middle of a long document.
Not everything has expired. Models still answer when the documents they were given hold no answer. And the picture differs by model. It is not AI that develops. Different models develop differently. Qwen3.7-Flash, released in July 2026, still invented 69% of its references. Among the models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol.
Before we build an argument or a rule on a failure of AI, we should check whether that failure still exists in the models we are talking about. This is not an easy task. We are talking about an emotional debate and a complicated technology that changes every few months, and it changes differently from model to model. What was true for a model in 2023 can be false for one model and still true for another in 2026. A simple first step could be to say which model and which date a claim is about. But the example shows us that we should approach the debate about AI carefully.