Article · 16 min read

Why does AI make things up?

Language models can be wrong in exactly the same confident tone as when they are right, and their made-up answers have ended up in courts, government reports and the news. Research has begun to understand why it happens, and what helps.

Short answer

Because a language model is trained to write plausible text, not to tell what it knows from what it guesses. Facts that rarely appear in its training data cannot be learned reliably, and most tests award points for a guess but none for “I don’t know”, so models learn to guess. Inside the model there is a brake on answering about unknown things, but it can let go when something seems familiar. Search and new training reduce the errors but do not remove them, so anything important needs checking.

Confident, fluent and wrong

In the spring of 2023, two lawyers in New York filed a brief with a federal court in a case against the airline Avianca. It cited earlier rulings, among them Varghese v. China Southern Airlines. Six of them did not exist. They had been written by ChatGPT, which one of the lawyers had used to look for cases. When he asked the chatbot whether the rulings were real, it said they were and could be found in the legal databases. The judge called the invented ruling’s legal analysis gibberish, and in June 2023 ordered the lawyers and their firm to pay 5,000 dollars.

It has happened again and again since. A database run by Damien Charlotin collects court decisions in which a party relied on material invented by AI. In May 2025 it held 87 cases. In November 2026 it holds 2,292, from courts all over the world. In January 2026 came a Swedish example: the district court in Eskilstuna cut the fee of a lawyer whose submissions cited four rulings from Sweden’s Supreme Court that do not exist. The court called the submissions “AI slop”. The Stockholm District Court writes in its annual report for 2025 that it has seen a marked increase in summons applications written with AI, and that they sometimes cite laws and cases that do not exist.

It does not stop at the courts. In 2024 Air Canada was ordered to compensate a passenger whom its chatbot had given wrong information about bereavement fares. The airline in effect argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal called this a remarkable submission and pointed out that the company is responsible for all the information on its website, whether it comes from an ordinary page or a chatbot. In 2025 the consulting giant Deloitte agreed to repay part of its fee for a report to the Australian government, after it emerged that the report contained invented sources and an invented quote from a court judgment. One of the invented reports was attributed to a researcher at Lund University in Sweden. In April 2026 South Africa’s government withdrew its draft national AI policy, because its reference list contained sources that did not exist. And research is affected too. When researchers in 2026 checked 2.5 million medical papers, they found 4,046 invented references in 2,810 papers, according to a report in The Lancet. The number had grown more than twelvefold since 2023.

The phenomenon is called hallucination. Cambridge Dictionary named “hallucinate” its word of the year for 2023, with a new sense: when an AI hallucinates, it produces false information. The word is still disputed, since it makes it sound as if the model sees something that is not there. Three researchers at the University of Glasgow have proposed calling it bullshit in the philosophical sense instead: text that is indifferent to whether it is true. Other researchers speak of confabulation. We use hallucination, because it is the word most people use.

Why does it happen? And why does an invented answer sound just as sure as a right one? In recent years research has produced three answers that fit together. They are about how a language model produces text, what it can learn at all, and what it has been rewarded for.

A machine that always answers

A language model has no database of facts to look things up in. It has been trained on vast amounts of text to predict the next word, and when it answers, it writes one token at a time, roughly a word or part of a word. We explain how this works in How does a language model work? What comes out is always fluent text, whether the model has learned the answer or is just guessing. Sweden’s Agency for Digital Government writes in its guidelines for the public sector that generative AI, unlike a person who can admit not knowing, tends to always give an answer, even when it is uncertain.

Researchers at OpenAI and Georgia Tech have shown this with an example. They asked the language model DeepSeek-V3 for the birthday of one of them, Adam Tauman Kalai, and told it to answer only if it knew. In three attempts they got three different dates, all of them wrong. When they asked three popular chatbots for the title of Kalai’s doctoral dissertation, they got three different titles, with three different universities and years. None was right.

Figure 1The same sure tone

This figure is interactive and needs JavaScript.

Choose a question and press Ask again. Then show what the model knows. The probabilities are made up to show the principle, but the dates and the letter-counting answers come from the researchers’ tests of real models.

So the uncertainty is there, but it does not show in the text. When the probability is spread out, the model still picks an answer and writes it as fluently as when it knows. The clearest example comes from a base model, one that has only been trained to continue text and not yet to be an assistant. When researchers at Anthropic ran their model Claude 3.5 Haiku that way, it continued the sentence “Michael Batkin plays the sport of” with “pickleball”. It had no idea who Michael Batkin was, but a sentence like that usually ends with the name of a sport.

The errors are not only about facts. Asked how many Ds there are in the word DEEPSEEK, DeepSeek-V3 answered either 2 or 3 in ten attempts. Meta AI and Claude 3.7 Sonnet answered similarly, with answers as large as 6 and 7. The right answer is one. The model does not read letters but tokens, and the word is split into chunks such as D/EEP/SEE/K. A model that was allowed to reason in several steps, DeepSeek-R1, instead spelled its way through the word and counted correctly.

What was seen only once

Why do models make up birthdays and book titles, yet hardly ever misspell a word? Kalai and his colleagues show that it comes down to what can be learned from text. Grammar and spelling follow patterns that repeat millions of times, and models learn them almost flawlessly. A birthday follows no pattern. It cannot be worked out, only remembered, and it can only be remembered if it was in the text the model was trained on. That is why the errors arise even if the training data itself is completely free of errors.

The researchers derive a lower bound on how often a base model must get such questions wrong. It depends on the share of facts that appear exactly once in the training data. If 20 percent of all birthdays are mentioned only once, a base model can be expected to make up at least 20 percent of the birthdays.

The idea builds on a trick from the Second World War. It emerged when Alan Turing and I. J. Good worked on breaking German ciphers at Bletchley Park: the share of a sample that has turned up only once reveals roughly how much of the whole you have not yet seen. Good published the method in 1953. He is the same Good who described the idea of an intelligence explosion, which we write about in Can AI improve itself?

Figure 2What was seen only once

This figure is interactive and needs JavaScript.

A real simulation in your browser. The people and the text are made up, and the model is generous: it remembers everything it has seen at least once and guesses the rest. Drag the slider and compare the two numbers, then ask the model questions.

The two numbers track each other, however much text the model gets. In the figure everyone is written about in the end. In reality the tail of rare facts never ends: there are always people, places and events that have been written about only once, or never. More data reduces the errors, but does not get rid of them.

The exam that rewards guessing

The statistics explain why a base model guesses. But chatbots are trained further, with methods such as RLHF, to become helpful and honest. Why does the guessing not go away? Kalai and his colleagues point to how models are graded. Most of the tests used to compare AI models, known as benchmarks, give one point for a right answer and zero for a wrong one. But “I don’t know” also scores zero. So guessing always pays.

It is the same logic as a multiple-choice exam with no negative marking: whoever guesses on everything scores more on average than whoever leaves blanks. The researchers looked at ten of the most widely used tests. Nine gave nothing at all for abstaining, and the tenth gave a little. Language models are optimised to be good test-takers, as the researchers put it.

Their proposal is simple: state in every question how wrong answers are penalised, for example “answer only if you are more than 75 percent sure, since a wrong answer costs 3 points”. Then answering only pays when the chance of being right is above the threshold. Try it yourself.

Figure 3The exam that rewards guessing

This figure is interactive and needs JavaScript.

Top: what guessing scores on average, with different penalties for wrong answers. Bottom: real results from Kalai and colleagues’ study in Nature, 2026, rescored with the penalty you choose.

The difference shows in real results. In the study, published in Nature in April 2026, two OpenAI models answered 4,326 short factual questions in the SimpleQA test. The model o4-mini answered almost everything and was wrong on 76.8 percent of the questions. The model gpt-5-mini abstained on 63.2 percent and was wrong on 20.8 percent. Counting only right answers, o4-mini still won, by 20.6 to 16.0 percent.

The same pattern appears as models become more powerful. When OpenAI released its reasoning model o3 in the spring of 2025, the company’s own test showed that it made up answers about people twice as often as its predecessor o1, on 33 against 16 percent of the questions. OpenAI explained that o3 makes more claims overall, and so both more correct and more incorrect ones.

The further training can also make models worse at knowing what they know. In OpenAI’s 2023 technical report on GPT-4, the base model’s confidence matched closely how often it was right. In technical terms, it was well calibrated. After the further training its calibration was considerably worse, and the measure of the error that the report uses was about ten times larger.

A brake that can misfire

What happens inside a model when it makes things up? Researchers at Anthropic have studied this in Claude 3.5 Haiku, using methods from interpretability research that make it possible to follow which concepts in the model become active as it answers. They found something unexpected. The model’s default is not to answer. A group of features that roughly mean “can’t answer” is active from the start, and it is switched off only when the model recognises what the question is about, such as a well-known name.

Hallucinations can arise when the brake lets go when it should not. According to the researchers, this happens when a name feels familiar but the model knows nothing more about the person. They could also cause the error themselves by boosting the features for known names.

Figure 4Does the model recognise the name?

This figure is interactive and needs JavaScript.

A simplified picture of what Anthropic’s researchers found in Claude 3.5 Haiku. The questions and answers are the model’s own. Choose an example and see which parts are active.

The findings concern a single, smaller model, and the researchers write that they capture only part of the story. But they fit with what others have seen from the outside. In the HalluHard test from February 2026, models mostly declined when asked about entirely made-up things, but made things up far more often about real but rarely covered ones, such as seldom-cited research papers and artworks in local galleries. So the danger is not the completely unknown but the half-familiar.

Can it be fixed?

In theory there are different answers. Researchers have proved that a language model used to solve any problem whatsoever will inevitably hallucinate sometimes, since no such model can learn everything. Kalai and his colleagues, on the other hand, write that hallucinations are inevitable only for base models. A model that is allowed, and dares, to say “I don’t know” does not have to guess.

In practice, several methods reduce the errors, but none removes them. The most common is to let the model search sources before it answers, known as RAG. When researchers at Stanford in 2024 asked ordinary chatbots questions about US court rulings, they made things up in between 58 and 88 percent of cases. Legal AI tools that search databases of case law did so less often, but still in between 17 and 33 percent of cases. In HalluHard, the best setup, Claude Opus 4.5 with web search, hallucinated in about 30 percent of cases. Without search it was about 60 percent.

Search has weaknesses of its own. When the Tow Center at Columbia University asked eight AI search engines in 2025 to find which news article a quote came from, they got more than 60 percent of the queries wrong, mostly without hesitating. ChatGPT’s search named the wrong article in 134 cases out of 200, expressed uncertainty only 15 times and never declined to answer.

The same goes for news. In the autumn of 2025, 22 public service broadcasters in 18 countries, among them Sweden’s SVT, published the largest study so far of how AI assistants present the news. Journalists reviewed more than 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity. 45 percent contained at least one significant issue, and one in five had major accuracy problems, including invented details and outdated information. When Finland’s Yle tested in Swedish in May 2025, ChatGPT answered that Pope Francis led the Catholic Church, although he had died the month before.

There are signs of improvement. When the analysis firm Artificial Analysis launched its AA-Omniscience test in November 2025, in which a wrong answer costs as much as a right one earns, only three of 36 models had more right answers than wrong ones. A model that never answered would have come fourth. In the test’s new version, in November 2026, some models rarely guess. Of the questions that Google’s Gemini 4 Carbon and Gemini 4 Argon could not answer correctly, they answered wrongly on only 4 percent. Mostly they declined instead. OpenAI says its GPT-6 Sol, from September 2026, makes about half as many mistakes as its predecessor, in a test built from conversations in which users reported errors.

But progress does not only go one way. Anthropic’s Claude Opus 5, from July 2026, did better than its predecessor on AA-Omniscience, but also answered more often when it was unsure, and its hallucination rate rose by 14 percentage points to 50 percent. More right answers are often bought with more guesses.

One way to catch guesses relies on the fact that a model that guesses often answers differently each time. Researchers at Oxford showed in Nature in 2024 that you can ask the same question several times, sort the answers by what they mean and measure how much they scatter. The measure is called semantic entropy, and it caught made-up answers better than the other methods the researchers compared it with. It needs no answer key. But it has a limitation that the figure shows.

Figure 5Ask ten times

This figure is interactive and needs JavaScript.

The answers are examples we wrote to show the method. The sotorasib question comes from the study, where a model sometimes answered right and sometimes wrong, but the split here is made up. The spread is measured as entropy in bits.

So the method does not catch errors the model makes consistently, such as myths it has learned, and the researchers themselves note that it does not help when a model is systematically wrong. Kalai and his colleagues tested a simpler variant in their Nature study: let the model answer twice and abstain if the answers differ. That reduced the errors, but also the number of right answers. In an ordinary test, where only right answers count, the model therefore looks worse. It is the same problem as in figure 3.

Models can also learn wrong things from what they read. In the spring of 2026 the AI researcher Almira Osmanovic Thunström, at Chalmers and the University of Gothenburg, showed how an invented disease from an obviously fake study spread within two months to major AI chatbots, which presented it as credible. A model like that answers just as surely every time, and then no scatter raises the alarm.

How to protect yourself

Hallucinations cannot be removed completely today, but they can often be caught. The research suggests a few rules of thumb:

  • Be most wary of the rare. Names, dates, figures, quotes, court cases and references are what is most often made up, especially when they concern things that are rarely written about.
  • Check the source, not just that it exists. Open the link or look up the ruling. A reference can lead to something real that still does not say what the answer claims.
  • Ask again. Ask the same question in a new conversation. If the answers differ in substance, that is a warning sign.
  • Allow it to say it does not know. Write that “I don’t know” is a good answer if the model is unsure. That helps, but a model can also be sure and wrong.
  • Let it search, but read for yourself. Answers with search are more often right, but they too can cite wrongly or point to pages that do not exist.
  • The more important the decision, the more checking. In law, healthcare and finance, a text from AI is a draft, not an answer key.

Our view

This section is SI-bolaget’s opinion. The rest of the article reports the research. Here we say what we ourselves believe.

AI making things up is not just a technical problem. It is a question of trust. The more we let AI write documents, summarise research and answer questions, the more it matters that we know when it knows, and when it is guessing.

In one way, the research in this article is hopeful. Hallucinations are not a mystery. They have causes that can be understood, and partly fixed. So we think:

  • Humility. An AI that says “I don’t know” is not worse, it is more honest. We would rather have models that sometimes abstain than models that always answer.
  • Openness. AI companies should report how often their models answer wrongly and how often they abstain, not just how often they answer correctly. Leaderboards that count only right answers reward guessing.
  • Humans at the centre. Responsibility for what is filed in a court, published or used as the basis for a decision should rest with a person who has checked it.

The more powerful AI becomes, the more important it is that it knows where its knowledge ends, and says so.

Sources

  1. United States District Court, Southern District of New York, Mata v. Avianca, Inc., Opinion and Order on Sanctions, 22 June 2023.
  2. Damien Charlotin, AI Hallucination Cases Database, updated 12 November 2026, and the count in May 2025 according to The Volokh Conspiracy.
  3. The Eskilstuna district court judgment according to Realtid, 13 January 2026, and Dagens PS, 9 January 2026 (in Swedish).
  4. Stockholm District Court, Verksamhetsberättelse 2025 (annual report, in Swedish), 2026.
  5. Civil Resolution Tribunal, British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024.
  6. Deloitte’s report for Australia’s Department of Employment and Workplace Relations according to The Register, 6 October 2025.
  7. South African Government News Agency, Minister announces withdrawal of draft AI Policy, 26 April 2026.
  8. Topaz et al., Fabricated citations: an audit across 2·5 million biomedical papers, The Lancet 407, 2026, according to the University of Eastern Finland.
  9. University of Cambridge, Cambridge Dictionary names ‘hallucinate’ Word of the Year 2023, November 2023.
  10. Hicks, Humphries and Slater, ChatGPT is bullshit, Ethics and Information Technology 26, 2024.
  11. Swedish Agency for Digital Government (Digg), Riktlinjer för generativ AI (guidelines for generative AI, in Swedish), 2025.
  12. Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate, OpenAI and Georgia Tech, 2025.
  13. Kalai, Nachum, Vempala and Zhang, Evaluating large language models for accuracy incentivizes hallucinations, Nature 653, 2026.
  14. Adam Tauman Kalai and Santosh Vempala, Calibrated Language Models Must Hallucinate, STOC 2024.
  15. William A. Gale and Geoffrey Sampson, Good-Turing Frequency Estimation Without Tears, Journal of Quantitative Linguistics 2, 1995 (on Turing and Good’s work at Bletchley Park), and I. J. Good, The population frequencies of species and the estimation of population parameters, Biometrika 40, 1953.
  16. OpenAI, GPT-4 Technical Report, 2023.
  17. OpenAI’s system card for o3 and o4-mini, April 2025, according to Simon Willison and TechCrunch.
  18. Lindsey et al., On the Biology of a Large Language Model, Anthropic, 2025, and Tracing the thoughts of a large language model, March 2025.
  19. Fan, Delsad, Flammarion and Andriushchenko, HalluHard: A Hard Multi-Turn Hallucination Benchmark, 2026, and the summary in The Decoder, February 2026.
  20. Xu, Jain and Kankanhalli, Hallucination is Inevitable: An Innate Limitation of Large Language Models, 2024.
  21. Dahl, Magesh, Suzgun and Ho, Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, Journal of Legal Analysis 16, 2024.
  22. Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Stanford, 2024.
  23. Klaudia Jaźwińska and Aisvarya Chandrasekar, AI Search Has a Citation Problem, Tow Center, Columbia Journalism Review, March 2025.
  24. EBU and BBC, News Integrity in AI Assistants, October 2025, and the summary from the BBC; SVT’s part according to Sveriges Radio and Yle’s test in Swedish according to Svenska Yle.
  25. Jackson et al., AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models, Artificial Analysis, 2025, and the results for version 1.1, read 14 November 2026.
  26. GPT-6 Sol according to TechCrunch, 22 September 2026, and Claude Opus 5 according to The Decoder, 25 July 2026.
  27. Farquhar, Kossen, Kuhn and Gal, Detecting hallucinations in large language models using semantic entropy, Nature 630, 2024.
  28. Sveriges Radio, AI spred falsk sjukdom: experimentet avbröts av forskare, P1 Morgon, 23 April 2026 (in Swedish).
  29. NASA, China’s Wall Less Great in View from Space, 2005.

Terms in this article

How we work: the articles are written with the help of AI, and every fact is checked against the sources listed above. Found a mistake? Write to hej@sibolaget.se.

More questions?

New to AI? Start with What is AI? The glossary explains the terms, and the front page has straight answers to the most common questions. Missing something? Write to us.