Hallucinations

The model is the part of an AI app that writes replies. It writes the words most likely to come next, and it does not check a fact against a source.

A detail is one fact, such as a name, date, number or quote. A hallucination is a wrong detail, such as a wrong notice period for ending a lease. The model states a wrong detail in the same tone as a right one, and tone is how sure it sounds. A sure tone shows how a reply was written, not its truth. The app may fetch web pages, but the model can still misread or mix up a detail.

The way models are built and tested rewards a guess over "I don't know", so a model tends to guess rather than leave a gap. Look the detail up outside the chat: anywhere but your chat with the app.

• Look up a name, number, date or quote from your AI app outside the chat before you use it, because wrong details sound as sure as right ones.
• Tell your AI app it is allowed to say "I don't know", because without that permission the model tends to fill a gap with a sure-sounding guess.
• When in doubt, look a detail up outside the chat instead of asking "Are you sure?", because the answer sounds sure either way and may change only because you pushed.

1. Pick one real job this week that rests on a name, number, date or quote, such as how much notice a tenant gives, when a club's fees fall due, or who said a line for a speech.
2. Ask your AI app for help with the job. Note the one detail the job depends on, and how sure the reply sounds.
3. Look that detail up outside the chat, such as on an official page, in your contract or on a club's own site. Note where you found it, so you can show where it came from.
4. Tell the chat what you found, and ask it to redo the reply with the right detail.
5. Use the right detail in your job, and keep a note of where you found it.

If no job this week rests on a detail, ask about something you can already look up, such as the year a local club began, and use that year as the detail to look up and keep.

What to expect:

• The tone reads the same whether the detail is right or wrong, so only the look-up tells you.
• If you sent "Are you sure?" before step 3, you may get a yes or a changed answer in the same tone, and neither checks anything.
• If the detail matches, the look-up was still worth doing, because it is the habit this page teaches.
• If the app says it cannot be sure, that is better than a sure guess, and the detail is still worth looking up.

• The model can fill a gap with a name, number or quote that looks right, even when the app looked at web pages first. Look the exact detail up outside the chat, and if you cannot find it, do not use it.
• Asking "Are you sure?" can get a yes in the same tone, or an answer the model changed only because you pushed, and neither is a check. Look the detail up outside the chat, then take what you found back to the chat.
• A sure tone does not show an answer is right. A flag is a line in a reply that says the answer may be wrong. A flag is a reason to look the detail up outside the chat, not proof the reply is right.

A sure tone is not proof. Looking up the one detail a job depends on, outside the chat, tells you whether to use it and lets you show where it came from.

• How AI works
• Context: what your AI sees
• Custom instructions

• wrong-and-sure-1 (Checking, waypoint 1): I can check the one detail a reply depends on somewhere other than the chat. Proves: for one real reply of theirs (the Try it, setup step 6, or any reply in passing), they name the detail that mattered, look it up outside the chat, and say what they found (matched, differed, could not be found) and what they did next.
• wrong-and-sure-2 (Checking, waypoint 1): I can say why an AI reply can be wrong and still sound sure, in my own words. Proves: they explain it to their AI or another person, saying that the model writes the likeliest words and does not look the fact up, so a sure tone tells you nothing about whether the reply is true, and give one reason it happens (a guess was rewarded).

Before they quote, send or act on a name, number, date or quote from their AI, they ask "where would I check that?" and do it. They stop treating "Are you sure?" or the AI's own confidence as a check, and they take what they found back to the chat.

• Explains: the model writes the words most likely to come next and does not look the fact up, so a wrong detail arrives in the same tone as a right one and a sure tone is no evidence; a guess was rewarded over "I don't know", so the model tends to guess; names, numbers, quotes and sources fail most. Needs the first point and one of the others.
• Ask: "A reply holds a date, a quote, a number and an opinion, and you can look one up before you act. Which first?" Shows it: the one the decision depends on, usually the quote or the number, looked up somewhere other than the chat.
• Ask: "A reply gives a case name, 'Hartley v. Brown (2018)', in a firm tone, and you were about to quote it. You ask 'Are you sure it exists?' and it says 'Yes, completely.' What do you do next?" Shows it: that yes told them nothing; they search the exact name outside the chat, in a source they trust, and do not use it if it cannot be found.

Hints for the person's AI. Write the words for the person, in the way their journal says they like to learn; never read these lines to them.

• Experiment: the Try it, on a real job, in under five minutes. For a second go, offered as a game and never as the Try it, the person makes up a plausible book title and author and asks for a summary. If the app summarises it, that is the lesson; if it says it cannot find the book, that is better behaviour.
• Challenge: for a week, the person looks up outside the chat any name, number, date or quote from their AI app before using it, and counts how often it was wrong.
• Quiz (only if their journal allows quizzes), two questions: one, "A reply cites 'Hartley v. Brown (2018)' in a firm tone; you ask 'Are you sure?' and it says 'Yes, completely.' What do you do next?"; two, "A short reply holds a date, a quote, a number and an opinion; you can look one up before you act. Which first?"
• Cards: five, covering same tone whether right or wrong, the way models are built and tested rewards a guess over "I don't know", details fail most, the app may say it does not know, and "Are you sure?" is not a check.
• Explain: use a time their AI app was wrong, or their Try it result, as the example; two sentences on why, then one on what to do.
• Game, "two truths and a fib": the AI writes three sentences about a topic the person knows well, two right and one wrong, all in one even tone; the person finds the wrong one; then the AI says it wrote all three the same way. It is labelled as a game, and the AI never plants a wrong detail outside it.

• Say the model "writes" and "does not look the fact up"; never "lies", "knows" or "glitch". OpenAI calls the cause "not mysterious": training and scoring reward guessing (checked 23 October 2026).
• Lead with the one detail and where to check it. The overcorrections are "never trust it" or "check every line" (how much to check belongs to a later page, check-by-what-is-at-stake) and "a hedge means it is right, a firm tone means it is wrong". A hedge or a flag is a prompt to look, not a measure: hedging helped people only a little and was poorly calibrated (research note section 4, checked 20 October 2026).
• Never say a newer, paid or search-enabled model has fixed it: better, not fixed. An app that searched the web first can still get a detail wrong, and ChatGPT's help says it searches only when that tool is on (read 23 October 2026). Never quote a percentage to the person; the figures are dated and live under For your AI.
• "A link proves it" and how to read a citation belong to checking-a-source; here say only that the first move is to look the detail up outside the chat. A source or link you gave is a lead, not proof: they open it and find the claim there.
• "Are you sure?" cannot be answered by the AI itself: say plainly that your yes is not a check, and point them to somewhere else. The habit of asking neutral questions is asking-neutrally's; here agreement is only one cause of a wrong answer.
• Allowing "I don't know" helps but does not fix: Anthropic says it can reduce false information a lot, not remove it, and that is a vendor claim with no independent chat study (checked 23 October 2026). Do not promise a result.
• Apps differ in whether they mark which claims came from a search or a source, and in what they call it. Gemini's "Double-check response" button is UNCONFIRMED (checked 20 and 23 October 2026); never tell a person to press it. Teach the habit, not a screen.
• Daypack, each risk: (1) a made-up detail: setup step 6, the first thing to try, is this check on a task they know well, done once, nothing recorded; (2) "Are you sure?": the AI says it has saved a note to the app's memory and, asked, says yes, so the person opens the memory entry and reads it (see app-memory); (3) a flag: the AI says "I am not sure; check the page" when it is unsure, and never states an app setting from memory as fact (see how-a-chat-starts).
• A Daypack keeps no progress and records nothing: the AI runs the Try it or the Fast track for the person's benefit and says "that is this one, you have it". Proofs and the "proven" line are Rucksack-only. The Rucksack is coming soon: say it as a general case, never as something they can set up today.
• Do not name a court case, a firm or a figure on the human page: they date, and the Mata v. Avianca detail that ChatGPT said the cases were real when asked is UNCONFIRMED at the court record. The invented case name in the Fast track ("Hartley v. Brown") is made up on purpose. (checked 27 October 2026)

• Q: Why does AI make mistakes and make things up? A: It writes the words most likely to come next, so a wrong detail comes out as fluently as a right one, even when it has searched the web first. The way apps are trained and scored rewarded a guess over saying "I don't know", and people tend to trust answers that agree with them, so it also tends to agree with you.
• Q: If I ask "are you sure?", will that catch a mistake? A: Not reliably, because you may get a yes in the same voice, or a changed answer given only because you pushed. Check the detail somewhere other than the chat.
• Q: How do I check an answer fast? A: Pick the one detail a decision hangs on, such as a name, number, date or quote, and look it up somewhere other than the chat, on a page you trust.
• Q: Can I ask it to flag when it is unsure? A: Yes, and it helps a little. But a flag is a prompt to look, not proof, and a firm tone or a hedge does not tell you whether the answer is right.

Facts (checked 27 October 2026 against the essentials page's facts of 23 October 2026; research notes research/2026-10-20-concepts/wrong-and-sure.md and research/2026-10-23-essentials/batch-1-facts.md, batch-4-facts.md).

• Why it happens: OpenAI defines hallucinations as "plausible but false statements"; training and scoring "reward guessing over acknowledging uncertainty"; one-off facts such as a birthday cannot be predicted from patterns, so they fail most. In OpenAI's test an older model was wrong 75 percent of the time and almost never abstained (1 percent); a newer one abstained 52 percent and was wrong 26. OpenAI says accuracy "will never reach 100%", even with search and reasoning (paper submitted 4 September 2025; checked 23 October 2026; https://openai.com/index/why-language-models-hallucinate/ ; https://arxiv.org/abs/2509.04664).
• Agreement, a second cause: all five assistants in Anthropic's study showed it, and people and preference models sometimes preferred a convincing flattering answer to a correct one (arXiv 20 October 2023, revised 10 May 2025; checked 23 October 2026; https://arxiv.org/abs/2310.13548). In Science, 26 March 2026, 11 models endorsed users 49 percent more than people did (checked 23 October 2026; https://news.stanford.edu/stories/2026/03/ai-advice-sycophantic-models-research).
• "Are you sure?": in that Anthropic study asking it cut accuracy by up to 27 percent and models changed a first answer 32 to 86 percent of the time, more often right to wrong than wrong to right. FlipFlop (arXiv 14 November 2023) found ten models flipped about 46 percent of the time. Both use 2023-era models and no 2026 study was found (checked 23 October 2026; https://arxiv.org/pdf/2310.13548 ; https://arxiv.org/abs/2311.08596). Models also do not reliably fix their own reasoning without outside evidence (https://arxiv.org/abs/2310.01798, read 20 October 2026).
• Allowing "I don't know": Anthropic's developer page says it "can drastically reduce false information" but techniques "don't eliminate them entirely"; undated, for prompts over supplied documents, no independent chat study (checked 23 October 2026; https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations). The tip was judged blind on Claude Sonnet 5.5 and ChatGPT gpt-6.1-sol, a small margin on both (23 October 2026).
• Searching does not remove the check: without search nothing in the app checks a fact; with search on, the app fetches real pages but the model still writes the sentence that cites them and can misread, mix up or invent a detail. Tow Center, 6 March 2025: eight AI search tools were wrong on over 60 percent of 1,600 citation queries and were "generally bad at declining"; no newer study found (checked 23 October 2026; https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php). ChatGPT's help says it searches only when that tool is on (read 23 October 2026; https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth).
• What the apps say (checked 20 October 2026, except ChatGPT, read 23 October): ChatGPT, "Use ChatGPT as a first draft, not a final source"; Claude can write "things that might look correct but are very mistaken" (16 March 2026, https://support.claude.com/en/articles/8525154); Gemini "can hallucinate and present inaccurate information as factual" (https://support.google.com/gemini/answer/16279220), with a sources button on some replies only (https://support.google.com/gemini/answer/14143489); Copilot replies "that sound convincing but are incomplete, inaccurate, or inappropriate" (older app only, https://support.microsoft.com/en-us/microsoft-copilot/privacy-faq-for-microsoft-copilot). Whether Gemini still has a "Double-check response" button is UNCONFIRMED.
• Cases for one-line examples in conversation, never on the human page: Deloitte Australia repaid the final instalment of an AU$440,000 government report that had a fabricated judge quote (AP, 7 October 2025, read 20 October 2026); Damien Charlotin's court database held 2,202 cases on 19 October 2026 (https://www.damiencharlotin.com/hallucinations/).
• Neutral questions and agreeing with the person are asking-neutrally's ground; point to openrucksack.com/map/asking-neutrally.md. Re-check: January 2027, or sooner if an app marks which claims were checked or a vendor changes how its model says "I don't know".

Teach it like this: check the detail, not the tone. Point at the one detail their decision depends on and send them outside the chat to check it. With no time to check, they leave the detail out or mark it unchecked. Asked "is it probably right?", say plainly that the tone does not tell and no app gives a reliable rate, then name the one detail to check. In a Daypack setup (openrucksack.com/start.md) this is the one check in step 6, the first thing to try. Never say you are always wrong or usually right.

An example of this taught well, in passing, after the person quotes a figure from a reply: "Quick thing worth a minute: that number came in the same confident voice as everything else, and a number is just the sort of detail a model gets wrong. Want to look it up somewhere other than this chat before it goes in the email?" Then, on a yes, one sentence on why, and back to the email.