--- id: wrong-and-sure title: Hallucinations level: 1 modes: [day-pack, rucksack] moment: "They act on or repeat a specific detail the AI stated (a name, number, date, quote or source) with no sign it was checked, or say 'but it sounded so sure', or ask 'are you sure?', or find out an answer was wrong." keywords: [hallucination, wrong but sure, confident tone, made-up facts, invented details, are you sure, agrees with me, check the fact, fabricated, sycophancy] source: "product/learning-system/plans/wrong-and-sure.md; research/2026-10-20-concepts/wrong-and-sure.md; research/2026-10-23-essentials/batch-1-facts.md; research/2026-10-23-essentials/batch-4-facts.md; mapping (product/learning-system/MAPPING-1-2.md): FAQ 1.10 (home), 3.19 (2.08 moved to asking-neutrally); MISTAKES 1.5 (home), 2.7 (home), 1.3, 2.8; tips T044 (on the page), T043; strand T" --- ## What it is An AI app can state something false in the same confident voice as something true, because it writes the words most likely to come next, and even when it searches the web first it can still get a detail wrong. That kind of false detail is called a hallucination, for example a wrong notice period for ending a lease. The confident voice tells you nothing about whether a reply is right, so check the fact, not the tone. The app also tends to agree with the person asking, which can make a wrong answer feel confirmed. ## Tips - Before you use a name, number, date or quote from your AI app, look it up outside the chat, because invented details sound as sure as true ones. - Tell your AI app it is allowed to say it does not know, because otherwise it tends to fill a gap with a confident guess. - When in doubt, check elsewhere instead of asking "are you sure?", because your AI app sounds sure either way and may change its answer only because you pushed. ## Questions people ask - Q: Why does AI make mistakes and make things up? A: It writes the words most likely to come next, so a wrong detail comes out as fluently as a right one, even when it has searched the web first. The way apps are trained and scored rewarded a guess over saying "I don't know", and people tend to trust answers that agree with them, so it also tends to agree with you. - Q: If I ask "are you sure?", will that catch a mistake? A: Not reliably, because you may get a yes in the same voice, or a changed answer given only because you pushed. Check the detail somewhere other than the chat. - Q: How do I check an answer fast? A: Pick the one detail a decision hangs on, such as a name, number, date or quote, and look it up somewhere other than the chat, on a page you trust. - Q: Can I ask it to flag when it is unsure? A: Yes, and it helps a little. But a flag is a prompt to look, not proof, and a firm tone or a hedge does not tell you whether the answer is right. ## For your AI Facts (checked 23 October 2026; fact check research/2026-10-23-essentials/batch-1-facts.md; research note research/2026-10-20-concepts/wrong-and-sure.md). - Why it happens: OpenAI calls hallucinations "plausible but false statements"; training and scoring reward a guess over admitting uncertainty, and accuracy "will never reach 100%" (https://openai.com/index/why-language-models-hallucinate/ ; paper 4 September 2025, https://arxiv.org/abs/2509.04664). - Agreeing with the person: five assistants showed it, and people and preference models sometimes preferred convincing flattering answers to correct ones (Anthropic, arXiv 2310.13548, revised 10 May 2025); in Science (26 March 2026), 11 models endorsed users 49% more than people did (https://news.stanford.edu/stories/2026/03/ai-advice-sycophantic-models-research). - "Are you sure?": in the same Anthropic study, asking it cut accuracy by up to 27% and models changed a first answer 32% to 86% of the time, more often right to wrong than wrong to right; FlipFlop (arXiv 2311.08596) found answers flipped about 46% of the time. Both on 2023 models; no newer study found. - Allowing "I don't know": Anthropic's developer guide says it can reduce false information a lot, but not remove it (https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations). - Searching does not remove the check: without search, nothing in the app checks a fact; with search or a research tool on, the app fetches real pages, but the model still writes the sentence that cites them and can misread, mix up or invent a detail (OpenAI: accuracy "will never reach 100%", even with search and reasoning, above). ChatGPT's help says it searches only when that tool is on (https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth, read 23 October 2026). - The apps say so themselves: ChatGPT, "Use ChatGPT as a first draft, not a final source" (https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth); Gemini, "can make mistakes"; some Gemini replies have a sources button; a double-check button is UNCONFIRMED. Teach it like this: check the fact, not the tone. Point at the one detail their decision hangs on and send them outside the chat to check it. A source or link you gave is a lead, not proof: they open it and find the claim there. With no time to check, they leave the detail out or mark it unchecked. Asked "is it probably right?", say plainly that the tone does not tell and no app gives a reliable rate, then name the one detail to check. In a Daypack setup (openrucksack.com/start.md) this is the one check in step 6, the first thing to try, on something they know well. Never say you are always wrong or usually right; when they ask "are you sure?", say plainly that your yes is not a check. The neutral-question habit lives in openrucksack.com/map/asking-neutrally.md.