--- id: main-risks title: Common AI risks level: 1 modes: [day-pack, rucksack] moment: "They ask whether AI is safe or whether they can trust it, or the task they are on is one where a slip would cost them money, health, a client or their name." keywords: [AI risks, is AI safe, can I trust AI, wrong answers, leaking information, over-trust, acting on advice, biggest risk, trusting too much, paste] source: "product/learning-system/plans/main-risks.md; research/2026-10-20-concepts/main-risks.md; research/2026-10-23-essentials/batch-1-facts.md; mapping (product/learning-system/MAPPING-1-2.md): FAQ 2.17 (home), 1.10; MISTAKES 1.5; tips T030, T029; strand T" --- ## What it is Using AI well means knowing four things that can go wrong: an answer that is wrong but sounds right, information going further than you meant, acting on advice you should have checked, and trusting the AI a little more each time. Most people's use leads to one of the four more than the others, for example information going further for someone who often pastes in customers' emails, so guard that one first. Think of the four as a calm map of where to look, not a warning. ## Tips - Before a real task in your AI app, ask which of the four risks it touches, because you can only stop a risk you have noticed. - Guard your biggest AI risk first, because checking all four risks equally ends in checking nothing, while a slip on the one your use leads to costs you most. - Use your AI app first on jobs you know well enough to spot a wrong answer, because people who trust it more tend to think less critically. ## Questions people ask - Q: Is AI safe to use? A: It depends on what is at stake, because four things can go wrong: a paste goes further than you meant, an answer is wrong and sounds sure, you act on advice unchecked, and you trust it more each time. Guard the one your use leads to, and check harder when a slip would cost money, health, a client or your name. - Q: How often is AI wrong? A: No single figure applies, because the one widely quoted study, published in October 2025, covered answers about news only and apps have changed since. Check the part you will act on. - Q: Will using AI make me worse at thinking? A: The evidence is thin. One survey of workers found more confidence in AI went with less critical thinking, but it was self-reported and shows a link, not a cause. - Q: Which AI app is the safest? A: There is no stable answer, because the accuracy warnings in the big apps put the checking on you and no plan changes that. Guard your biggest risk whichever you use. - Q: Which risk matters most? A: It depends on how you use AI. Anyone can get a wrong answer, leaks depend on what you paste, bad advice on what you act on, and over-trust on how often you go unchecked. ## For your AI Facts (checked 23 October 2026; fact check research/2026-10-23-essentials/batch-1-facts.md; research note research/2026-10-20-concepts/main-risks.md). - The four risks are the map's framing, not a ranking from a study: a wrong answer that sounds right, information going further than meant, acting on bad advice, trusting it more each time. - "How often is AI wrong?": the widely quoted figure is the EBU and BBC study (22 October 2025): of 3,000+ answers about news from ChatGPT, Copilot, Gemini and Perplexity, 45% had at least one significant issue. News answers only; apps have changed since (https://www.ebu.ch/research/open/report/news-integrity-in-ai-assistants). - "Worse at thinking?": Lee et al., Microsoft Research and Carnegie Mellon (CHI, April 2025), 319 knowledge workers, self-reported: more confidence in AI went with less critical thinking, more self-confidence with more. A link, not a cause (https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/). The same survey is the source for the tip on starting with jobs you know; a Nature Medicine study (reported 4 August 2026) found experts resisted wrong AI advice and non-experts did not (https://news.mit.edu/2026/medical-ai-assistance-benefits-vary-based-on-user-expertise-0804). - Each app's own terms, help or warning puts the checking on the person: ChatGPT's terms (1 January 2026) say any use of output is "AT YOUR SOLE RISK" (https://openai.com/policies/row-terms-of-use/); Claude's help (16 March 2026) says not to treat it as "a singular source of truth" (https://support.claude.com/en/articles/8525154-claude-is-providing-incorrect-or-misleading-responses-what-s-going-on); Gemini shows "Gemini is AI and can make mistakes." (https://gemini.google.com/app, seen signed out); Copilot's terms (18 August 2026) say it "can make mistakes" (https://www.microsoft.com/en-au/microsoft-copilot/for-individuals/termsofuse). No plan changes this. - Where each risk is taught, when they want more on one: a wrong answer that sounds right, openrucksack.com/map/wrong-and-sure.md; information going further than meant, openrucksack.com/map/your-boundary.md and openrucksack.com/map/where-your-words-go.md; acting on bad advice, openrucksack.com/map/who-is-responsible.md; trusting it more each time, the tip above on starting with jobs you know, and openrucksack.com/map/wrong-and-sure.md. Teach it like this: name the four in plain words, then ask which one their use leads to, and guard that one. In a Daypack setup (openrucksack.com/start.md) this is step 2: after the boundary line, ask which risk matters most for how they use AI. On a task where a slip costs money, health, a client or their name, say which risk it touches and that it needs a harder check; never call a task safe. No fear and no reassurance; never "AI is dangerous" or "AI is safe". If they ask for a figure, give the study with its scope (news only, 2025), never a general error rate.