Common AI risks

The company that runs your AI app receives everything you send. Sending too much means adding names, numbers or details the job doesn't need. When you read, a wrong answer that sounds sure is a reply as confident as a right one. When you act, acting without checking means signing, sending or paying before you check the part you rely on. Over weeks, trusting too much means checking less after good answers, or believing an answer because your AI app agrees with you.

Pasting a rental lease to ask if it is fair sends names and addresses, and the answer can sound sure.

A moment is a point where one of the four risks can happen, such as sending. A stop is the one thing you do at a moment to guard against its risk. Your biggest risk is the one your own use leads to most often.

• Before real work in your AI app, ask which of the four risks could happen, because you can only guard against a risk you have noticed.
• Guard your biggest risk first, because guarding against all four equally tends to end in guarding against none.
• Use your AI app first on jobs you know well, because in one medical study experts spotted wrong AI advice and non-experts did not.

1. Pick a real job you will do with your AI app this week, or your last real chat if no job is coming.
2. In a phone note or on paper, write the four risks by name, one per line: sending too much, a wrong answer that sounds sure, acting without checking, trusting too much.
3. In your note, mark each risk the job could lead to:
   • sending too much, if the job has names, numbers or private details it doesn't need;
   • a wrong answer that sounds sure, if you would rely on a fact in the answer;
   • acting without checking, if you would sign, send or pay because of the answer;
   • trusting too much, if you have stopped checking answers.
4. Pick, for this job, the marked risk that is your biggest, the one your own use leads to most often. Then do the stop for that risk, before you send, read on or act:
   • for sending too much, take out names, numbers and details the job doesn't need;
   • for a wrong answer that sounds sure, check the part of the answer you will act on;
   • for acting without checking, check that part with the original source or a professional before you sign, send or pay;
   • for trusting too much, use your AI app on jobs you know well first.
5. Under the four risks in your note, write one line, "My biggest risk: ..., my stop: ...", and keep the note.

Then finish the job as normal.

To see your biggest risk more clearly, mark several recent chats instead of one.

If nothing is marked, write "nothing at stake this time" in your note, and do the stops when a job puts money, health, a client or your name at stake.

• You may add names, numbers or details the job doesn't need, and your message carries them to the company that runs your AI app. That is sending too much, so take them out before you send.
• Your AI app may give you a wrong answer that sounds sure, so a polished reply can pass as accurate. Check the part of the answer you will act on.
• Your AI app warns that answers can be wrong, which leaves the checking to you, and paying for a plan doesn't change that. Before you sign, send or pay, check the part you rely on with the original source or a professional.
• Your AI app tends to go along with what you say, which can lead to trusting too much. People rate agreeing answers higher. Start with jobs you know well enough to spot a wrong answer.

The risks can overlap, but each shows up at one point in your use, or slowly over weeks, and has one stop. Knowing which risk your use leads to most tells you where to check first.

• How AI works
• What not to share
• Hallucinations
• Responsibility for AI output

• main-risks-1 (Staying safe, waypoint 1): I can name the four main risks of using AI. Proves: in a chat, in their own words and without being shown the list first, they name sending too much, a wrong answer that sounds sure, acting without checking, and trusting too much, each with one thing they would see or one example. Merging two honestly (acting without checking is often a wrong answer acted on) counts.
• main-risks-2 (Staying safe, waypoint 1): I can say which risk matters most for how I use AI and what I do about it. Proves: they pick one, tie it to a real use of their own (a task from "I use AI for" or one of their last chats), and name one stop they will use.
• main-risks-3 (Checking, waypoint 1): I can tell which risk a case touches and the first thing to do. Proves: given a short case, they name the live risk or risks and a first move that is a real action, not "be careful".

In a Rucksack, write the dated journal line when you see the proof ("outcome · main-risks-n · what they did"), and "proven · main-risks" when all three are in. A Daypack keeps no progress: teach it, run the same activities and the fast track, say "that is covered", and record nothing.

Before something that matters, they say which of the four it touches ("this one has a client's details and I'll send it, so the sending stop and the acting stop"). They can say their biggest risk and what they do about it. They stop asking "is AI safe" as one question and ask "which of these is this", and they check more on the jobs that touch their risk and less on the rest.

• Explains: the four risks in any words, each with a sign or an example, not just the label: a wrong answer that sounds sure; names, numbers or details sent that the job didn't need; acting (signing, sending, paying) before checking the part they rely on; and trusting more over weeks, or because the AI agreed with them. Merging the second-last two honestly is a pass. They also say which one their own use leads to.
• Ask: "Think of the last time you used AI for something that mattered. Which of these could it have touched, and what would you have seen if it had?" Shows it: a risk tied to a real use of theirs, with a sign that would have shown it (a figure that did not match something they knew, a name in the paste, an instant "great idea"). "I'd be careful" is not a pass; ask once more for the sign.
• Ask: one case that differs from their own example. Lease: "You paste your rental lease and ask 'is this fair?'. It says it is standard. What is live, and what do you do first?" Client email: "You ask your AI to reply to an angry client and it promises a refund date; you are about to send it." Instant yes: "You say 'I've decided to quit, tell me why that's a good idea' and it says it is a great idea." Diet plan: "It gives you a 1,200-calorie plan and you start tomorrow." Shows it: they name the live risk or risks (lease: sending and acting; email: a wrong answer and acting; instant yes: trusting too much; diet plan: acting) and a first move that is a real action (take the names out; confirm or remove the promise before sending; ask for the case against; take the plan to a GP or dietitian). Never the same case twice to the same person.

Hints for the person's AI. Write the words for the person, in the way their journal says they like to learn; never read these lines to them.

• Explain: Walk the person's last real chat through the four moments, then ask which stop the person skipped.
• Experiment: Ask the person to try a real choice in two new chats. Use "I've decided to [quit / stay] in [a real choice]. Tell me why that is a good idea." Compare the answers. Say, "You may see your AI app find good reasons for both choices." If both answers support the choice given, point out how your AI app went along with both choices. Also offer Try it and help the person sort the marks afterwards.
• Cards: Make four cards, one per risk, each with its moment, a visible sign and the stop from Try it. For sending too much, use a name in a message. For a wrong answer that sounds sure, use a claim that contradicts something the person knows. For acting without checking, use something the person is about to sign, send or pay. For trusting too much, use an instant "great idea" or checking less than before. Keep each stop to one action.
• Quiz: If the person's journal allows quizzes, give four short cases. Use a lease, a reply to an angry client, an instant "great idea" and a diet plan. Ask the person to name the risk and the first move for each case.
• Game, "which stop?": Give a short made-up task from the person's own week. Ask which risk the task could lead to and what the person would do. Play three quick rounds.
• Challenge: For a week, ask the person to name a risk before each real task and note once what the stop caught.

• Name the risk, give its stop in one line and send the person on: the depth belongs to "wrong-and-sure" (why a wrong answer sounds sure, the check that works), "your-boundary" (what never goes in) and "who-is-responsible" (the result is yours; essentials page only until its full page lands). Do not repeat their teaching here.
• The four are the map's framing, not a ranking from a study. No source ranks them; which matters most turns on how the person uses AI. Ask them, never tell them.
• Say "can" and "may", never "will". "AI makes you worse at thinking" is not supported: the evidence is a self-reported link, and in the same survey more self-confidence went with more critical thinking (Lee et al., checked 23 October 2026).
• Never give how often AI is wrong as a number. The widely quoted "almost half" is about news answers only, from October 2025, and the apps have changed since.
• No dread and no reassurance: never "AI is dangerous" or "AI is safe", and do not call a task safe. The overcorrection is checking everything equally, or stopping use. The right thing to do is guard the biggest risk first. Documented harms to ordinary users are a wrong answer, a paste that was kept, an action taken and a habit; never "hackers" or "rogue AI".
• On a task where a slip costs money, health, a client or their name, say which risk it touches and that it needs a harder check.
• Apps differ in how they word the accuracy warning. Each puts the checking on the person, and no plan changed that in anything read on 20 October 2026. The reported Copilot line "for entertainment purposes only" is not on the current page: do not cite it.
• Do not cite, because they could not be confirmed: the bromide case (press only, the chat log was never seen) and the KPMG 57 percent figure (never confuse it with TELUS's 57 percent on "your-boundary").
• The agreement test under Ways to teach it (Experiment) was not run before this page was written: say "you may see it find good reasons for both", never promise it. It runs in the teaching tests.
• Daypack: at setup step 2 the person writes their boundary line, hears the four in a line each and names which matters most; nothing is recorded. The boundary line is the sending stop already in place, and the pasted instructions warn before a paste that breaks it. The other three risks have no setup step in a Daypack, so teach them from the person's own chats. The Rucksack is coming soon: do not offer it as something to set up today. (checked 27 October 2026)

• Q: Is AI safe to use? A: It depends on what is at stake, because four things can go wrong: a paste goes further than you meant, an answer is wrong and sounds sure, you act on advice unchecked, and you trust it more each time. Guard the one your use leads to, and check harder when a slip would cost money, health, a client or your name.
• Q: How often is AI wrong? A: No single figure applies, because the one widely quoted study, published in October 2025, covered answers about news only and apps have changed since. Check the part you will act on.
• Q: Will using AI make me worse at thinking? A: The evidence is thin. One survey of workers found more confidence in AI went with less critical thinking, but it was self-reported and shows a link, not a cause.
• Q: Which AI app is the safest? A: There is no stable answer, because the accuracy warnings in the big apps put the checking on you and no plan changes that. Guard your biggest risk whichever you use.
• Q: Which risk matters most? A: It depends on how you use AI. Anyone can get a wrong answer, leaks depend on what you paste, bad advice on what you act on, and over-trust on how often you go unchecked.

Facts (checked 27 October 2026 against the essentials page's facts of 23 October 2026; research notes research/2026-10-20-concepts/main-risks.md and research/2026-10-23-essentials/batch-1-facts.md).

• The four risks are the map's framing, not a ranking from a study. No source ranks them; it turns on use (research note, section 1, "Which matters most", checked 20 October 2026).
• "How often is AI wrong?": the widely quoted figure is the EBU and BBC study (published 21 October 2025): of 3,000+ answers about news from ChatGPT, Copilot, Gemini and Perplexity, "almost half" had at least one significant issue. News answers only; apps have changed since (https://www.ebu.ch/research/open/report/news-integrity-in-ai-assistants, checked 5 November 2026).
• "Worse at thinking?": Lee et al., Microsoft Research and Carnegie Mellon (CHI, April 2025), 319 knowledge workers, self-reported: more confidence in AI went with less critical thinking, more self-confidence with more. A link, not a cause (https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/, checked 23 October 2026). The same survey is the source for tip 3.
• Know the job first: a Nature Medicine study, reported 4 August 2026, found experts resisted wrong AI advice and non-experts did not (https://news.mit.edu/2026/medical-ai-assistance-benefits-vary-based-on-user-expertise-0804, checked 23 October 2026).
• Agreeing answers: in Science (26 March 2026), 11 systems endorsed users' actions 49 percent more often than people did, and about 2,400 people trusted the flattering ones more (known through the Stanford release; paper not read; https://news.stanford.edu/stories/2026/03/ai-advice-sycophantic-models-research, checked 23 October 2026). Anthropic (28 January 2026), 1.5 million Claude.ai chats: about 1 in 1,300 showed potential for severe reality distortion; this is potential, not harm, and users rated those chats higher "at least in the moment" (https://www.anthropic.com/research/disempowerment-patterns, read 20 October 2026). Give these only if asked, with their limits.
• Each app's own terms or help put the checking on the person, and no plan changes this (read 20 October 2026 unless dated): ChatGPT's terms (effective 1 January 2026) say any use of output is "AT YOUR SOLE RISK" (https://openai.com/policies/row-terms-of-use/); Claude's help (16 March 2026) says to treat answers as not "a singular source of truth" (https://support.claude.com/en/articles/8525154-claude-is-providing-incorrect-or-misleading-responses-what-s-going-on); Gemini shows "Gemini is AI and can make mistakes." (https://gemini.google.com/app, seen signed out, checked 23 October 2026); Copilot's individuals' terms (effective 18 August 2026) say it "can make mistakes" and to check before you act (https://www.microsoft.com/en-au/microsoft-copilot/for-individuals/termsofuse).
• Cases, each only if asked and with its limit: Moffatt v Air Canada (2024 BCCRT 149, 14 February 2024), a company held liable for its own chatbot's wrong refund rule (https://dailyhive.com/canada/air-canada-chatbot-decision); Deloitte partly refunded an AU$440,000 report for the Australian Government that had a fabricated court quote and non-existent papers (AP, 7 October 2025: https://www.ctvnews.ca/business/article/deloitte-to-partially-refund-australian-government-for-report-with-apparent-ai-generated-errors/); 9.9 percent of 2,034 Australian adults used ChatGPT for health in the six months to mid-2024, and 61 percent of them asked a higher-risk question (MJA 2025: https://www.sydneyhealthliteracylab.org.au/generative-ai-and-health-literacy-publications/use-of-chatgpt-to-obtain-health-information-in-australia-2024-insights-from-a-nationally-representative-survey, read 20 October 2026). The Oxford study in Nature Medicine (February 2026, 1,298 UK adults, chatbot users decided no better than people without one) was known through its press release; paper not read (https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice, read 20 October 2026).
• Re-check by 20 January 2027, sooner if a new EBU and BBC accuracy round is published, an app ships a fix for over-agreement, Microsoft changes the Copilot terms, or a controlled study of over-trust appears.

Teach it like this: name the four in plain words, then ask which one their use leads to, and guard that one. If they ask for a figure, give the study with its scope (news only, 2025), never a general error rate. Mistakes to avoid, stated as the right thing to do: ask the person which risk is theirs rather than ranking the four for them; name the live risk and a real first move rather than "be careful"; say "can" and "may" about thinking and trust.

An example of this taught well, in passing, after the person says they are about to paste a lease and ask if it is fair: "That one touches three things. It has names and addresses in it, which go to the company that runs the app, so swap them for labels first. The answer will sound sure whether or not it knows your state's rules, and you would be the one signing. So check what it flags with a tenants' advice service before you sign. Which of those would you most likely skip on a busy day?"