--- id: comparing-two-models title: Comparing AI models level: 2 modes: [day-pack, rucksack] moment: "The person wonders which AI is better for their job, has tried two apps or settings once each and is deciding which to use, or is choosing a model for a job that matters." keywords: [compare models, which AI is better, ChatGPT vs Claude, same task, fresh chat, different answers, model picker, fair test, auto routing, side by side] source: "research/2026-10-20-concepts/comparing-two-models.md; research/2026-10-23-essentials/batch-6-facts.md; mapping (product/learning-system/MAPPING-1-2.md): FAQ 1.05, 3.12, 4.17; MISTAKES 1.6, 3.6; tips T061, T169; strand U" --- ## What it is Running the same real task, such as the weekly email to your customers, through two AI apps, or two settings of one app, shows which suits your job. Give each a fresh chat and the same words, so the earlier conversation cannot tilt the result. Because the same ask can give different text each time, one reply is only a sample, so repeat the test before you trust a difference. Then judge each reply against what a good reply needs, not how long it is. ## Tips - Test two AI apps on your own task in fresh chats with the same wording, because one reply is only a sample of what each can do. - Write down what a good reply needs before you compare, then judge both replies against that list and not length, because long answers can look better without being better. ## Questions people ask - Q: How do I compare two AI apps fairly? A: Give both the same real task in the same words, each in a fresh chat, and judge each reply against what a good reply needs, not its length. Run it again before you trust a difference, because one reply is only a sample. - Q: Why do I get a different answer every time I ask? A: The AI behind your app picks each next small piece of text from a set of likely ones, so the same ask can give different text. One reply is a sample, not a verdict. - Q: Can I compare models inside one app? A: Yes, where your plan lets you choose between versions or settings. Use a new chat for each choice, because if you switch partway, the new one can read the earlier replies. - Q: How do I know which model answered? A: Some apps choose for you automatically, and some can switch partway through a chat, so a test of two can end up as one twice. Check whether your app names the version behind a reply, and choose it yourself where you can. ## For your AI Facts (checked 23 October 2026; fact check research/2026-10-23-essentials/batch-6-facts.md; research note research/2026-10-20-concepts/comparing-two-models.md). - One reply is a sample: replies are drawn with some randomness, piece by piece (tokens, often parts of words), and even at the lowest setting results are not fully repeatable; one test of 1,000 runs at temperature 0 gave 80 different outputs (Thinking Machines, 10 September 2025, https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/). "One output cannot establish how well an AI system performs" (Nielsen Norman Group, 14 August 2026, https://www.nngroup.com/articles/eval-ai-output/). - Length sways raters: in public model votes, length was the strongest style factor (LMSYS, 29 August 2024, https://lmsys.org/blog/2024-08-28-style-control/); that sways judgement, it does not show long answers are worse. Anthropic's advice: compare where the answers differ, not how long they are (https://academy.claude.com/tutorials/choosing-the-right-claude-model). - Some apps choose the model for you: Gemini's Auto (its "See response details" shows which model answered, https://support.google.com/gemini/answer/18648722), Copilot's router, Claude's labelled fallback when a request is flagged, and ChatGPT when a thinking limit is reached. Claude lets you change the model mid-chat, from the next reply (https://support.claude.com/en/articles/8664678-change-the-model-effort-and-thinking-settings). Per app (plans move): ChatGPT, Free and Go get one model family with Instant and Think; Plus adds higher thinking levels. Claude, Free has Haiku and Sonnet, Pro adds Opus (https://claude.com/pricing). Gemini, its help page lists Auto or Flash-Lite with no plan, and a news report (9 October 2026) says free accounts now get Auto only; paid plans add more. Copilot, modes rather than named models. Teach it like this: when they are choosing, suggest one real task in both, fresh chats, the same words, and a short list of what a good reply needs, written before they compare (best before they read either reply). Never rank the apps yourself.