Why does AI give different answers to the same question?
Published 8 min read
OpenAI model via Codex CLI with web search, not the ChatGPT app, 01-02.11.2026, 13 software questions.
Why does ChatGPT give different answers to the same question? It picks words with some chance, it searches again each time, and the model behind it changes. So one answer is a snapshot.
Card needed. Cancel before day 7 and nothing is charged.
11 websites were cited across 10 runs of one CRM question. 4 came back in 8 or 9 runs, 7 in 4 runs or fewer. None in all 10.
- 4Cited in 8 or 9 of 10 runs
- 7Cited in 1 to 4 of 10 runs
The short answer
ChatGPT gives different answers because of chance in each word, a new search each time and new models. So one answer is a snapshot. Count how often you are named across many runs and weeks.
OpenAI model via Codex CLI with web search, not the ChatGPT app, 01-02.11.2026.
Key points
The search moves too. Not only the order of names changes. The searches and the cited pages change from run to run.
Models change. OpenAI changes the model behind ChatGPT over time. Older answers may come from an older model.
Count, do not screenshot. Named in 6 of 10 runs tells you more than one answer.
Reason 1: AI picks each word with some chance
AI writes an answer one word at a time, from several likely words. A little chance in each pick means two runs can end up different.
- Small changes early in an answer grow. A different first name can change the whole list.
- This is how the tools are built. It is not a fault, and you cannot switch it off in the chat app.
Reason 2: AI runs a new search every time
With search on, each run writes its own searches and reads other pages. In our runs, most searches were worded in a new way, and no website was cited in all 10.
| Question (by category) | Searches per run | Different wordings of all searches | Different pages cited | Most-cited website, runs |
|---|---|---|---|---|
| Website builder for a law office | 2 to 10 | 57 of 59 | 35 | 9 of 10 |
| AI assistant for 20 people | 4 to 8 | 60 of 66 | 25 | 8 of 10 |
| Project tool for 20 people | 1 to 9 | 50 of 56 | 21 | 9 of 10 |
| Website builder for a bakery | 4 to 8 | 54 of 66 | 20 | 8 of 10 |
| CRM for 10 people | 5 to 10 | 51 of 70 | 18 | 9 of 10 |
| Accounting software for 10 people | 3 to 8 | 46 of 52 | 24 | 7 of 10 |
6 buying questions, 10 runs each, every run a fresh chat. The 10 runs of each question started within one minute. Different wordings = how many distinct search texts there were among all searches for that question. Product names left out. OpenAI model (gpt-5.6-luna) via Codex CLI with web search, not the ChatGPT app, 01-02.11.2026.
- Take the CRM question. Four makers' sites came back in 8 or 9 of 10 runs. Run 9 cited none of them, only a review site and a software list.
- So the sources behind an answer move. A page of yours can be read in one run and missed in the next.
- The order of names moves too. One question asked 10 times gave 9 different top-5 orders. The full table is on the LLM visibility tracker page.
| Run | Of the 4 most-cited makers' sites | Other sites cited |
|---|---|---|
| 1 | 4 | 1 maker's site |
| 2 | 4 | 1 maker's site |
| 3 | 4 | 1 maker's site |
| 4 | 3 | 1 maker's site, 1 help page |
| 5 | 4 | 1 maker's site |
| 6 | 4 | 1 maker's site |
| 7 | 4 | 1 maker's site |
| 8 | 4 | 1 maker's site |
| 9 | 0 | 1 review site, 1 software list |
| 10 | 4 | 1 maker's site |
Cited websites per run, counted by web address. Product names left out. OpenAI model (gpt-5.6-luna) via Codex CLI with web search, not the ChatGPT app.
Reason 3: The model behind the answer changes
OpenAI changes the model behind ChatGPT over time, as in September and October 2026. A new model can search, read and pick names in a new way.
| Date | What changed |
|---|---|
| 2026-09-03 | OpenAI introduced GPT-6 Astra, rolled out in stages. |
| 2026-10-14 | GPT-5.5 left ChatGPT. It stayed available in the OpenAI API. |
| 2026-11-01 | Our runs on this page (1-2 November) used gpt-5.6-luna through Codex CLI. |
Dates follow OpenAI's release notes, checked on 11 November 2026.
- So an answer from last month may show an older model. Compare answers from the same weeks only.
- We keep this log on one page instead of one page per model.
Does ChatGPT give the same answer to everyone?
No. Answers can differ by country, language, chat history and settings like memory, so two people can get different companies.
- With search on, ChatGPT may add a place to the search, for example your city.
- A logged-in person with memory on may get answers shaped by earlier chats.
- So a friend's screenshot shows their answer, not your customers'.
What do changing AI answers mean for your company?
Measure a share, not a single answer. Ask the same questions many times, on several engines, every week, and count how often you are named.
Same questions
Keep the wording fixed, so a change comes from the answers.
Several runs
Named in 6 of 10 runs is a pattern. Named once is luck.
Three engines
ChatGPT, Perplexity and Google AI each search in their own way.
Week by week
A trend over weeks shows if a fix worked.
If a rival keeps coming back across runs, read why AI recommends my competitor.
How does Nenn measure answers that keep changing?
Nenn asks the same questions every week on 3 engines and shows how often you are named.
Mentioned in
The share of answers that name your company.
Share of voice and position
How you compare with the names AI gives, by topic and engine.
Trend
From the second weekly check on, you see the line move.
- It is a sample of real answers, not a promise about every chat.
Questions about changing AI answers
Does ChatGPT give different answers every time?
Often, yes. In our runs, every buying question came back in more than one top-5 order.
Why does ChatGPT recommend different companies each time?
Each run writes new searches and reads other pages. In our runs, no website was cited in all 10 runs of a question. So the companies those pages support can change too.
Does a new ChatGPT model change who it recommends?
It can. A new model may search and pick sources in a new way. Compare answers from the same weeks, and check the model log on this page.
Can I make ChatGPT give me the same answer every time?
Not in the chat app. You can repeat the question in new chats and count the names. That count is more useful than one fixed answer.
Do Perplexity and Google AI also give different answers?
They also search for each answer, so they can change too. Our numbers here come from one OpenAI model only. Check each engine on its own.
One answer is luck. Count the answers.
Nenn asks the same questions every week on 3 engines and shows how often you are named. Try it on your own customer questions for 7 days.
Card needed. Cancel before day 7 and nothing is charged.
Which CRM should a company with 10 people use?
- 1Named in ? of 10
Keep reading
From one answer to a number you can follow.