The calculator that isnβt
Type =2+2 into a spreadsheet. You get 4. Every time, without variation, for every person on Earth. The spreadsheet calculates β it applies a fixed rule to an input and arrives at the only correct answer.
Now ask a large language model βWhat is 2+2?β You will likely get 4 too β but for a completely different reason. The model has seen the question and its answer countless times in its training data, so it predicts that 4 is the most probable continuation. It isnβt doing arithmetic. It is pattern-matching.
That distinction β calculate versus estimate β is the single most useful frame for working with AI.
What βpredict the next tokenβ actually means
An LLM (a large language model, the technology behind ChatGPT and similar tools) is trained on an enormous volume of text. Its job is to answer one question over and over: given everything written so far, what word β or word-fragment β is most likely to come next?
Each output is a probability distribution over the vocabulary: this word 40%, that word 25%, another word 15%, and so on. The model samples from that distribution, appends the result, and asks again. There is no lookup table, no formula, no notion of βthe correct answer.β Only: what is most probable?
This works surprisingly well across a vast range of language tasks. It fails in ways that are equally surprising β and important to understand.
Why it can be fluent and wrong at the same time
Fluency and accuracy are independent properties. A sentence can be grammatically perfect and factually false. It can sound authoritative while describing something that never happened.
An LLM doesnβt have a fact-checker sitting alongside it, cross-referencing claims against reality. It only knows that certain patterns of words tend to follow other patterns. If a confident, well-structured sentence is the most probable continuation, the model produces it β regardless of whether the underlying claim is true.
This is what the field calls hallucination: the model generates plausible-sounding content that is wrong. It isnβt lying, in the human sense. It isnβt deliberately deceiving. It simply has no mechanism to distinguish βlikely to appear in textβ from βtrue in the world.β
Research papers studying various models have documented hallucination rates ranging from a few percent to over twenty percent, depending on the task type and domain β though the numbers vary widely by model and evaluation method. The point isnβt the exact figure. It is that hallucination is an inherent property of the architecture, not a bug that will be patched away entirely.
Why the same prompt gives different answers
Because the output is sampled from a probability distribution rather than computed from a formula, adding a controlled amount of randomness to the sampling (a setting called temperature) means the same prompt can produce different outputs each time. That isnβt a defect β it is sometimes desirable for creative tasks. But it means you cannot treat an LLM like a deterministic function.
Ask it to count the words in a sentence: it might get it right, and it might not. Ask it to verify a number: it will reason around the number rather than count precisely. Ask it to recall a publication date: it may produce a plausible but incorrect one. These arenβt edge cases. They are predictable weaknesses that follow directly from the architecture.
Why it struggles with exact arithmetic
A calculator holds and manipulates exact values. An LLM manipulates tokens β text fragments that happen to look like numbers. When you ask it to multiply 847 by 293, it isnβt running multiplication. It is producing the token sequence that most resembles the output of a multiplication, based on examples it has seen. For small, common calculations, the pattern-match usually works. For larger or unusual ones, it often doesnβt.
This isnβt a solvable problem within the current architecture. It is why well-designed systems pair a language model with actual computational tools β a real calculator, a real database β when precision is needed. You donβt ask the probabilistic component to do the deterministic work.
The practical posture: verify, donβt trust
None of this means the technology is useless. A skilled estimator β a doctor reading symptoms, an editor reading a draft, an engineer reviewing a specification β is genuinely valuable even though their output isnβt guaranteed correct. The appropriate response to an estimate is to verify it, not to take it on faith.
The same posture applies here. An LLM can draft, summarise, suggest, and reason at speeds that were impossible a few years ago. The output is a high-quality first guess, not a signed certificate. For anything that matters β a factual claim, a calculation, a legal reference, a piece of code that will run in production β check the output against a source you trust.
That shift in posture, from trusting to verifying, is what separates people who use AI effectively from people who get burned by it.
Where to go from here
The six limits of a language model on its own β including the knowledge boundary, the inability to act, and more β are explored in From LLMs to agents. And if the vocabulary around βAI,β βLLM,β and βChatGPTβ is still blurry, AI is not one thing builds that map first.