A model does not sound less certain when it is making something up. The sentence arrives with the same rhythm, the same specificity and the same tidy structure as the sentences around it that happen to be true. That is the whole problem, and it is not a bug anyone is going to fix, because fluency and accuracy are produced by different things and only one of them is visible to you.
Which means the instinct you have used your entire life, that a confident and well organised answer is more likely to be right than a hesitant one, is now working against you.
- Confidence carries no information about accuracy. It never did, but with people it correlated a little.
- Sort every output into checkable and uncheckable. They need completely different handling.
- Errors enter at four points, and each one has a different fix.
- Suspicious specificity is a better warning sign than hedging language.
- Review that asks "is this right" misses what review asking "where did this come from" catches.
Why confidence and accuracy came apart
When a person tells you something, their delivery leaks information. They pause, they qualify, they say they think so but would want to check. None of this is reliable, and it is better than nothing, and you have spent decades calibrating on it.
A language model produces the most plausible continuation of the text so far. Plausible prose is fluent prose. So the output reads as assured whether the underlying material was solid, thin or absent, and the absence of hedging tells you nothing about whether hedging was warranted.
People try to solve this by prompting, asking the model to say when it is unsure. It helps a little and it cannot work properly, because the model is generating a claim about its own certainty using the same process that generated the answer. A wrong answer can arrive with a sincere sounding declaration of confidence attached, and now you have two problems.
The only division that matters
Forget grading output by how right it looks. Sort it by whether you could establish that it is wrong, and how much that would cost.
| Kind of output | Examples | How you check it |
|---|---|---|
| Checkable, cheaply | A date, a name, a quote, a function signature, whether a setting exists | Open the source. Seconds to minutes. |
| Checkable, expensively | A legal position, a medical claim, whether a library really behaves that way under load | Requires an expert or a test. Hours, or money. |
| Not checkable | Emphasis in a summary, "most teams do this", a judgement about what matters | There is nothing to open. It is an opinion wearing a fact’s clothing. |
The rule that falls out of this is short. Uncheckable output is fine for thinking with and unacceptable as a deliverable anywhere being wrong has a cost. You can absolutely use a model to work out what questions to ask. You should not let it be the last thing that touched a claim before someone else relies on it.
Most of the damage people describe comes from putting an uncheckable output into a checkable-looking container. A summary paragraph in a report looks exactly like the researched paragraph beside it.
Where errors actually enter
Four places, and confusing them is why people apply the wrong fix.
The information never reached the model. You asked about a document and pasted half of it, or the PDF you copied from dropped its tables silently. The model then answered about the material it had, which is the correct behaviour and produces a wrong answer. This is the most common cause and the least discussed, because it looks identical to invention from the outside.
Your question contained a false premise. Ask why a feature behaves a certain way and you will get an explanation of why it behaves that way, including when it does not. Models are strongly disposed to accept the frame of the question. Asking whether something is true, before asking why it is true, catches more of this than any amount of careful phrasing afterwards.
The model filled a gap with something plausible. This is the failure everyone means when they say hallucination. It happens at the edges of what the model knows, which is exactly where your obscure and specific questions live.
It was true and went stale. Pricing, interface labels, library behaviour, tax thresholds. Nothing was invented and the answer is still wrong. No amount of source checking against the model helps here, because the model is faithfully reporting a world that has moved.
The first cause, handled
Most of the wrongness people blame on the model is material that never arrived. Our PDF Content Extractor pulls the complete text out of a document in your browser, so what you paste in is what the file actually says. Free, no account, nothing uploaded.
Try the PDF Content ExtractorThe warning signs, and why most of them are useless
People watch for hedging, and hedging is a poor signal in both directions. Plenty of correct answers are stated flatly, and a fabrication can arrive wrapped in caveats.
What is worth watching is specificity that has outrun the question. A round number where you asked something that would not produce a round number. A citation formatted perfectly, with volume and page, for a claim you cannot find anywhere. A quotation that supports your argument slightly too neatly. Unanimity, where the model reports a settled consensus in a field you know to be contested.
All of those share a shape: the output is more precise than the underlying question could support. Precision is cheap to generate and expensive to verify, which is why it accumulates in exactly the places nobody checks.
A fabricated claim and a well sourced one are equally easy to write. Only one of them is easy to check, and that is the only difference you can act on.
A verification step that survives a busy week
Elaborate checking procedures get abandoned by Thursday. This one has three questions and does not.
Which sentence here would embarrass me if it were wrong? Usually one or two out of twenty. Check those and consciously accept the rest. Trying to verify everything means verifying nothing, which is how people end up checking the easy claims and skipping the load bearing one.
Where would this have come from? Not whether it sounds right. Whether a source plausibly exists. A claim with no plausible origin is a claim to open before you use.
Would I put my name on this sentence alone? Detached from the confident paragraph around it, most weak claims stop looking convincing immediately. The paragraph was doing the work, not the fact.
Alongside those, one hard rule earns its keep: anything presented as a quote, a figure or a citation gets opened or gets deleted. No exceptions for claims that seem obviously fine, because obviously fine is the category that gets through.
The error that survived two reviews
We had a model help draft a short spec from an existing document. The draft stated a processing limit, phrased confidently, in the right place, in the right units. The number was not in the source. Nothing in the source implied it.
Two people read that spec before it was used. Both approved it. Neither was careless, and neither caught it, because both were asking whether the document read correctly and made sense. It did read correctly. It made complete sense. A limit like that is exactly what you would expect to find in a spec like that, which is why the sentence sailed past two attentive readers.
It surfaced when somebody went to implement it and could not find where the number came from.
The lesson is about review rather than about models. Reviewing for sense catches contradictions and gaps. It cannot catch a plausible invention, because plausibility is the thing it is testing for. Catching that needs a different question, asked deliberately: for each specific claim, where did this come from? That question is slower and it is the only one that would have worked.
When to stop using a model for this
There are tasks where verification costs more than doing the work yourself, and using a model on them is a net loss dressed up as a saving.
Anything where you cannot evaluate the answer and cannot cheaply find someone who can. Anything where a plausible wrong answer is worse than no answer, which covers most medical, legal and financial specifics. Anything you would have to check line by line, since at that point you have swapped writing for proofreading and proofreading is the slower of the two.
Conversely, the tasks that stay worth it are the ones where you can check quickly or where being wrong is cheap. Drafting something you will rewrite. Reformatting. Finding candidate answers you will then confirm. Explaining a concept you can immediately test against something you already know.
What to do this week
Take the last thing a model produced that you actually used. Go through it and mark each sentence checkable or uncheckable. Then look at how many of the uncheckable ones you passed on to someone else as though they were established.
That count is the size of the problem, and for most people it is larger than expected and easy to halve. Once you have seen it, the structural version of this is worth reading: the verification step is one of four parts every AI workflow needs, and writing the instruction so the gaps are closed in advance prevents a good share of this before it starts.
Frequently asked questions
How can I tell when a model is guessing?
You mostly cannot, from the text alone, and that is the honest answer. Guessing and knowing produce the same fluent prose. What you can do is notice where guessing is likely: obscure specifics, anything numeric, anything about a named source, and anything the model would have had little training data on. Treat those as unverified by default rather than trying to read confidence off the wording.
Does asking for sources fix it?
It helps and it introduces a new failure. Asking for citations makes the claims checkable, which is the useful part. It also means any invented claim now arrives with an invented citation attached, which looks more credible than the bare claim did. Sources are only worth requesting if you are going to open them.
Are newer or larger models more reliable?
Generally yes on breadth of knowledge, and it does not remove the problem. A better model is wrong less often, which means you check less often, which means the errors that do get through meet less resistance. The verification step matters more as accuracy improves, not less.
Should I ask the model how confident it is?
A stated confidence is generated the same way the answer was, so a wrong answer can come with a high number attached. It is worth asking what would have to be true for the answer to be wrong, because that produces something you can actually go and check.
What if I do not know enough to check the answer?
Then you are not in a position to use it as an answer, only as a starting point for learning. That is a legitimate use. The failure is treating output you cannot evaluate as a deliverable, which is where an error stops being yours to catch and becomes someone else’s to discover.
Does this mean AI is not useful for research?
It means it is useful for a different part of research than people expect. It is strong at organising, drafting, finding structure and telling you what questions exist. It is weak at being the final authority on a fact. Use it for the first and go elsewhere for the second.
How this was put together: the four failure modes and the review distinction come from our own work building the tools on this site and from errors we have had to trace afterwards. The spec described above is ours. We have quoted no figures on how often models are wrong, because published rates vary enormously by task and model and any single number would mislead more than it informed. Nothing here is legal, medical or financial advice.