We Measured Our LLM's Confidence Scores. They Were Wrong 16% of the Time — Even at Maximum Confidence.

We were building a multi-agent customer support system — the kind of thing you've probably seen a dozen versions of by now. An LLM classifies what the customer wants, a planner figures out what to do about it, some tools run, a response comes back. Standard stuff. Buried in the middle of that pipeline was one small, unglamorous decision that turned out to be the most interesting part of the whole project: when should the system hand a request off to a human instead of answering it itself? The answer we started with was the answer almost everyone starts with. The LLM reports a confidence score alongside its intent classification — a number between 0 and 1 that's supposed to mean "how sure am I." If that number dropped below some threshold, we'd escalate to a person. We picked 0.7, the way most people pick these thresholds: it sounded reasonable. It took one afternoon of actually looking at real data to realize that number meant almost nothing. The assumption nobody checks Here's the thing about "escalate if confidence < 0.7": it only works if the confidence score is telling you the truth. If the model says 0.9, you're implicitly trusting that it's right roughly 90% of the time when it says that. Nobody on our team had actually checked whether that was true. We don't think most people building systems like this check it either — the number is just there, reported alongside the answer, looking authoritative. So we checked. We ran our real intent classifier — Llama 3, running locally — against 662 labeled support messages and recorded two things for each one: what confidence it reported, and whether it was actually right. The result wasn't subtle. The model reported confidence values almost exclusively in the range 0.8 to 1.0 — only 12 distinct values across all 662 predictions. It was, for practical purposes, always "very sure." And when we looked specifically at the requests where it reported maximum confidence — the cases where you'd most want to trust it and skip the human — it was still wrong about 16% of the time. Imagine a system that tells you "I am completely certain" once every six times it's wrong. That's what we were about to build an escalation policy on top of. Fixing it properly, not just eyeballing a better number The obvious next move is to just pick a stricter threshold. Eyeball a chart, move the number from 0.7 to 0.85, ship it. We didn't do that, for a reason that turned out to matter a lot: a threshold you pick by looking at a chart doesn't come with any guarantee. It's a vibe, not a fact. And a vibe is a strange thing to base a human-oversight policy on. Instead we used a method called Selective Guaranteed Risk (Geifman & El-Yaniv, 2017), built originally for classic classification problems, not LLMs — which meant applying it here was itself an interesting experiment, not just plugging in a library. The idea is simple to state: given a set of labeled examples, find the most permissive confidence threshold such that — accounting for the fact you only have a limited sample, not infinite data — you can prove, with a stated probability, that the error rate above that threshold stays under some target (we used 10%). If no threshold can make that promise, the method doesn't pretend otherwise. It tells you to escalate everything. That second part turned out to be the whole story. The honest result: it correctly told us "not yet" We ran the calibration. It fell back — no threshold could be certified. Given that raw accuracy sat around 79%, this was the mathematically correct outcome; you can't promise 90%+ reliability out of a classifier that's only right 79% of the time, no matter how you slice the confidence scores. We could have stopped there and called it a disappointing result. Instead we asked a more specific question: is the problem that the model is overconfident, or that the model is just wrong a lot? Those sound like the same problem. They aren't. We rewrote the classification prompt — added an explicit rubric for what each confidence band should mean, gave it a handful of examples spanning confident-and-clear to genuinely-ambiguous cases, and fixed an actual contradiction in the old prompt (it asked the model to explain its reasoning while also telling it not to explain itself — not great). Re-ran the same 662 examples. Here's what changed and what didn't: Before After Distinct confidence values used 12 8 Confidence range 0.80 – 1.00 0.60 – 0.95 Raw accuracy 79.0% 82.3% Calibration certified a threshold? No Still no The model started actually using its confidence range — spreading out, expressing doubt on ambiguous cases instead of defaulting to "very sure" every time. That's real progress on the calibration problem specifically. But raw accuracy barely moved. Calibration and accuracy are not the same problem, and fixing one doesn't fix the other. That's the finding we'd actually stand behind putting our name on. It's a small, specific, checkable claim — and it's the kind of thing that's easy to conflate if you're not looking closely, because "the model seems more thoughtful now" feels like it should mean "the model is more reliable now." It doesn't, necessarily. What we didn't get to yet In the interest of not doing the thing this whole post is about — reporting a number as more solid than it is — here's what's still open: We never got to see whether a certified threshold actually holds up on data it wasn't calibrated on, because calibration kept correctly refusing to certify one. The held-out validation machinery is built and tested; it just hasn't had anything real to validate yet. The bottleneck now is plainly the classifier's raw accuracy, not the calibration math. That's next. We also built an adaptive version — one that can recalibrate itself from real outcomes over time instead of a single offline dataset — but it's currently a well-tested mechanism waiting for a real feedback signal (confirmation of whether an escalation was actually necessary), which doesn't exist yet in our system. One more thing, because it's a good example of why you check your own numbers before trusting them: while measuring how much faster our system got when it skipped unnecessary steps for easy requests, we initially found zero savings in LLM calls — only in raw latency, which didn't make sense. Turned out our own instrumentation was only counting LLM calls from one part of the pipeline, silently ignoring three others. Fixed the counter, re-ran it, found a real 28% reduction in LLM calls that had been sitting there the whole time, just uncounted. It's a small thing, but it's the same lesson at a different scale: the number sitting in front of you is not automatically the truth. Check it. The takeaway If you're building anything that uses an LLM's self-reported confidence to make a decision — escalate or don't, retry or don't, trust or don't — the actual question worth answering before you ship it isn't "what threshold should I pick." It's "does this number mean anything at all, and how would I know if it didn't." We didn't know. Then we checked. Now we know exactly how much we don't know, which — genuinely — is a better place to be standing.