Why an AI's own confidence score is not worth trusting, and what to do instead

In a blog post, Amrut Joshi argues that when you ask an LLM how confident it is, the number it gives (say 0.85) is just more generated text, not a measurement. It was never checked against how often the model is actually right, so it can change if you ask again or reword the question. His fix: let the AI do the task, but work out confidence from things you can observe, such as whether search found a matching help article or whether two runs agree, and keep a record of which answers turned out correct. Then check regularly that, for example, replies scored 90% really are right about 90% of the time. On Hacker News (202 points, 96 comments from 30 top-level commenters when we read it), one camp agreed that a score means nothing without a record of real outcomes to check it against. Another camp pushed back: some said models can be trained or fine-tuned to give useful confidence, others suggested reading the model's token probabilities instead, and the most-replied comment said splitting the rating into a few small yes or no checks works much better than a 0 to 100 scale. No consensus. Sorted take: whichever camp is right, both agree on one thing. Do not route work on a number nobody has checked against real outcomes.