When you say you're 90% sure, are you right 90% of the time? Most people aren't — confidence runs ahead of accuracy. Log a batch of past calls with how sure you were and what actually happened, and read your reliability diagramA reliability diagram plots your stated confidence against how often you were actually right. Perfect calibration sits on the diagonal; points below it mean you were more sure than you should have been.: the gap from the diagonal is the gap between feeling sure and being right.
| Prediction | ConfidenceHow likely you said it was, as a percentage — your stated probability at the time. | Outcome | |
|---|---|---|---|
| % | |||
| % | |||
| % | |||
| % | |||
| % | |||
| % | |||
| % | |||
| % | |||
| % | |||
| % |
Across 10 calls, you were 82% sure on average and right 60% of the time — you're overconfident. Your Brier scoreThe Brier score is the mean squared error of your probabilities: 0 is perfect, and a flat 50%-on-everything guess scores 0.25. Higher than that means your confidence is actively hurting you. is 0.354 (0 is perfect; a flat 50% guess scores 0.25).
| Confidence band | You said | Happened | n |
|---|---|---|---|
| 60–70% | 60% | 100% | 1 |
| 70–80% | 73% | 100% | 2 |
| 80–90% | 82% | 67% | 3 |
| 90–100% | 93% | 25% | 4 |
Calibration is a large-sample idea — a handful of calls is noisy. Aim for 20+ before you trust the verdict.