Think Better · 55 sec
A 70% Forecast Should Be Right About 70% of the Time
Confidence becomes meaningful only when it matches results across many predictions.
The useful idea
- Judge a group
A single outcome cannot show whether a probability was good; calibration is evaluated across many similar forecasts.
- Match frequency
Among a large group of 70% forecasts, the event should occur about 70% of the time if the forecasts are calibrated.
- Find overconfidence
If events labeled 90% happen only 60% of the time, the forecaster is systematically more confident than the record supports.
Why this matters
Tracking calibration separates a repeatable estimate of uncertainty from confidence that merely sounds precise.
Try this
Record repeated predictions with probabilities, then compare each confidence range with how often it actually occurred.
Test your recall
Which action best applies “A 70% Forecast Should Be Right About 70% of the Time”?
- Ask: What remains true when the examples change?
- Record repeated predictions with probabilities, then compare each confidence range with how often it actually occurred. — correct
- Change the two most uncertain assumptions across plausible ranges and note where your preferred option switches.
Tracking calibration separates a repeatable estimate of uncertainty from confidence that merely sounds precise.