Calibration
Maximum calibration error
Implemented
calibration.mceDefinition
Largest gap between confidence and observed frequency over all bins.
Formula
MCE = maxₘ |acc(Bₘ) − conf(Bₘ)|
Range: [0, 1]
Inputs and outputs
- y_true: array of binary labels {0, 1}
- y_prob: predicted probability of the positive class in [0, 1]
- n_bins, strategy
Returns: float
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
- Dominated by sparsely populated bins.
Python API
import evalsuite as es
es.maximum_calibration_error(y_true, y_prob, n_bins=10)References
- Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1).