Calibration
Expected calibration error
Implemented
calibration.eceDefinition
Weighted average gap between predicted confidence and observed frequency across probability bins.
Formula
ECE = Σₘ (|Bₘ| / N) · |acc(Bₘ) − conf(Bₘ)|
Range: [0, 1]
Inputs and outputs
- y_true: array of binary labels {0, 1}
- y_prob: predicted probability of the positive class in [0, 1]
- n_bins, strategy: uniform | quantile
Returns: float
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
- Depends on the number of bins and the binning strategy, which are always reported with the value.
Python API
import evalsuite as es
es.expected_calibration_error(y_true, y_prob, n_bins=10)References
- Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1).