Skip to content
EvalSuite
Documentation menu

Calibration and uncertainty

Adaptive calibration error (ACE)

Implementeduncertainty.adaptive_calibration_error

Definition

Calibration error with equal-mass bins: the unweighted mean |accuracy − confidence| over R bins that each hold the same number of predictions, so sparse high-confidence regions do not hide errors.

Formula

ACE = (1/R) Σ_r |acc(r) − conf(r)|, bins of equal count

Range: [0, 1]

Inputs and outputs

  • correct: see the signature of es.adaptive_calibration_error
  • confidence: see the signature of es.adaptive_calibration_error

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

PythonSince v0.5.0
import evalsuite as es

es.adaptive_calibration_error(correct, confidence, n_bins=15)

References

  1. Nixon J, Dusenberry M, Jerfel G, Nguyen T, Liu J, Zhang L, Tran D. Measuring calibration in deep learning. CVPR Workshops. 2019.

Implementation status