Calibration and uncertainty
Area under the risk-coverage curve (AURC)
Implemented
uncertainty.aurcDefinition
Mean selective risk over all coverages when questions are answered in order of decreasing confidence; lower means confidence ranks errors last. E-AURC subtracts the AURC of a perfect ranking.
Formula
AURC = (1/n) Σ_k risk(k/n); E-AURC = AURC − AURC*
Range: [0, 1]
Inputs and outputs
- correct: see the signature of es.aurc
- confidence: see the signature of es.aurc
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.aurc(correct, confidence) # E-AURC in paramsReferences
- Geifman Y, Uziel G, El-Yaniv R. Bias-reduced uncertainty estimation for deep neural classifiers. ICLR. 2019.