Calibration and uncertainty
Coverage at fixed risk
Implemented
uncertainty.coverage_at_riskDefinition
The largest share of questions the model can answer (abstaining on the rest by confidence) while keeping the selective error rate at or below the target risk.
Formula
max coverage over curve points with risk ≤ target
Range: [0, 1]
Inputs and outputs
- correct: see the signature of es.coverage_at_risk
- confidence: see the signature of es.coverage_at_risk
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.coverage_at_risk(correct, confidence, risk=0.05)References
- El-Yaniv R, Wiener Y. On the foundations of noise-free selective classification. JMLR. 2010;11:1605-1641.
- Geifman Y, Uziel G, El-Yaniv R. Bias-reduced uncertainty estimation for deep neural classifiers. ICLR. 2019.