Calibration and uncertainty
Risk at fixed coverage
Implemented
uncertainty.risk_at_coverageDefinition
Selective risk at the target coverage, read from the risk-coverage curve at the smallest confidence threshold whose coverage reaches the target (ties are answered together).
Formula
risk at the first curve point with coverage ≥ target
Range: [0, 1]
Inputs and outputs
- correct: see the signature of es.risk_at_coverage
- confidence: see the signature of es.risk_at_coverage
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.risk_at_coverage(correct, confidence, coverage=0.9)References
- El-Yaniv R, Wiener Y. On the foundations of noise-free selective classification. JMLR. 2010;11:1605-1641.
- Geifman Y, Uziel G, El-Yaniv R. Bias-reduced uncertainty estimation for deep neural classifiers. ICLR. 2019.