Calibration and uncertainty
Selective risk
Implemented
uncertainty.selective_riskDefinition
Error rate on the questions the model answers when it abstains on the least confident ones so that a given share (coverage) is answered.
Formula
risk(c) = errors among the ⌈c·n⌉ most confident / ⌈c·n⌉
Range: [0, 1]
Inputs and outputs
- correct: see the signature of es.selective_risk
- confidence: see the signature of es.selective_risk
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.selective_risk(correct, confidence, coverage=0.8)References
- El-Yaniv R, Wiener Y. On the foundations of noise-free selective classification. JMLR. 2010;11:1605-1641.
- Geifman Y, Uziel G, El-Yaniv R. Bias-reduced uncertainty estimation for deep neural classifiers. ICLR. 2019.