Robustness and reliability
Out-of-distribution accuracy
Implemented
robustness.ood_accuracyDefinition
Accuracy on out-of-distribution examples, with in-distribution accuracy and the gap (WILDS reports both; the gap is the generalisation cost).
Formula
acc_OOD = mean[correct | OOD]; gap = acc_ID − acc_OOD
Range: [0, 1]
Inputs and outputs
- correct: see the signature of es.ood_accuracy
- is_ood: see the signature of es.ood_accuracy
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.ood_accuracy(correct, is_ood)References
- Koh PW, Sagawa S, Marklund H, et al. WILDS: a benchmark of in-the-wild distribution shifts. ICML. 2021:5637-5664.