Robustness and reliability
Recovery success rate
Implemented
robustness.recovery_success_rateDefinition
Among episodes in which a failure occurred (an error, a failed tool call, a wrong intermediate step), the share from which the system recovered and still completed the task.
Formula
#(failed ∧ recovered) / #failed
Range: [0, 1]
Inputs and outputs
- failed: see the signature of es.recovery_success_rate
- recovered: see the signature of es.recovery_success_rate
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.recovery_success_rate(failed=[True, True, False], recovered=[True, False, False])References
- Liang P, Bommasani R, Lee T, et al. Holistic evaluation of language models. TMLR. 2023.