Robustness and reliability
Failure / timeout / error rate
Implemented
robustness.error_rateDefinition
Share of requests that did not complete normally, split into errors and timeouts (any other status label is counted as its own failure kind).
Formula
(#error + #timeout + …) / #requests
Range: [0, 1]
Inputs and outputs
- statuses: see the signature of es.error_rate
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
- ``statuses``: one status per request (``"ok"``, ``"error"``, ``"timeout"``, an HTTP code, ...). Values in ``ok`` count as success.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.error_rate(["ok", "timeout", "ok", "error"])References
- Liang P, Bommasani R, Lee T, et al. Holistic evaluation of language models. TMLR. 2023.