Skip to content
EvalSuite
Documentation menu

Robustness and reliability

Counterfactual invariance violation rate

Implementedrobustness.invariance_violation_rate

Definition

Share of examples whose prediction changes when only a protected or irrelevant attribute is changed (names, gender terms, dialect): violations of an invariance test.

Formula

mean[pred(x) ≠ pred(x')]

Range: [0, 1]

Inputs and outputs

  • original_predictions: see the signature of es.invariance_violation_rate
  • counterfactual_predictions: see the signature of es.invariance_violation_rate

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

PythonSince v0.5.0
import evalsuite as es

es.invariance_violation_rate(["pos", "neg"], ["pos", "pos"])

References

  1. Ribeiro MT, Wu T, Guestrin C, Singh S. Beyond accuracy: behavioral testing of NLP models with CheckList. ACL. 2020:4902-4912.
  2. Kaushik D, Hovy E, Lipton ZC. Learning the difference that makes a difference with counterfactually-augmented data. ICLR. 2020.

Implementation status