Skip to content
EvalSuite
Documentation menu

Robustness and reliability

Typographical-noise robustness

Implementedrobustness.noise_robustness

Definition

Accuracy on inputs with typos (character swaps, deletions, insertions, substitutions) relative to clean inputs: perturbed accuracy, absolute and relative drop and flip rate. ``add_typos`` makes the noisy inputs reproducibly.

Formula

acc_noisy; drop = acc_clean − acc_noisy

Range: [0, 1]

Inputs and outputs

  • clean_correct: see the signature of es.noise_robustness
  • noisy_correct: see the signature of es.noise_robustness

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

PythonSince v0.5.0
import evalsuite as es

es.noise_robustness(clean_correct, noisy_correct)  # inputs from es.add_typos

References

  1. Belinkov Y, Bisk Y. Synthetic and natural noise both break neural machine translation. ICLR. 2018.
  2. Ribeiro MT, Wu T, Guestrin C, Singh S. Beyond accuracy: behavioral testing of NLP models with CheckList. ACL. 2020:4902-4912.

Implementation status