Factuality and QA

Claim verification accuracy

Definition

Accuracy of supported / contradicted / not-enough-info verdicts against gold labels (FEVER label accuracy); macro F1 is reported alongside because the classes are often imbalanced.

Formula

Range: [0, 1]

Inputs and outputs

• gold_verdicts: gold claim labels
• predicted_verdicts: predicted claim labels

Returns: MetricResult (float, or per-example array with average=None)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.claim_verification_accuracy(gold_verdicts, predicted_verdicts)

References

1. Thorne J, Vlachos A, Christodoulopoulos C, Mittal A. FEVER: a large-scale dataset for fact extraction and VERification. NAACL. 2018:809-819.

Implementation status