Factuality and QA
Knowledge consistency
Implemented
factuality.knowledge_consistencyDefinition
Share of facts extracted from the outputs (e.g. subject–relation–object triples) that are present in a specified trusted knowledge source.
Formula
|extracted facts ∩ knowledge base| / |extracted facts|
Range: [0, 1]
Inputs and outputs
- extracted_facts: facts extracted from each output
- knowledge_base: set of trusted facts
Returns: MetricResult (float, or per-example array with average=None)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.knowledge_consistency(extracted_facts, knowledge_base)References
- Min S, Krishna K, Lyu X, et al. FActScore: fine-grained atomic evaluation of factual precision in long form text generation. EMNLP. 2023:12076-12100.