Multilingual
Language consistency
Implemented
multilingual.language_consistencyDefinition
Share of responses written in the language the user asked or wrote in (language confusion), given the detected language of each response.
Formula
mean[detected = expected]
Range: [0, 1]
Inputs and outputs
- expected_languages: see the signature of es.language_consistency
- detected_languages: see the signature of es.language_consistency
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.language_consistency(["hi", "en"], ["hi", "hi"])References
- Marchisio K, Ko WY, Bérard A, Dehaze T, Ruder S. Understanding and mitigating language confusion in LLMs. EMNLP. 2024:6653-6677.