Multilingual
Translation adequacy / fluency (direct assessment)
Implemented
multilingual.direct_assessmentDefinition
Human direct-assessment scores (0–100 adequacy or fluency) standardised per rater (z-scores) to remove rater bias, then averaged per system, as in WMT DA.
Formula
z = (score − mean_rater) / sd_rater; system score = mean z
Range: (−∞, ∞) for z; [0, 100] raw
Inputs and outputs
- scores: see the signature of es.direct_assessment
- raters: see the signature of es.direct_assessment
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.direct_assessment([70, 85, 40, 55], raters=["r1", "r1", "r2", "r2"])References
- Graham Y, Baldwin T, Moffat A, Zobel J. Continuous measurement scales in human evaluation of machine translation. LAW. 2013:33-41.