Structured output and tools
Constraint satisfaction rate
Implemented
structured-output.constraint_satisfaction_rateDefinition
Share of individual constraints satisfied, pooled over all outputs (IFEval instruction-level accuracy).
Formula
satisfied constraints / constraints
Range: [0, 1]
Inputs and outputs
- checks: per output, pass/fail of each verifiable instruction
Returns: MetricResult (float, or per-example array with average=None)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.constraint_satisfaction_rate(checks)References
- Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models (IFEval). arXiv:2311.07911. 2023.