Structured output and tools
Instruction compliance rate
Implemented
structured-output.instruction_compliance_rateDefinition
Share of outputs that satisfy every instruction checked for them (IFEval prompt-level strict accuracy).
Formula
outputs with all checks passed / outputs
Range: [0, 1]
Inputs and outputs
- checks: per output, pass/fail of each verifiable instruction
Returns: MetricResult (float, or per-example array with average=None)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.instruction_compliance_rate(checks)References
- Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models (IFEval). arXiv:2311.07911. 2023.