Structured output and tools

Format compliance

Definition

Share of outputs that fully match a required format, given as a regular expression (for example a refusal template or an answer line such as 'Answer: <letter>').

Formula

Range: [0, 1]

Inputs and outputs

• outputs: model outputs (strings)
• pattern: regular expression

Returns: MetricResult (float, or per-example array with average=None)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.format_compliance(outputs, r"Answer: [A-D]")

References

1. Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models (IFEval). arXiv:2311.07911. 2023.

Implementation status