Structured output and tools

Required-field accuracy

Definition

Share of expected fields (dotted paths) present in the parsed output with the expected value, pooled over all examples; a missing field or unparseable output counts as wrong.

Formula

Range: [0, 1]

Inputs and outputs

• expected: per example, field path -> expected value
• outputs: model outputs (strings)

Returns: MetricResult (float, or per-example array with average=None)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.required_field_accuracy(expected_fields, outputs)

References

1. Patil SG, Mao H, Yan F, et al. The Berkeley Function Calling Leaderboard (BFCL): from tool use to agentic evaluation of large language models. ICML. 2025.

Implementation status