Structured output and tools

JSON validity

Definition

Share of outputs that parse as JSON (the whole output, or the fenced / embedded JSON in lenient modes).

Formula

Range: [0, 1]

Inputs and outputs

• outputs: model outputs (strings)

Returns: MetricResult (float, or per-example array with average=None)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.json_validity(outputs, mode="fenced")

References

1. Bray T. The JavaScript Object Notation (JSON) data interchange format. RFC 8259. 2017.

Implementation status