Structured output and tools

JSON Schema compliance

Definition

Share of outputs that parse as JSON and validate against a JSON Schema (types, required and additional properties, enums, ranges, patterns, arrays, combinators, local $ref).

Formula

Range: [0, 1]

Inputs and outputs

• outputs: model outputs (strings)
• schema: a JSON Schema

Returns: MetricResult (float, or per-example array with average=None)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.json_schema_compliance(outputs, schema)

References

1. JSON Schema: a media type for describing JSON documents. Draft 2020-12. json-schema.org.

Implementation status