Code generation

Compilation / syntax validity rate

Definition

Share of generated programs that parse (Python: compiled with ``compile`` in 'exec' mode, never run). Other languages: pass a ``checker`` that returns True when the code compiles.

Formula

Range: [0, 1]

Inputs and outputs

• code: see the signature of es.syntax_validity_rate

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.syntax_validity_rate(["x = 1", "def (:"])  # parsed, never run

References

1. Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. arXiv:2107.03374. 2021.

Implementation status