Agents and tool use

Task / goal completion rate

Definition

Share of tasks the agent completes (its goal state reached or judged successful), with the mean progress rate when partial progress is scored (AgentBoard).

Formula

Range: [0, 1]

Inputs and outputs

• completed: see the signature of es.task_completion_rate

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.task_completion_rate([True, False, True], progress=[1, 0.5, 1])

References

1. Liu X, Yu H, Zhang H, et al. AgentBench: evaluating LLMs as agents. ICLR. 2024.
2. Ma C, Zhang J, Zhu Z, et al. AgentBoard: an analytical evaluation board of multi-turn LLM agents. NeurIPS Datasets and Benchmarks. 2024.

Implementation status