Agents and tool use

End-to-end execution time / cost per task

Definition

Mean wall-clock time and monetary cost per task, and the cost per successful task (total cost divided by completed tasks), the figure that matters when failures are retried.

Formula

Range: [0, ∞)

Inputs and outputs

• costs: see the signature of es.agent_cost_per_task

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.agent_cost_per_task([0.02, 0.05], durations=[30, 80], completed=[True, False])

References

1. Liu X, Yu H, Zhang H, et al. AgentBench: evaluating LLMs as agents. ICLR. 2024.

Implementation status