Skip to content
EvalSuite
Documentation menu

Agents and tool use

Tool-use efficiency

Implementedagents.tool_use_efficiency

Definition

How close the number of tool calls is to the minimum needed: the mean of optimal / actual calls per task (1 = no wasted calls), with the mean excess calls.

Formula

mean_t min(1, optimal_t / actual_t)

Range: [0, 1]

Inputs and outputs

  • n_calls: see the signature of es.tool_use_efficiency
  • optimal_calls: see the signature of es.tool_use_efficiency

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

PythonSince v0.5.0
import evalsuite as es

es.tool_use_efficiency(n_calls=[4, 2], optimal_calls=[2, 2])

References

  1. Ma C, Zhang J, Zhu Z, et al. AgentBoard: an analytical evaluation board of multi-turn LLM agents. NeurIPS Datasets and Benchmarks. 2024.

Implementation status