Agents and tool use
Unnecessary tool-call / loop rate
Implemented
agents.unnecessary_tool_call_rateDefinition
Share of tool calls that exactly repeat an earlier call (same tool and arguments) in the same task, and the share of tasks stuck in a loop (the same call repeated consecutively at least ``loop_length`` times).
Formula
repeated calls / calls
Range: [0, 1]
Inputs and outputs
- trajectories: see the signature of es.unnecessary_tool_call_rate
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.unnecessary_tool_call_rate(trajectories, loop_length=3)References
- Ma C, Zhang J, Zhu Z, et al. AgentBoard: an analytical evaluation board of multi-turn LLM agents. NeurIPS Datasets and Benchmarks. 2024.