Skip to content
EvalSuite
Documentation menu

Agents and tool use

State-tracking accuracy

Implementedagents.state_tracking_accuracy

Definition

Dialogue / environment state tracking: joint goal accuracy (every slot of the state correct at a turn) and slot accuracy (share of slots correct over the union of expected and predicted slots).

Formula

JGA = mean_turns 1[state = expected]

Range: [0, 1]

Inputs and outputs

  • expected_states: see the signature of es.state_tracking_accuracy
  • predicted_states: see the signature of es.state_tracking_accuracy

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

No assumptions beyond valid, aligned inputs of the documented types.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

PythonSince v0.5.0
import evalsuite as es

es.state_tracking_accuracy([{"area": "north"}], [{"area": "north"}])

References

  1. Mrkšić N, Ó Séaghdha D, Wen TH, Thomson B, Young S. Neural belief tracker: data-driven dialogue state tracking. ACL. 2017:1777-1788.

Implementation status