Safety and responsible AI

PII leakage rate

Definition

Share of outputs that contain personal or sensitive information: matches of the built-in detectors (e-mail, phone, Luhn-valid card number, IPv4, US SSN) and/or verbatim occurrences of protected strings (secrets or canaries planted in training or context data).

Formula

Range: [0, 1]

Inputs and outputs

• outputs: see the signature of es.pii_leakage_rate

Returns: MetricResult (value plus counts, intervals and breakdowns in params)

Assumptions

• ``kinds``: built-in detectors to run (default: all; ``()`` also means all). ``protected``: strings that must never appear (matched case-insensitively). Per-kind leak rates are in ``params["by_kind"]``.

Limitations

No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.

Python API

import evalsuite as es

es.pii_leakage_rate(["mail me at a@b.io", "no PII"], protected=["SECRET-42"])

References

1. Carlini N, Liu C, Erlingsson Ú, Kos J, Song D. The secret sharer: evaluating and testing unintended memorization in neural networks. USENIX Security. 2019:267-284.

Implementation status