API reference

Everything is available from the top-level package: import evalsuite as es. Run help(es.f1) or es.metric_info("classification.f1") for full documentation of any function.

Python API

Common signature

metric(
  y_true,
  y_pred,                # or y_prob for score-based metrics
  *,
  average="auto",        # binary, micro, macro, weighted, samples, None
  labels=None,
  pos_label=None,
  sample_weight=None,
  zero_division="warn",  # or 0, 1, np.nan
)

Every metric returns a MetricResult that behaves like a number (float(r), f"{r:.3f}", comparisons) and records the parameters used.

HTTP API

The FastAPI service lives in backend/ of the website repository. It provides accounts, personal API keys and authenticated evaluation. Evaluation runs on the released EvalSuite package, and every response names the version in engine.label and engine.version. The API is hosted at https://evalsuite-api.onrender.com.

Authentication

1. Create an account on the website. You must be 18 or over; registration asks for your name, date of birth, role, country and intended use (organisation is optional).
2. Sign in and open the dashboard, then create your API key. Each account has one active key. It is shown once; only a hash is stored. To replace it, use Rotate key: the old key stops working immediately and a new one is issued.
3. Send it with every request, either as X-API-Key: <key> or Authorization: Bearer <key>.

Keys can be revoked instantly from the dashboard. A key can call evaluation endpoints but cannot create or revoke other keys.

curl https://evalsuite-api.onrender.com/api/v1/evaluate \
-H "X-API-Key: $EVALSUITE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "task": "binary-classification",
  "yTrue": [1, 0, 1, 1, 0],
  "yPred": [1, 0, 0, 1, 0],
  "yProb": [0.9, 0.2, 0.4, 0.8, 0.1],
  "metrics": ["classification.accuracy", "classification.roc_auc"],
  "confidence": {"method": "wilson", "level": 0.95, "nBootstrap": 1000, "randomState": 42}
}'

import os
import requests

response = requests.post(
  "https://evalsuite-api.onrender.com/api/v1/evaluate",
  headers={"X-API-Key": os.environ["EVALSUITE_API_KEY"]},
  json={
      "task": "regression",
      "yTrue": [3.1, 2.4, 5.0],
      "yPred": [2.9, 2.8, 4.6],
      "metrics": ["regression.mae", "regression.rmse"],
      "confidence": {"method": "none", "level": 0.95, "nBootstrap": 1000, "randomState": 0},
  },
  timeout=30,
)
response.raise_for_status()
print(response.json())

Endpoints

Sessions and passwords

Sessions are short-lived and tied to the password they were issued under: changing or resetting the password signs out every other session immediately. Reset links are single-use, expire after 30 minutes by default, and carry the token in the URL fragment so it never reaches server logs. The forgot-password endpoint answers identically whether or not an email is registered.

Limits and errors

Requests are limited to 1 MB, 5,000 observations and 2,000 bootstrap resamples, and evaluation is rate-limited per account (default 60 per minute; 429 with Retry-After). Every error has the shape {"error": {"code": "...", "message": "..."}}.

Data handling

Submitted data is processed in memory and never stored or logged. Only counters (time, endpoint, number of observations) are kept to show your usage. Do not send identifiable patient information.