Roadmap

Package releases

1. v0.1.0

   Implemented

   Core

   Released 8 October 2026. The foundation every later module plugs into: result types, validation, the metric registry, and the first two task families.

   Foundation

   • Result systemStatus: implemented
   • Input validationStatus: implemented
   • Metric registryStatus: implemented
   • Evaluation contextStatus: implemented

   Metrics

   • ClassificationStatus: implemented
   • RegressionStatus: implemented
   • Model comparisonStatus: implemented

   Research output

   • PlotsStatus: implemented
   • ReportingStatus: implemented
   • LaTeX exportStatus: implemented
   • CLIStatus: implemented

   Engineering

   • BenchmarksStatus: implemented
   • CI/CDStatus: implemented
   • PyPI release pipelineStatus: implemented

2. v0.2.0

   Implemented

   Clinical and statistics

   Released 8 October 2026. Diagnostic and calibration metrics, uncertainty quantification, and statistical testing.

   Clinical

   • Clinical metricsStatus: implemented
   • Calibration curve and ECEStatus: implemented
   • Hosmer–LemeshowStatus: implemented
   • Decision curve analysisStatus: implemented

   Uncertainty

   • Confidence intervalsStatus: implemented
   • Bootstrap (percentile, BCa)Status: implemented

   Statistics

   • Paired tests (McNemar, DeLong, paired bootstrap)Status: implemented
   • Further statistical testsStatus: implemented
   • Effect sizes (Cohen's d, Hedges' g, Cliff's delta, Cramér's V)Status: implemented
   • Multiple-testing correctionsStatus: implemented

3. v0.3.0

   Implemented

   Computer vision

   Released 9 October 2026. Segmentation and detection evaluation, with comparison, plotting and reporting extended to every task.

   Vision

   • Semantic segmentationStatus: implemented
   • Boundary and surface metricsStatus: implemented
   • Object detection (AP, mAP)Status: implemented

   Updated for all tasks

   • Model comparisonStatus: implemented
   • PlotsStatus: implemented
   • ReportingStatus: implemented

   Engineering

   • BenchmarksStatus: implemented
   • CI/CDStatus: implemented

4. v0.4.0

   Implemented

   LLM evaluation

   Released 9 October 2026. Text generation, semantic similarity, factuality and hallucination, LLM-as-a-judge, reasoning benchmarks, retrieval-augmented generation and structured output, with the same intervals, comparison, plots and reports as every other task.

   Text generation and language quality

   • BLEUStatus: implemented
   • ROUGE-1 / ROUGE-2 / ROUGE-L / ROUGE-LsumStatus: implemented
   • METEORStatus: implemented
   • chrF / chrF++Status: implemented
   • TERStatus: implemented
   • CIDEr-DStatus: implemented
   • Perplexity (PPL)Status: implemented
   • Cross-entropy / Negative log-likelihoodStatus: implemented
   • Distinct-1 / Distinct-2Status: implemented
   • Self-BLEUStatus: implemented
   • MAUVEStatus: implemented
   • SPICE (needs a Java scene-graph parser)Status: planned

   Semantic similarity and learned text metrics

   • BERTScore Precision / Recall / F1Status: implemented
   • Embedding cosine similarityStatus: implemented
   • Embedding Euclidean / Manhattan distanceStatus: implemented
   • MoverScoreStatus: implemented
   • BLEURT (via model_score)Status: implemented
   • COMET (via model_score)Status: implemented
   • BARTScore (via model_score)Status: implemented
   • AlignScore (via model_score)Status: implemented
   • MTEB task metrics (retrieval, classification and similarity metrics)Status: implemented

   Factuality, correctness, and hallucination

   • Exact Match (EM)Status: implemented
   • Token-level Precision / Recall / F1Status: implemented
   • Answer correctnessStatus: implemented
   • Factual consistency / faithfulnessStatus: implemented
   • GroundednessStatus: implemented
   • Hallucination / unsupported-claim rateStatus: implemented
   • Citation precision / recall / correctnessStatus: implemented
   • Claim verification accuracyStatus: implemented
   • Knowledge consistencyStatus: implemented
   • Answer relevanceStatus: implemented
   • Abstention accuracyStatus: implemented

   LLM-as-a-judge and preference evaluation

   • Correctness, helpfulness, relevance, coherence, fluencyStatus: implemented
   • Completeness, clarity, conciseness, readabilityStatus: implemented
   • Instruction-following / context-adherence scoreStatus: implemented
   • Reasoning-quality scoreStatus: implemented
   • Tone and style adherenceStatus: implemented
   • Pairwise preference win rateStatus: implemented
   • Elo / Bradley–Terry scoresStatus: implemented
   • Inter-judge agreement / human agreementStatus: implemented
   • Judge calibration and bias sensitivityStatus: implemented

   Reasoning, knowledge, and mathematical ability

   • Accuracy / Exact MatchStatus: implemented
   • Pass@1 / Pass@kStatus: implemented
   • Majority-vote accuracyStatus: implemented
   • GSM8K / MATH benchmark accuracyStatus: implemented
   • GPQA / MMLU / BBH / ARC accuracyStatus: implemented
   • TruthfulQA scoreStatus: implemented
   • Constraint satisfaction rateStatus: implemented
   • Tool-assisted task successStatus: implemented

   Retrieval-augmented generation (RAG)

   • Precision@k / Recall@k / Hit Rate@kStatus: implemented
   • MRRStatus: implemented
   • MAPStatus: implemented
   • NDCG@kStatus: implemented
   • Retrieval latencyStatus: implemented
   • Context precision / recall / relevanceStatus: implemented
   • Faithfulness / groundednessStatus: implemented
   • Answer correctness / relevancy / completenessStatus: implemented
   • Citation precision / recallStatus: implemented
   • Unsupported-claim rateStatus: implemented
   • End-to-end task success / query resolutionStatus: implemented
   • Failure attributionStatus: implemented

   Instruction following and structured output

   • Instruction compliance rateStatus: implemented
   • Constraint satisfaction rateStatus: implemented
   • Required-field accuracyStatus: implemented
   • JSON validity / JSON Schema complianceStatus: implemented
   • XML validityStatus: implemented
   • Function-call / tool-selection accuracyStatus: implemented
   • Tool-argument accuracyStatus: implemented
   • API-call success rateStatus: implemented
   • Format / refusal-format complianceStatus: implemented
   • Multi-turn instruction retentionStatus: implemented
   • Unwanted extra-content rateStatus: implemented

5. v0.5.0

   Planned

   LLM systems: safety, agents and operations

   Planned. Safety and responsible AI, robustness, calibration and uncertainty for LLMs, agent and tool use, multilingual, code generation, long-context and summarization, and inference efficiency and cost.

   8 areas, 85 metrics planned

   Safety, security, and responsible AI

   • Harmful response rate / unsafe compliance rateStatus: planned
   • Refusal rate / appropriate refusal rateStatus: planned
   • Over-refusal rateStatus: planned
   • Jailbreak / prompt-injection attack success rateStatus: planned
   • Toxicity / hate-speech / harassment scoresStatus: planned
   • Bias and stereotype association scoresStatus: planned
   • Sensitive-information / PII leakage rateStatus: planned
   • Memorization exposureStatus: planned
   • Policy violation rateStatus: planned
   • Red-team attack success rateStatus: planned

   Robustness and reliability

   • Adversarial robustnessStatus: planned
   • Paraphrase consistencyStatus: planned
   • Typographical-noise robustnessStatus: planned
   • Out-of-distribution accuracyStatus: planned
   • Distribution-shift performance dropStatus: planned
   • Counterfactual consistency / invariance violation rateStatus: planned
   • Contradiction rate / response stabilityStatus: planned
   • Failure / timeout / error rateStatus: planned
   • Recovery success rateStatus: planned
   • Prompt wording sensitivityStatus: planned
   • Long-context robustness / truncation sensitivityStatus: planned

   Calibration and uncertainty

   • Expected Calibration Error (ECE)Status: planned
   • Adaptive Calibration Error (ACE)Status: planned
   • Maximum Calibration Error (MCE)Status: planned
   • Brier scoreStatus: planned
   • Log loss / Negative log-likelihoodStatus: planned
   • Selective risk / coverage-risk curveStatus: planned
   • Area under the risk-coverage curve (AURC)Status: planned
   • Risk at fixed coverage / coverage at fixed riskStatus: planned
   • Confidence-accuracy correlationStatus: planned
   • Abstention qualityStatus: planned

   Agent and tool-use evaluation

   • Task / goal completion rateStatus: planned
   • Tool-call precision / recallStatus: planned
   • Tool-selection / argument correctnessStatus: planned
   • Invalid tool-call / execution failure rateStatus: planned
   • Tool-use efficiencyStatus: planned
   • Steps / tool calls per taskStatus: planned
   • Planning accuracy / plan adherenceStatus: planned
   • State-tracking accuracyStatus: planned
   • Recovery from tool failuresStatus: planned
   • Unnecessary tool-call / loop rateStatus: planned
   • Human intervention rateStatus: planned
   • End-to-end execution time / cost per taskStatus: planned

   Multilingual and cross-lingual evaluation

   • Language identification accuracyStatus: planned
   • Translation BLEU / chrF / COMETStatus: planned
   • Multilingual semantic similarityStatus: planned
   • Cross-lingual retrieval Recall@k / MRRStatus: planned
   • Language-specific accuracy / performance parityStatus: planned
   • Code-switching robustnessStatus: planned
   • Translation adequacy / fluencyStatus: planned
   • Cultural appropriatenessStatus: planned
   • Language consistency / cross-lingual factual consistencyStatus: planned

   Code generation and software engineering

   • Pass@kStatus: planned
   • Unit-test pass rateStatus: planned
   • Compilation / syntax validity rateStatus: planned
   • Static-analysis violation rateStatus: planned
   • Code execution success / functional correctnessStatus: planned
   • Generated-code test coverageStatus: planned
   • Bug reproduction rate / patch acceptance rateStatus: planned
   • Repository task success / SWE-bench resolved rateStatus: planned
   • CodeBLEUStatus: planned
   • Cyclomatic complexity / maintainability indexStatus: planned
   • Security vulnerability rateStatus: planned
   • Runtime efficiency / memory consumptionStatus: planned

   Long-context and summarization evaluation

   • Long-context retrieval accuracyStatus: planned
   • Needle-in-a-haystack accuracyStatus: planned
   • Position-dependent retrieval accuracyStatus: planned
   • Context utilization / retentionStatus: planned
   • Summary factual consistency / coverage / completenessStatus: planned
   • Summary compression ratioStatus: planned
   • ROUGE-L / BERTScoreStatus: planned
   • Citation coverageStatus: planned
   • Lost-in-the-middle sensitivityStatus: planned
   • Truncation robustness / cross-document consistencyStatus: planned

   Inference efficiency and operational cost

   • Time to First Token (TTFT)Status: planned
   • Time per Output Token (TPOT)Status: planned
   • End-to-end latencyStatus: planned
   • Throughput / tokens per secondStatus: planned
   • Input / output token countsStatus: planned
   • Cost per request / per 1,000 tokensStatus: planned
   • Peak memory / CPU / GPU utilizationStatus: planned
   • Energy per requestStatus: planned
   • Requests per secondStatus: planned
   • Error rate / availabilityStatus: planned
   • p50 / p95 / p99 latencyStatus: planned