Statistical tests
Paired t-test
Implemented
statistics.t_test_pairedDefinition
Tests whether the mean of paired differences is zero, for example per-fold scores of two models.
Formula
t = d̄ / (s_d / √n)
Inputs and outputs
- paired sample a
- paired sample b
Returns: StatisticalTestResult
Assumptions
- Differences are approximately normal.
Limitations
- Cross-validation folds are not independent; the test can be anti-conservative for model comparison.
Python API
import evalsuite as es
es.paired_t_test(scores_a, scores_b)References
- Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25.