Object detection

Ground truth and predictions are one dict per image. Boxes are xyxy by default (box_format="xywh" or "cxcywh" for others); ground truth may carry iscrowd and area.

import evalsuite as es

y_true = [{"boxes": [[10, 10, 50, 50], [60, 60, 100, 100]], "labels": [1, 2]}]
y_pred = [{"boxes": [[12, 11, 49, 52], [58, 61, 99, 102], [0, 0, 5, 5]],
         "labels": [1, 2, 1], "scores": [0.9, 0.8, 0.3]}]

report = es.detection_report(y_true, y_pred)   # the 12 COCO numbers + AP per class
report["map"], report["map_50"], report["mar_100"]   # 0.80, 1.00, 0.80
print(report)
report.save("detection.html")                  # also .csv, .md, .tex, .json

es.mean_average_precision(y_true, y_pred, iou_threshold=0.5)          # mAP@.50
es.average_precision_detection(y_true, y_pred, interpolation="voc")   # per class, VOC style
es.box_iou(boxes_a, boxes_b, box_format="xywh")
y_true, y_pred = es.from_coco("instances_val.json", "detections.json")
es.plot.detection_pr(y_true, y_pred)

Evaluation convention

mAP@[.50:.95] averages AP over the ten IoU thresholds 0.50, 0.55, …, 0.95 and over classes. EvalSuite follows the COCO protocol exactly: greedy matching by score, crowd regions that absorb matches without counting as false positives, small, medium and large area ranges, 1, 10 and 100 maximum detections, and 101-point interpolated precision. All twelve numbers match pycocotools in the test suite. interpolation="voc" gives PASCAL VOC all-point AP instead.

Comparison and command line

es.compare(y_true, {"yolo": preds_a, "detr": preds_b}) resamples images and compares mAP with paired bootstrap intervals and tests. From the command line: evalsuite detection instances_val.json detections.json --plot pr.png.

Metrics