Key takeaway: Inter-annotator agreement is an upper bound on achievable model accuracy. Measuring it first tells you whether your problem is the model or the labels.
The Ceiling Nobody Measures
A team trains a classifier, reaches 82% accuracy, and starts trying larger architectures and longer schedules. Progress stalls.
Nobody measured whether two humans labelling the same examples agree more than 82% of the time. If they agree 84% of the time, the model has essentially learned everything the labels contain, and every further architecture change is noise.
Contradictory labels are not merely unhelpful — they are actively harmful. The model receives gradient updates pushing in opposite directions for indistinguishable inputs, which degrades confidence calibration and produces erratic behaviour near boundaries.
Where Disagreement Originates
Ambiguous guidelines. “Label vehicles” leaves bicycles, trailers, parked versus moving, and partially occluded objects undefined. Each annotator resolves these differently and consistently, producing systematic conflict.
Boundary conventions. For bounding boxes: tight to visible pixels or including the occluded extent? For segmentation: how are shadows and reflections handled? These decisions change results substantially and are usually unstated.
Unrepresented cases. Guidelines written from an initial sample encounter situations nobody anticipated. Without a defined escalation path, annotators guess.
Fatigue and drift. Quality declines within long sessions, and conventions shift across weeks as annotators develop personal habits.
Measuring It Properly
Assign the same 200 examples to three annotators independently and compute agreement. Use Cohen’s kappa for categorical labels or IoU distribution for boxes and masks.
| Kappa | Interpretation |
|---|---|
| Below 0.4 | Guidelines are broken; stop labelling |
| 0.4 – 0.6 | Significant ambiguity to resolve |
| 0.6 – 0.8 | Workable, refine edge cases |
| Above 0.8 | Solid foundation |
Then examine the disagreements individually. They cluster, and each cluster identifies one guideline gap that can be fixed with a written rule and an example image. Resolving five clusters often moves kappa more than any amount of additional labelling.
Building the Feedback Loop
Guidelines should be a living document with a decision log. Every ambiguous case that arises gets a ruling, an example, and a date. New annotators read the log rather than rediscovering the same questions.
Keep a permanent gold set of expert-labelled examples mixed into every batch. It measures ongoing quality without additional review effort and catches drift as it happens rather than at the end.
Route genuinely ambiguous items to a queue rather than forcing a decision. An annotator guessing under throughput pressure injects noise; an annotator flagging uncertainty generates a guideline improvement.
Cleaning What You Have
Existing datasets can be improved without full relabelling. Train a model, then inspect examples where it is confidently wrong — these are disproportionately mislabelled rather than difficult. Relabelling that subset frequently improves accuracy more than doubling the dataset.
The Bottom Line
Measure inter-annotator agreement before optimising the model. If kappa is below 0.6, fix guidelines rather than architecture. Maintain a gold set for continuous monitoring, and use confident model errors to find label noise cheaply.
