Inter-Annotator Agreement
Domain: Data & Datasets
🎯 Motivation
How often different annotators agree on labels. High agreement = clear guidelines and consistent labels.
📋 Concrete Example
5 annotators label 1000 texts. Agreement: 92%. High - labels are consistent. If agreement was 60%, guidelines need improvement.
🔗 Analogy
Judges at a diving competition - if all judges give similar scores (high agreement), the scoring is consistent. Inter-annotator agreement measures labeling consistency.