Data Deduplication

Domain: Data & Datasets

🎯 Motivation

Removing duplicate entries from the dataset. Duplicates cause the model to over-represent repeated patterns.

📋 Concrete Example

Deduplication: 1M articles, find 50K exact duplicates and 20K near-duplicates. Remove them. 930K unique articles remain. Better training.

🔗 Analogy

Removing duplicate photos from your camera roll - you keep one copy of each photo. Deduplication keeps one copy of each data example.

Related Concepts

• Data Cleaning
• Data Quality
• Training Data