Data Pipeline
Domain: Data & Datasets
🎯 Motivation
The end-to-end process: collect -> clean -> preprocess -> augment -> split -> train -> evaluate. Automated for efficiency.
📋 Concrete Example
Pipeline: 1) Scrape 1M articles. 2) Clean (remove duplicates, fix encoding). 3) Tokenize. 4) Split (80/10/10). 5) Train model. 6) Evaluate. Automated end-to-end.
🔗 Analogy
A water treatment pipeline - water flows through: collection -> filtration -> chemical treatment -> testing -> distribution. Each stage processes the water. A data pipeline processes data.