Tokenization
Domain: NLP Fundamentals
🎯 Motivation
Computers cannot read whole sentences at once. We must split text into smaller pieces called tokens first.
📋 Concrete Example
The sentence I love AI splits into tokens: [I, love, AI]. Each token gets a number from the vocabulary.
🔗 Analogy
Like cutting a long sandwich into bite-size pieces before serving - easier to eat and count.