Subword Tokenization
Domain: NLP Fundamentals
🎯 Motivation
Splits text into pieces smaller than words but bigger than characters. Handles rare and new words well.
📋 Concrete Example
Unbelievable might split into [Un, believe, able]. ChatGPT splits text into subword units like this.
🔗 Analogy
Like breaking a big Lego build into medium modules - flexible enough to build anything, efficient to store.