Word Tokenization
Domain: NLP Fundamentals
🎯 Motivation
Splitting text by spaces and punctuation into individual words. The simplest tokenization method.
📋 Concrete Example
Hello, world! becomes [Hello, ,, world, !]. Punctuation marks are often separate tokens.
🔗 Analogy
Like tearing a sentence into individual word cards, one word per card.