Information Theory
Domain: Math & Logic for AI
🎯 Motivation
The study of quantifying information, uncertainty, and communication. Foundation of modern ML loss functions.
📋 Concrete Example
Shannon entropy measures the average bits needed to encode messages from a source.
🔗 Analogy
Like a postal pricing system - it costs more bits to send surprising news than predictable news.