Knowledge Distillation
Domain: MLOps
🎯 Motivation
A small student model learns from a large teacher model. Student is small but inherits teacher’s knowledge.
📋 Concrete Example
Teacher: 175B parameters, 92% accuracy. Student: 7B parameters, 85% accuracy. Student learned from teacher’s outputs. Small, fast, decent accuracy.
🔗 Analogy
A master teaching an apprentice - the master (teacher) has deep knowledge. The apprentice (student) learns from the master. Smaller but capable.