user@devops:~$ cat README.md
Knowledge Distillation — Teacher → Student
# Description
Knowledge Distillation (KD, Hinton et al. 2015) implemented from scratch in PyTorch on Fashion-MNIST (6,000 train / 2,000 test, seed 42). A teacher CNN with 824,650 parameters (Conv 32→64ch, BatchNorm, Dropout, FC 256) reaches 88.95% test accuracy and transfers its knowledge to a student CNN with 105,866 parameters (Conv 16→32ch, 7.8× smaller) via softened labels: loss = α·CE(student, y) + (1−α)·T²·KL(softmax(z_s/T) ‖ softmax(z_t/T)) with T=4, α=0.7, using precomputed teacher logits (no teacher forward pass during student training). Results: student with hard labels 82.85% vs student with KD 83.85% — KD recovers +1.00 pp with the same architecture and keeps 94.3% of the teacher's accuracy. The advantage concentrates where it matters most: Coat +11.4 pp (52.1→63.5%) and Shirt +6.1 pp (68.0→74.1%), while on easy classes (Trouser, Sandal) both tie. Dark knowledge is measurable: the KD student's mean test softmax entropy (1.306) reproduces the teacher's soft labels at T=4 (1.401), while the hard student is overconfident (0.514 vs teacher 0.238). Sweeps: temperature T∈{1,2,4,8,16} (best T=1 with short training; the classic T=4 regime wins with longer training) and alpha α∈{0,0.3,0.5,0.7,0.9,1.0} (best α=0.9; α=0, soft labels only, collapses to 10.15% ≈ chance — the hard label is essential). 8 visualizations: teacher curves, soft labels at different T, hard vs KD, T and α sweeps, params vs accuracy, per-class accuracy and entropy distribution.
# Key features
$ Knowledge Distillation (Hinton et al. 2015) from scratch in PyTorch: softened labels with temperature T and loss α·CE + (1−α)·T²·KL
$ Teacher CNN 824,650 params (32→64ch, BN, Dropout) at 88.95% test; student CNN 105,866 params (7.8× smaller) with KD at 83.85%
$ KD recovers +1.00 pp over the same architecture trained with hard labels (82.85→83.85%): 94.3% of teacher accuracy with 7.8× fewer params
$ Dark knowledge concentrates on the hard classes: Coat +11.4 pp (52.1→63.5%) and Shirt +6.1 pp (68.0→74.1%)
$ Transferred calibration: mean softmax entropy on test — teacher 0.238 · hard 0.514 (overconfident) · KD 1.306 ≈ teacher soft labels at T=4 (1.401)
$ Sweeps: temperature T∈{1,2,4,8,16} and alpha α∈{0,0.3,…,1.0}; α=0 (soft labels only) collapses to 10.15% ≈ chance
$ Precomputed teacher logits: deterministic training with seed 42, no teacher forward pass during student training
$ 8 visualizations, saved models (artifacts/) and reproducible ml_distill_history.json
# Gallery
# Technologies used
