768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 007-distillation-future-experiment.md
2.4 KB
1# Future experiment: distilling a Lua model 2 3This is a note for a future learning experiment we want to do after the basic Lua GPT pipeline is comfortable. 4 5## Big idea 6 7Distillation means training a smaller **student** model to imitate a stronger **teacher** model. 8 9For this project, a fun version would be: 10 11```text 12larger Lua GPT teacher -> smaller Lua GPT student 13``` 14 15The goal is not to build a production model. The goal is to learn what extra information a teacher can provide and whether a small model can become better by copying a bigger one. 16 17## Why this is interesting 18 19Normal next-token training only tells the model the one token that actually appeared in the dataset. 20 21Example: 22 23```lua 24local x = 25``` 26 27The dataset may say the next token is `1`, but other continuations could also make sense: 28 29```lua 30nil 31{} 32function() 33"hello" 34``` 35 36A teacher model can provide a softer probability distribution over many possible next tokens. That can teach the student about what is plausible, not just what happened once. 37 38## Possible experiment plan 39 401. Train a slightly larger character-level Lua GPT as the teacher. 412. Train a smaller Lua GPT normally as a baseline student. 423. Train another same-size student with an additional distillation loss. 434. Compare the normal student against the distilled student. 44 45Things to compare: 46 47- validation loss 48- perplexity 49- generated Lua-like samples 50- model size 51- training speed 52- inference speed 53- whether syntax looks more coherent 54 55## Conceptual loss 56 57The student can learn from two signals: 58 59```text 60student_loss = real_dataset_loss + teacher_imitation_loss 61``` 62 63Where: 64 65- `real_dataset_loss` teaches the student to predict the real corpus text 66- `teacher_imitation_loss` teaches the student to match the teacher's next-token probabilities 67 68## Questions to explore 69 70- Does distillation help a very small model learn Lua syntax faster? 71- Does it improve validation loss, or only generation quality? 72- Does a student copy the teacher's mistakes? 73- How much bigger does the teacher need to be before distillation helps? 74- Is it better to use only teacher probabilities, only real labels, or a mixture? 75- What happens if the teacher is overfit to the tiny corpus? 76 77## Keep in mind 78 79Distillation is not magic. A tiny student still has limited capacity. The experiment is valuable even if the result is "no improvement", because we will learn how teacher/student training works in practice.