xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 007-distillation-future-experiment.md

2.4 KB
1# Future experiment: distilling a Lua model
2
3This is a note for a future learning experiment we want to do after the basic Lua GPT pipeline is comfortable.
4
5## Big idea
6
7Distillation means training a smaller **student** model to imitate a stronger **teacher** model.
8
9For this project, a fun version would be:
10
11```text
12larger Lua GPT teacher -> smaller Lua GPT student
13```
14
15The goal is not to build a production model. The goal is to learn what extra information a teacher can provide and whether a small model can become better by copying a bigger one.
16
17## Why this is interesting
18
19Normal next-token training only tells the model the one token that actually appeared in the dataset.
20
21Example:
22
23```lua
24local x =
25```
26
27The dataset may say the next token is `1`, but other continuations could also make sense:
28
29```lua
30nil
31{}
32function()
33"hello"
34```
35
36A teacher model can provide a softer probability distribution over many possible next tokens. That can teach the student about what is plausible, not just what happened once.
37
38## Possible experiment plan
39
401. Train a slightly larger character-level Lua GPT as the teacher.
412. Train a smaller Lua GPT normally as a baseline student.
423. Train another same-size student with an additional distillation loss.
434. Compare the normal student against the distilled student.
44
45Things to compare:
46
47- validation loss
48- perplexity
49- generated Lua-like samples
50- model size
51- training speed
52- inference speed
53- whether syntax looks more coherent
54
55## Conceptual loss
56
57The student can learn from two signals:
58
59```text
60student_loss = real_dataset_loss + teacher_imitation_loss
61```
62
63Where:
64
65- `real_dataset_loss` teaches the student to predict the real corpus text
66- `teacher_imitation_loss` teaches the student to match the teacher's next-token probabilities
67
68## Questions to explore
69
70- Does distillation help a very small model learn Lua syntax faster?
71- Does it improve validation loss, or only generation quality?
72- Does a student copy the teacher's mistakes?
73- How much bigger does the teacher need to be before distillation helps?
74- Is it better to use only teacher probabilities, only real labels, or a mixture?
75- What happens if the teacher is overfit to the tiny corpus?
76
77## Keep in mind
78
79Distillation is not magic. A tiny student still has limited capacity. The experiment is valuable even if the result is "no improvement", because we will learn how teacher/student training works in practice.