xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 006-inference-sampling.md

1.7 KB
1# Inference and sampling from a checkpoint
2
3This note explains the separate sampling script:
4
5```text
6src/lua_code_model/sample.py
7```
8
9## Training vs inference
10
11Training means:
12
13```text
14load data -> predict next tokens -> compute loss -> update weights -> save checkpoint
15```
16
17Inference/sampling means:
18
19```text
20load checkpoint -> encode prompt -> predict next token -> sample -> repeat
21```
22
23During inference, the model weights are not updated. We call `model.eval()` and use `@torch.no_grad()` so PyTorch does not track gradients.
24
25## Basic command
26
27From the project root:
28
29```bash
30PYTHONPATH=src uv run python -m lua_code_model.sample
31```
32
33By default, this loads:
34
35```text
36checkpoints/gpt_tiny.pt
37```
38
39and continues the prompt:
40
41```text
42local function
43```
44
45## Useful options
46
47```bash
48PYTHONPATH=src uv run python -m lua_code_model.sample \
49  --checkpoint checkpoints/gpt_tiny.pt \
50  --prompt "local function" \
51  --max-new-tokens 300 \
52  --temperature 0.8 \
53  --seed 1337
54```
55
56The same script can load all current checkpoint types:
57
58```text
59checkpoints/bigram_tiny.pt
60checkpoints/mlp_tiny.pt
61checkpoints/gpt_tiny.pt
62```
63
64## Temperature reminder
65
66Temperature changes how random sampling is:
67
68- lower temperature: safer, more repetitive, more likely to choose high-probability characters
69- higher temperature: more surprising, more chaotic, more likely to choose low-probability characters
70
71The model produces logits. During sampling, the script uses:
72
73```text
74softmax(logits / temperature)
75```
76
77then samples one next character from that probability distribution.
78
79## Vocabulary limitation
80
81The prompt must use only characters that existed in the training corpus. This is because the current tokenizer is character-level and has no unknown-token fallback yet.