768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 006-inference-sampling.md
1.7 KB
1# Inference and sampling from a checkpoint 2 3This note explains the separate sampling script: 4 5```text 6src/lua_code_model/sample.py 7``` 8 9## Training vs inference 10 11Training means: 12 13```text 14load data -> predict next tokens -> compute loss -> update weights -> save checkpoint 15``` 16 17Inference/sampling means: 18 19```text 20load checkpoint -> encode prompt -> predict next token -> sample -> repeat 21``` 22 23During inference, the model weights are not updated. We call `model.eval()` and use `@torch.no_grad()` so PyTorch does not track gradients. 24 25## Basic command 26 27From the project root: 28 29```bash 30PYTHONPATH=src uv run python -m lua_code_model.sample 31``` 32 33By default, this loads: 34 35```text 36checkpoints/gpt_tiny.pt 37``` 38 39and continues the prompt: 40 41```text 42local function 43``` 44 45## Useful options 46 47```bash 48PYTHONPATH=src uv run python -m lua_code_model.sample \ 49 --checkpoint checkpoints/gpt_tiny.pt \ 50 --prompt "local function" \ 51 --max-new-tokens 300 \ 52 --temperature 0.8 \ 53 --seed 1337 54``` 55 56The same script can load all current checkpoint types: 57 58```text 59checkpoints/bigram_tiny.pt 60checkpoints/mlp_tiny.pt 61checkpoints/gpt_tiny.pt 62``` 63 64## Temperature reminder 65 66Temperature changes how random sampling is: 67 68- lower temperature: safer, more repetitive, more likely to choose high-probability characters 69- higher temperature: more surprising, more chaotic, more likely to choose low-probability characters 70 71The model produces logits. During sampling, the script uses: 72 73```text 74softmax(logits / temperature) 75``` 76 77then samples one next character from that probability distribution. 78 79## Vocabulary limitation 80 81The prompt must use only characters that existed in the training corpus. This is because the current tokenizer is character-level and has no unknown-token fallback yet.