768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 005-session-handoff.md
5.9 KB
1# Session handoff: tiny Lua language model 2 3This note captures enough context to continue the project in a future session without rereading the full chat. 4 5## Project intent 6 7This is an educational language-model project for Lua code/documentation continuation experiments. 8 9The goal is learning the pieces of a GPT-style language model, not building a useful coding assistant. 10 11Current implementation is character-level, local-first, small, and intentionally readable. 12 13## Environment and dependency state 14 15- Package manager: `uv` 16- Python project file: `pyproject.toml` 17- Third-party dependency added so far: `torch` 18- Do not add new dependencies without explicit user approval. 19 20Useful commands: 21 22```bash 23uv sync 24PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus 25PYTHONPATH=src uv run python -m lua_code_model.train_bigram 26PYTHONPATH=src uv run python -m lua_code_model.train_mlp 27PYTHONPATH=src uv run python -m lua_code_model.train_gpt 28PYTHONPATH=src uv run python -m lua_code_model.sample 29``` 30 31PyTorch may print a warning about NumPy not being installed. The current scripts do not need NumPy, so we intentionally did not add it. 32 33## Data pipeline 34 35Raw input: 36 37```text 38data/raw/tiny_lua_corpus.txt 39``` 40 41Source tracking: 42 43```text 44data/SOURCES.md 45``` 46 47Preparation script: 48 49```text 50src/lua_code_model/prepare_tiny_corpus.py 51``` 52 53Generated outputs: 54 55```text 56data/processed/tiny_train.txt 57data/processed/tiny_val.txt 58data/processed/tiny_train_ids.txt 59data/processed/tiny_val_ids.txt 60data/processed/tiny_vocab.json 61data/processed/tiny_metadata.json 62``` 63 64The current tokenizer is character-level: 65 66```text 67text -> characters -> integer token IDs 68``` 69 70The current split is 90% train / 10% validation. 71 72Last observed prep stats: 73 74```text 75raw characters: 1008 76vocab size: 51 77train tokens: 907 78validation tokens: 101 79``` 80 81## Models implemented 82 83### 1. Bigram model 84 85File: 86 87```text 88src/lua_code_model/train_bigram.py 89``` 90 91Checkpoint: 92 93```text 94checkpoints/bigram_tiny.pt 95``` 96 97Concept: 98 99```text 100current character -> next character 101``` 102 103Purpose: teach tensors, logits, loss, gradients, optimizer steps, checkpointing, and simple generation. 104 105### 2. Fixed-context MLP 106 107File: 108 109```text 110src/lua_code_model/train_mlp.py 111``` 112 113Checkpoint: 114 115```text 116checkpoints/mlp_tiny.pt 117``` 118 119Concept: 120 121```text 122previous 32 characters -> embeddings -> flatten -> hidden layer -> logits -> next character 123``` 124 125Purpose: teach embeddings, hidden layers, ReLU activation, fixed context windows, temperature during generation, and overfitting. 126 127Last observed behavior: it memorized the tiny training corpus very strongly. 128 129### 3. Tiny GPT-style Transformer 130 131File: 132 133```text 134src/lua_code_model/train_gpt.py 135``` 136 137Checkpoint: 138 139```text 140checkpoints/gpt_tiny.pt 141``` 142 143Concept: 144 145```text 146token IDs -> token embeddings + position embeddings -> Transformer blocks -> logits 147``` 148 149Implemented GPT concepts: 150 151- causal self-attention 152- query/key/value projections 153- causal mask using a lower-triangular matrix 154- multi-head attention 155- feed-forward MLP inside each Transformer block 156- residual connections 157- LayerNorm 158- autoregressive generation 159 160Tiny GPT settings at last run: 161 162```python 163BATCH_SIZE = 16 164BLOCK_SIZE = 64 165MAX_STEPS = 2_000 166LEARNING_RATE = 1e-3 167N_EMBD = 64 168N_HEAD = 4 169N_LAYER = 2 170DROPOUT = 0.0 171SAMPLE_TEMPERATURE = 0.8 172``` 173 174Last observed GPT stats: 175 176```text 177train tokens: 907 178val tokens: 101 179vocab size: 51 180parameters: 110387 181step 0 train loss: 4.0653 182step 2000 train loss: 0.0829 183step 2000 val loss: 6.3499 184``` 185 186Interpretation: the model learned/memorized the tiny training data and overfit, which is expected at this stage. 187 188## Learning concepts already discussed 189 190The user has a working understanding of: 191 192- why we track data sources/licenses 193- character tokenization 194- train/validation split 195- tensors as numeric arrays 196- logits as raw prediction scores 197- softmax converting logits to probabilities 198- cross-entropy loss as model surprise/error signal 199- gradients from `loss.backward()` 200- optimizers using gradients to update parameters 201- checkpoints as saved model weights/metadata 202- training steps vs epochs 203- activation functions, especially ReLU 204- hidden layers as internal learned representation spaces 205- temperature as `softmax(logits / temperature)` during sampling 206- overfitting as low train loss but poor validation performance 207- broad Transformer/GPT concepts 208- GPT vs BERT vs BART-style model families 209 210## Important observed behavior 211 212Repeated or malformed generated strings from tiny GPT, such as repeated letters, are considered symptoms rather than obvious bugs. 213 214Likely causes: 215 216- character-level generation is fragile 217- corpus is extremely small 218- model is overfitting and partially recalling phrases 219- probabilistic sampling with temperature can choose imperfect next characters 220- no top-k/top-p/repetition-penalty decoding yet 221 222## Current notes files 223 224```text 225notes/001-first-dataset.md 226notes/002-bigram-pytorch.md 227notes/003-fixed-context-mlp.md 228notes/004-tiny-gpt.md 229notes/005-session-handoff.md 230``` 231 232## Sampling script added 233 234The project now has a separate inference/sampling script: 235 236```text 237src/lua_code_model/sample.py 238``` 239 240It can: 241 242- choose a checkpoint path with `--checkpoint` 243- load saved vocabulary and model weights 244- accept a prompt with `--prompt` 245- expose generation settings such as `--temperature`, `--max-new-tokens`, and `--seed` 246- generate text without running training again 247 248This reinforces the distinction between: 249 250```text 251training: update model weights 252inference/sampling: use saved weights to generate text 253``` 254 255## Later possible milestones 256 2571. Refactor duplicated helper code across training and sampling scripts. 2582. Add top-k sampling to make generation controls more visible. 2593. Add a small amount of real, legally usable Lua documentation/source data after recording it in `data/SOURCES.md`. 2604. Compare model behavior before/after adding data. 2615. Add experiment logs for runs and hyperparameters. 2626. Eventually consider a byte-level or subword tokenizer, but not yet.