768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / README.md
3.7 KB
1# Lua Code Model 2 3A tiny, educational language-model project focused on Lua code and Lua documentation. 4 5The goal is to learn the pieces of a GPT-style model one at a time, starting with the smallest possible end-to-end dataset pipeline. 6 7## Current milestone 8 9This repo currently contains: 10 11- a small hand-written Lua corpus in `data/raw/tiny_lua_corpus.txt` 12- a source log in `data/SOURCES.md` 13- a standard-library-only preparation script in `src/lua_code_model/prepare_tiny_corpus.py` 14- a PyTorch bigram training script in `src/lua_code_model/train_bigram.py` 15- a PyTorch fixed-context MLP training script in `src/lua_code_model/train_mlp.py` 16- a tiny GPT-style Transformer training script in `src/lua_code_model/train_gpt.py` 17- a checkpoint sampling script in `src/lua_code_model/sample.py` 18- short learning notes in `notes/` 19 20## Install/sync dependencies 21 22This project uses `uv`. 23 24```bash 25uv sync 26``` 27 28The only third-party dependency at this stage is PyTorch. 29 30## Prepare the tiny corpus 31 32From the project root: 33 34```bash 35PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus 36``` 37 38The script writes generated files to `data/processed/`: 39 40- `tiny_train.txt` and `tiny_val.txt`: human-readable text splits 41- `tiny_train_ids.txt` and `tiny_val_ids.txt`: one character token ID per line 42- `tiny_vocab.json`: character-to-ID mapping 43- `tiny_metadata.json`: basic dataset statistics 44 45## Train the tiny bigram model 46 47After preparing the corpus: 48 49```bash 50PYTHONPATH=src uv run python -m lua_code_model.train_bigram 51``` 52 53This trains a very small next-character model and writes: 54 55```text 56checkpoints/bigram_tiny.pt 57``` 58 59## Train the fixed-context MLP model 60 61After preparing the corpus: 62 63```bash 64PYTHONPATH=src uv run python -m lua_code_model.train_mlp 65``` 66 67This trains a model that uses the previous 32 characters to predict the next character and writes: 68 69```text 70checkpoints/mlp_tiny.pt 71``` 72 73## Train the tiny GPT-style Transformer model 74 75After preparing the corpus: 76 77```bash 78PYTHONPATH=src uv run python -m lua_code_model.train_gpt 79``` 80 81This trains a small character-level GPT-style Transformer with causal self-attention and writes: 82 83```text 84checkpoints/gpt_tiny.pt 85``` 86 87Checkpoints are ignored by git because they are generated output. 88 89## Generate from a saved checkpoint 90 91After training at least one model, sample from a checkpoint without retraining: 92 93```bash 94PYTHONPATH=src uv run python -m lua_code_model.sample 95``` 96 97By default this loads `checkpoints/gpt_tiny.pt`. You can choose a checkpoint, prompt, temperature, and output length: 98 99```bash 100PYTHONPATH=src uv run python -m lua_code_model.sample \ 101 --checkpoint checkpoints/gpt_tiny.pt \ 102 --prompt "local function" \ 103 --max-new-tokens 300 \ 104 --temperature 0.8 105``` 106 107The script can load the current bigram, MLP, and GPT checkpoints. 108 109## Learning notes 110 111- `notes/001-first-dataset.md`: first dataset, tokenization, and train/validation split 112- `notes/002-bigram-pytorch.md`: tensors, batches, logits, loss, gradients, optimizers, and checkpoints 113- `notes/003-fixed-context-mlp.md`: embeddings, fixed context windows, hidden layers, and ReLU 114- `notes/004-tiny-gpt.md`: token/position embeddings, self-attention, causal masks, Transformer blocks, residuals, and LayerNorm 115- `notes/005-session-handoff.md`: current project state, learned concepts, run commands, and recommended next steps 116- `notes/006-inference-sampling.md`: loading checkpoints and sampling without training 117- `notes/007-distillation-future-experiment.md`: planned teacher/student distillation experiment 118- `notes/008-future-rlm-harness.md`: future RLM-style harness/sampler experiment 119 120## Next likely step 121 122A good next milestone is to refactor duplicated tokenizer/checkpoint helpers out of the training and sampling scripts.