xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / README.md

3.7 KB
1# Lua Code Model
2
3A tiny, educational language-model project focused on Lua code and Lua documentation.
4
5The goal is to learn the pieces of a GPT-style model one at a time, starting with the smallest possible end-to-end dataset pipeline.
6
7## Current milestone
8
9This repo currently contains:
10
11- a small hand-written Lua corpus in `data/raw/tiny_lua_corpus.txt`
12- a source log in `data/SOURCES.md`
13- a standard-library-only preparation script in `src/lua_code_model/prepare_tiny_corpus.py`
14- a PyTorch bigram training script in `src/lua_code_model/train_bigram.py`
15- a PyTorch fixed-context MLP training script in `src/lua_code_model/train_mlp.py`
16- a tiny GPT-style Transformer training script in `src/lua_code_model/train_gpt.py`
17- a checkpoint sampling script in `src/lua_code_model/sample.py`
18- short learning notes in `notes/`
19
20## Install/sync dependencies
21
22This project uses `uv`.
23
24```bash
25uv sync
26```
27
28The only third-party dependency at this stage is PyTorch.
29
30## Prepare the tiny corpus
31
32From the project root:
33
34```bash
35PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus
36```
37
38The script writes generated files to `data/processed/`:
39
40- `tiny_train.txt` and `tiny_val.txt`: human-readable text splits
41- `tiny_train_ids.txt` and `tiny_val_ids.txt`: one character token ID per line
42- `tiny_vocab.json`: character-to-ID mapping
43- `tiny_metadata.json`: basic dataset statistics
44
45## Train the tiny bigram model
46
47After preparing the corpus:
48
49```bash
50PYTHONPATH=src uv run python -m lua_code_model.train_bigram
51```
52
53This trains a very small next-character model and writes:
54
55```text
56checkpoints/bigram_tiny.pt
57```
58
59## Train the fixed-context MLP model
60
61After preparing the corpus:
62
63```bash
64PYTHONPATH=src uv run python -m lua_code_model.train_mlp
65```
66
67This trains a model that uses the previous 32 characters to predict the next character and writes:
68
69```text
70checkpoints/mlp_tiny.pt
71```
72
73## Train the tiny GPT-style Transformer model
74
75After preparing the corpus:
76
77```bash
78PYTHONPATH=src uv run python -m lua_code_model.train_gpt
79```
80
81This trains a small character-level GPT-style Transformer with causal self-attention and writes:
82
83```text
84checkpoints/gpt_tiny.pt
85```
86
87Checkpoints are ignored by git because they are generated output.
88
89## Generate from a saved checkpoint
90
91After training at least one model, sample from a checkpoint without retraining:
92
93```bash
94PYTHONPATH=src uv run python -m lua_code_model.sample
95```
96
97By default this loads `checkpoints/gpt_tiny.pt`. You can choose a checkpoint, prompt, temperature, and output length:
98
99```bash
100PYTHONPATH=src uv run python -m lua_code_model.sample \
101  --checkpoint checkpoints/gpt_tiny.pt \
102  --prompt "local function" \
103  --max-new-tokens 300 \
104  --temperature 0.8
105```
106
107The script can load the current bigram, MLP, and GPT checkpoints.
108
109## Learning notes
110
111- `notes/001-first-dataset.md`: first dataset, tokenization, and train/validation split
112- `notes/002-bigram-pytorch.md`: tensors, batches, logits, loss, gradients, optimizers, and checkpoints
113- `notes/003-fixed-context-mlp.md`: embeddings, fixed context windows, hidden layers, and ReLU
114- `notes/004-tiny-gpt.md`: token/position embeddings, self-attention, causal masks, Transformer blocks, residuals, and LayerNorm
115- `notes/005-session-handoff.md`: current project state, learned concepts, run commands, and recommended next steps
116- `notes/006-inference-sampling.md`: loading checkpoints and sampling without training
117- `notes/007-distillation-future-experiment.md`: planned teacher/student distillation experiment
118- `notes/008-future-rlm-harness.md`: future RLM-style harness/sampler experiment
119
120## Next likely step
121
122A good next milestone is to refactor duplicated tokenizer/checkpoint helpers out of the training and sampling scripts.