README.md
Lua Code Model
A tiny, educational language-model project focused on Lua code and Lua documentation.
The goal is to learn the pieces of a GPT-style model one at a time, starting with the smallest possible end-to-end dataset pipeline.
Current milestone
This repo currently contains:
- a small hand-written Lua corpus in
data/raw/tiny_lua_corpus.txt - a source log in
data/SOURCES.md - a standard-library-only preparation script in
src/lua_code_model/prepare_tiny_corpus.py - a PyTorch bigram training script in
src/lua_code_model/train_bigram.py - a PyTorch fixed-context MLP training script in
src/lua_code_model/train_mlp.py - a tiny GPT-style Transformer training script in
src/lua_code_model/train_gpt.py - a checkpoint sampling script in
src/lua_code_model/sample.py - short learning notes in
notes/
Install/sync dependencies
This project uses uv.
uv sync
The only third-party dependency at this stage is PyTorch.
Prepare the tiny corpus
From the project root:
PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus
The script writes generated files to data/processed/:
tiny_train.txtandtiny_val.txt: human-readable text splitstiny_train_ids.txtandtiny_val_ids.txt: one character token ID per linetiny_vocab.json: character-to-ID mappingtiny_metadata.json: basic dataset statistics
Train the tiny bigram model
After preparing the corpus:
PYTHONPATH=src uv run python -m lua_code_model.train_bigram
This trains a very small next-character model and writes:
checkpoints/bigram_tiny.pt
Train the fixed-context MLP model
After preparing the corpus:
PYTHONPATH=src uv run python -m lua_code_model.train_mlp
This trains a model that uses the previous 32 characters to predict the next character and writes:
checkpoints/mlp_tiny.pt
Train the tiny GPT-style Transformer model
After preparing the corpus:
PYTHONPATH=src uv run python -m lua_code_model.train_gpt
This trains a small character-level GPT-style Transformer with causal self-attention and writes:
checkpoints/gpt_tiny.pt
Checkpoints are ignored by git because they are generated output.
Generate from a saved checkpoint
After training at least one model, sample from a checkpoint without retraining:
PYTHONPATH=src uv run python -m lua_code_model.sample
By default this loads checkpoints/gpt_tiny.pt. You can choose a checkpoint, prompt, temperature, and output length:
PYTHONPATH=src uv run python -m lua_code_model.sample \
--checkpoint checkpoints/gpt_tiny.pt \
--prompt "local function" \
--max-new-tokens 300 \
--temperature 0.8
The script can load the current bigram, MLP, and GPT checkpoints.
Learning notes
notes/001-first-dataset.md: first dataset, tokenization, and train/validation splitnotes/002-bigram-pytorch.md: tensors, batches, logits, loss, gradients, optimizers, and checkpointsnotes/003-fixed-context-mlp.md: embeddings, fixed context windows, hidden layers, and ReLUnotes/004-tiny-gpt.md: token/position embeddings, self-attention, causal masks, Transformer blocks, residuals, and LayerNormnotes/005-session-handoff.md: current project state, learned concepts, run commands, and recommended next stepsnotes/006-inference-sampling.md: loading checkpoints and sampling without trainingnotes/007-distillation-future-experiment.md: planned teacher/student distillation experimentnotes/008-future-rlm-harness.md: future RLM-style harness/sampler experiment
Next likely step
A good next milestone is to refactor duplicated tokenizer/checkpoint helpers out of the training and sampling scripts.