xonecas.io

learning-ai-lua-code-model

A small implementation of LLM architectures for learning and experimentation

post-AI small frontend pytorch

8 commits · 2026-06-14 to 2026-06-16 · 25 files

xonecasfeat: adds discussion and experiement idea about RLMed2252d2026-06-16
▸ data/ 2026-06-14
▸ notes/ 2026-06-16
▸ src/ 2026-06-14
· .gitignore 2026-06-14
· AGENTS.md 2026-06-14
· pi-session-2026-06-14T15-38-16-737Z_019ec6c8-9d21-7ec3-a6c8-7b20e72a4bf0.html 2026-06-14
· pi-session-2026-06-16T08-14-44-950Z_019ecf7f-44d6-7517-9eda-a8d6da0293ca.html 2026-06-16
· pyproject.toml 2026-06-14
· README.md 2026-06-16
· uv.lock 2026-06-14

README.md

Lua Code Model

A tiny, educational language-model project focused on Lua code and Lua documentation.

The goal is to learn the pieces of a GPT-style model one at a time, starting with the smallest possible end-to-end dataset pipeline.

Current milestone

This repo currently contains:

  • a small hand-written Lua corpus in data/raw/tiny_lua_corpus.txt
  • a source log in data/SOURCES.md
  • a standard-library-only preparation script in src/lua_code_model/prepare_tiny_corpus.py
  • a PyTorch bigram training script in src/lua_code_model/train_bigram.py
  • a PyTorch fixed-context MLP training script in src/lua_code_model/train_mlp.py
  • a tiny GPT-style Transformer training script in src/lua_code_model/train_gpt.py
  • a checkpoint sampling script in src/lua_code_model/sample.py
  • short learning notes in notes/

Install/sync dependencies

This project uses uv.

uv sync

The only third-party dependency at this stage is PyTorch.

Prepare the tiny corpus

From the project root:

PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus

The script writes generated files to data/processed/:

  • tiny_train.txt and tiny_val.txt: human-readable text splits
  • tiny_train_ids.txt and tiny_val_ids.txt: one character token ID per line
  • tiny_vocab.json: character-to-ID mapping
  • tiny_metadata.json: basic dataset statistics

Train the tiny bigram model

After preparing the corpus:

PYTHONPATH=src uv run python -m lua_code_model.train_bigram

This trains a very small next-character model and writes:

checkpoints/bigram_tiny.pt

Train the fixed-context MLP model

After preparing the corpus:

PYTHONPATH=src uv run python -m lua_code_model.train_mlp

This trains a model that uses the previous 32 characters to predict the next character and writes:

checkpoints/mlp_tiny.pt

Train the tiny GPT-style Transformer model

After preparing the corpus:

PYTHONPATH=src uv run python -m lua_code_model.train_gpt

This trains a small character-level GPT-style Transformer with causal self-attention and writes:

checkpoints/gpt_tiny.pt

Checkpoints are ignored by git because they are generated output.

Generate from a saved checkpoint

After training at least one model, sample from a checkpoint without retraining:

PYTHONPATH=src uv run python -m lua_code_model.sample

By default this loads checkpoints/gpt_tiny.pt. You can choose a checkpoint, prompt, temperature, and output length:

PYTHONPATH=src uv run python -m lua_code_model.sample \
  --checkpoint checkpoints/gpt_tiny.pt \
  --prompt "local function" \
  --max-new-tokens 300 \
  --temperature 0.8

The script can load the current bigram, MLP, and GPT checkpoints.

Learning notes

  • notes/001-first-dataset.md: first dataset, tokenization, and train/validation split
  • notes/002-bigram-pytorch.md: tensors, batches, logits, loss, gradients, optimizers, and checkpoints
  • notes/003-fixed-context-mlp.md: embeddings, fixed context windows, hidden layers, and ReLU
  • notes/004-tiny-gpt.md: token/position embeddings, self-attention, causal masks, Transformer blocks, residuals, and LayerNorm
  • notes/005-session-handoff.md: current project state, learned concepts, run commands, and recommended next steps
  • notes/006-inference-sampling.md: loading checkpoints and sampling without training
  • notes/007-distillation-future-experiment.md: planned teacher/student distillation experiment
  • notes/008-future-rlm-harness.md: future RLM-style harness/sampler experiment

Next likely step

A good next milestone is to refactor duplicated tokenizer/checkpoint helpers out of the training and sampling scripts.