768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / AGENTS.md
3.7 KB
1# Lua Code Model Learning Project 2 3## Purpose 4 5This folder is for a first hands-on language-model project: a tiny GPT-style model focused on Lua code and Lua documentation. 6 7The goal is **learning**, not producing a useful coding assistant. 8 9We want to build a codebase that can be studied, modified, and experimented with by a software engineer who is new to AI/ML engineering. 10 11## High-level objective 12 13Train a small autoregressive language model on a focused Lua corpus and use it for simple code/documentation continuation experiments. 14 15The project should teach, through code: 16 17- dataset collection and cleaning 18- train/validation splits 19- tokenization 20- next-token prediction 21- transformer/GPT architecture basics 22- training loops 23- loss and perplexity 24- sampling/generation 25- overfitting and evaluation 26- practical experiment tracking 27 28## Scope 29 30Initial scope: 31 32- Single language: **Lua** 33- Small model 34- Small dataset 35- Local-first workflow 36- Clear, readable code over maximum performance 37- Prefer simple implementations before abstractions 38 39Out of scope for the first phase: 40 41- Building a generally useful coding assistant 42- Training a large model 43- Distributed training 44- RLHF/RLAIF 45- Tool use / agent behavior 46- Multi-language datasets 47- Massive web-scale scraping 48 49## Likely data sources 50 51Start with high-quality, legally usable Lua material such as: 52 53- Lua reference manual 54- official Lua documentation 55- official examples, where available 56- small curated Lua source files from permissive open-source projects 57 58Before adding any dataset, record: 59 60- source URL or repository 61- license 62- what was included 63- what was excluded 64- any cleaning/filtering performed 65 66Avoid generated, vendored, minified, obfuscated, or low-quality code. 67 68## Preferred implementation approach 69 70Start with a minimal educational implementation inspired by `nanoGPT`: 71 721. Build a tiny character-level or byte-level baseline first. 732. Train on a very small Lua corpus. 743. Verify the full pipeline end to end. 754. Add better tokenization later if useful. 765. Keep the code readable and heavily commented where concepts are new. 77 78Use PyTorch unless we explicitly decide otherwise. 79 80## Python and package management 81 82Use `uv` for Python environment and package management. 83 84Do not install new Python packages, system packages, frameworks, or tooling without explicit user approval first. When proposing a dependency, explain what it is for and why the project needs it. 85 86## Working style for future agents 87 88When working in this folder: 89 90- Read this `AGENTS.md` first. 91- Keep changes small and explainable. 92- Prefer incremental milestones over large rewrites. 93- Do not assume the user already knows ML terminology; explain key concepts briefly. 94- When adding code, favor clarity over cleverness. 95- When adding dependencies, explain why they are needed. 96- Ask before downloading large datasets or model weights. 97- Ask before introducing Hugging Face, Lightning, WandB, or other larger frameworks. 98- Preserve this as a learning codebase, not just a training script dump. 99 100## Suggested first milestones 101 1021. Create a minimal Python project structure. 1032. Add scripts to download or manually ingest Lua documentation. 1043. Build a simple text dataset file. 1054. Implement a character-level tokenizer. 1065. Implement a tiny GPT model in PyTorch. 1076. Train until it overfits a tiny sample. 1087. Train on the initial Lua corpus. 1098. Add a simple sampling script. 1109. Write notes explaining what each part does. 111 112## Success criteria for the first adventure 113 114We should be able to run a local command that: 115 1161. prepares a small Lua training corpus, 1172. trains a tiny model, 1183. saves a checkpoint, 1194. generates Lua-like text from a prompt, 1205. and leaves behind clear notes explaining what happened.