xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / AGENTS.md

3.7 KB
1# Lua Code Model Learning Project
2
3## Purpose
4
5This folder is for a first hands-on language-model project: a tiny GPT-style model focused on Lua code and Lua documentation.
6
7The goal is **learning**, not producing a useful coding assistant.
8
9We want to build a codebase that can be studied, modified, and experimented with by a software engineer who is new to AI/ML engineering.
10
11## High-level objective
12
13Train a small autoregressive language model on a focused Lua corpus and use it for simple code/documentation continuation experiments.
14
15The project should teach, through code:
16
17- dataset collection and cleaning
18- train/validation splits
19- tokenization
20- next-token prediction
21- transformer/GPT architecture basics
22- training loops
23- loss and perplexity
24- sampling/generation
25- overfitting and evaluation
26- practical experiment tracking
27
28## Scope
29
30Initial scope:
31
32- Single language: **Lua**
33- Small model
34- Small dataset
35- Local-first workflow
36- Clear, readable code over maximum performance
37- Prefer simple implementations before abstractions
38
39Out of scope for the first phase:
40
41- Building a generally useful coding assistant
42- Training a large model
43- Distributed training
44- RLHF/RLAIF
45- Tool use / agent behavior
46- Multi-language datasets
47- Massive web-scale scraping
48
49## Likely data sources
50
51Start with high-quality, legally usable Lua material such as:
52
53- Lua reference manual
54- official Lua documentation
55- official examples, where available
56- small curated Lua source files from permissive open-source projects
57
58Before adding any dataset, record:
59
60- source URL or repository
61- license
62- what was included
63- what was excluded
64- any cleaning/filtering performed
65
66Avoid generated, vendored, minified, obfuscated, or low-quality code.
67
68## Preferred implementation approach
69
70Start with a minimal educational implementation inspired by `nanoGPT`:
71
721. Build a tiny character-level or byte-level baseline first.
732. Train on a very small Lua corpus.
743. Verify the full pipeline end to end.
754. Add better tokenization later if useful.
765. Keep the code readable and heavily commented where concepts are new.
77
78Use PyTorch unless we explicitly decide otherwise.
79
80## Python and package management
81
82Use `uv` for Python environment and package management.
83
84Do not install new Python packages, system packages, frameworks, or tooling without explicit user approval first. When proposing a dependency, explain what it is for and why the project needs it.
85
86## Working style for future agents
87
88When working in this folder:
89
90- Read this `AGENTS.md` first.
91- Keep changes small and explainable.
92- Prefer incremental milestones over large rewrites.
93- Do not assume the user already knows ML terminology; explain key concepts briefly.
94- When adding code, favor clarity over cleverness.
95- When adding dependencies, explain why they are needed.
96- Ask before downloading large datasets or model weights.
97- Ask before introducing Hugging Face, Lightning, WandB, or other larger frameworks.
98- Preserve this as a learning codebase, not just a training script dump.
99
100## Suggested first milestones
101
1021. Create a minimal Python project structure.
1032. Add scripts to download or manually ingest Lua documentation.
1043. Build a simple text dataset file.
1054. Implement a character-level tokenizer.
1065. Implement a tiny GPT model in PyTorch.
1076. Train until it overfits a tiny sample.
1087. Train on the initial Lua corpus.
1098. Add a simple sampling script.
1109. Write notes explaining what each part does.
111
112## Success criteria for the first adventure
113
114We should be able to run a local command that:
115
1161. prepares a small Lua training corpus,
1172. trains a tiny model,
1183. saves a checkpoint,
1194. generates Lua-like text from a prompt,
1205. and leaves behind clear notes explaining what happened.