xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 005-session-handoff.md

5.9 KB
1# Session handoff: tiny Lua language model
2
3This note captures enough context to continue the project in a future session without rereading the full chat.
4
5## Project intent
6
7This is an educational language-model project for Lua code/documentation continuation experiments.
8
9The goal is learning the pieces of a GPT-style language model, not building a useful coding assistant.
10
11Current implementation is character-level, local-first, small, and intentionally readable.
12
13## Environment and dependency state
14
15- Package manager: `uv`
16- Python project file: `pyproject.toml`
17- Third-party dependency added so far: `torch`
18- Do not add new dependencies without explicit user approval.
19
20Useful commands:
21
22```bash
23uv sync
24PYTHONPATH=src uv run python -m lua_code_model.prepare_tiny_corpus
25PYTHONPATH=src uv run python -m lua_code_model.train_bigram
26PYTHONPATH=src uv run python -m lua_code_model.train_mlp
27PYTHONPATH=src uv run python -m lua_code_model.train_gpt
28PYTHONPATH=src uv run python -m lua_code_model.sample
29```
30
31PyTorch may print a warning about NumPy not being installed. The current scripts do not need NumPy, so we intentionally did not add it.
32
33## Data pipeline
34
35Raw input:
36
37```text
38data/raw/tiny_lua_corpus.txt
39```
40
41Source tracking:
42
43```text
44data/SOURCES.md
45```
46
47Preparation script:
48
49```text
50src/lua_code_model/prepare_tiny_corpus.py
51```
52
53Generated outputs:
54
55```text
56data/processed/tiny_train.txt
57data/processed/tiny_val.txt
58data/processed/tiny_train_ids.txt
59data/processed/tiny_val_ids.txt
60data/processed/tiny_vocab.json
61data/processed/tiny_metadata.json
62```
63
64The current tokenizer is character-level:
65
66```text
67text -> characters -> integer token IDs
68```
69
70The current split is 90% train / 10% validation.
71
72Last observed prep stats:
73
74```text
75raw characters:    1008
76vocab size:        51
77train tokens:      907
78validation tokens: 101
79```
80
81## Models implemented
82
83### 1. Bigram model
84
85File:
86
87```text
88src/lua_code_model/train_bigram.py
89```
90
91Checkpoint:
92
93```text
94checkpoints/bigram_tiny.pt
95```
96
97Concept:
98
99```text
100current character -> next character
101```
102
103Purpose: teach tensors, logits, loss, gradients, optimizer steps, checkpointing, and simple generation.
104
105### 2. Fixed-context MLP
106
107File:
108
109```text
110src/lua_code_model/train_mlp.py
111```
112
113Checkpoint:
114
115```text
116checkpoints/mlp_tiny.pt
117```
118
119Concept:
120
121```text
122previous 32 characters -> embeddings -> flatten -> hidden layer -> logits -> next character
123```
124
125Purpose: teach embeddings, hidden layers, ReLU activation, fixed context windows, temperature during generation, and overfitting.
126
127Last observed behavior: it memorized the tiny training corpus very strongly.
128
129### 3. Tiny GPT-style Transformer
130
131File:
132
133```text
134src/lua_code_model/train_gpt.py
135```
136
137Checkpoint:
138
139```text
140checkpoints/gpt_tiny.pt
141```
142
143Concept:
144
145```text
146token IDs -> token embeddings + position embeddings -> Transformer blocks -> logits
147```
148
149Implemented GPT concepts:
150
151- causal self-attention
152- query/key/value projections
153- causal mask using a lower-triangular matrix
154- multi-head attention
155- feed-forward MLP inside each Transformer block
156- residual connections
157- LayerNorm
158- autoregressive generation
159
160Tiny GPT settings at last run:
161
162```python
163BATCH_SIZE = 16
164BLOCK_SIZE = 64
165MAX_STEPS = 2_000
166LEARNING_RATE = 1e-3
167N_EMBD = 64
168N_HEAD = 4
169N_LAYER = 2
170DROPOUT = 0.0
171SAMPLE_TEMPERATURE = 0.8
172```
173
174Last observed GPT stats:
175
176```text
177train tokens: 907
178val tokens:   101
179vocab size:   51
180parameters:   110387
181step 0 train loss:    4.0653
182step 2000 train loss: 0.0829
183step 2000 val loss:   6.3499
184```
185
186Interpretation: the model learned/memorized the tiny training data and overfit, which is expected at this stage.
187
188## Learning concepts already discussed
189
190The user has a working understanding of:
191
192- why we track data sources/licenses
193- character tokenization
194- train/validation split
195- tensors as numeric arrays
196- logits as raw prediction scores
197- softmax converting logits to probabilities
198- cross-entropy loss as model surprise/error signal
199- gradients from `loss.backward()`
200- optimizers using gradients to update parameters
201- checkpoints as saved model weights/metadata
202- training steps vs epochs
203- activation functions, especially ReLU
204- hidden layers as internal learned representation spaces
205- temperature as `softmax(logits / temperature)` during sampling
206- overfitting as low train loss but poor validation performance
207- broad Transformer/GPT concepts
208- GPT vs BERT vs BART-style model families
209
210## Important observed behavior
211
212Repeated or malformed generated strings from tiny GPT, such as repeated letters, are considered symptoms rather than obvious bugs.
213
214Likely causes:
215
216- character-level generation is fragile
217- corpus is extremely small
218- model is overfitting and partially recalling phrases
219- probabilistic sampling with temperature can choose imperfect next characters
220- no top-k/top-p/repetition-penalty decoding yet
221
222## Current notes files
223
224```text
225notes/001-first-dataset.md
226notes/002-bigram-pytorch.md
227notes/003-fixed-context-mlp.md
228notes/004-tiny-gpt.md
229notes/005-session-handoff.md
230```
231
232## Sampling script added
233
234The project now has a separate inference/sampling script:
235
236```text
237src/lua_code_model/sample.py
238```
239
240It can:
241
242- choose a checkpoint path with `--checkpoint`
243- load saved vocabulary and model weights
244- accept a prompt with `--prompt`
245- expose generation settings such as `--temperature`, `--max-new-tokens`, and `--seed`
246- generate text without running training again
247
248This reinforces the distinction between:
249
250```text
251training: update model weights
252inference/sampling: use saved weights to generate text
253```
254
255## Later possible milestones
256
2571. Refactor duplicated helper code across training and sampling scripts.
2582. Add top-k sampling to make generation controls more visible.
2593. Add a small amount of real, legally usable Lua documentation/source data after recording it in `data/SOURCES.md`.
2604. Compare model behavior before/after adding data.
2615. Add experiment logs for runs and hyperparameters.
2626. Eventually consider a byte-level or subword tokenizer, but not yet.