xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 004-tiny-gpt.md

4.4 KB
1# Tiny GPT-style Transformer
2
3This milestone introduces a very small GPT-style model.
4
5The model is still character-level and trained on the tiny Lua corpus, but the architecture is now Transformer-based.
6
7## Previous models
8
9Bigram:
10
11```text
12current character -> next character
13```
14
15Fixed-context MLP:
16
17```text
18previous 32 characters -> flatten everything -> next character
19```
20
21Tiny GPT:
22
23```text
24previous characters -> self-attention -> next-character predictions at every position
25```
26
27## Token embeddings and position embeddings
28
29The model starts with token IDs.
30
31Each token ID gets a learned token embedding:
32
33```text
34what character is this?
35```
36
37GPT also adds a position embedding:
38
39```text
40where is this character in the block?
41```
42
43The input representation is:
44
45```text
46token embedding + position embedding
47```
48
49This matters because order changes meaning. `local function` and `function local` contain similar characters, but in different positions.
50
51## Self-attention
52
53Self-attention lets each position gather information from other positions.
54
55For every token position, the model computes:
56
57- query: what am I looking for?
58- key: what information do I contain?
59- value: what information should I pass along?
60
61Attention compares queries to keys. Then it uses softmax to create attention weights. Finally, it mixes values using those weights.
62
63Conceptually:
64
65```text
66attention scores  = query dot key
67attention weights = softmax(scores)
68output            = weighted sum of values
69```
70
71So self-attention means:
72
73```text
74for this position, choose which earlier positions are useful
75```
76
77## Causal mask
78
79GPT predicts future tokens from past tokens.
80
81During training, the answer is present later in the sequence, so we must prevent cheating.
82
83The causal mask blocks each position from seeing future positions.
84
85For example, position 4 may see:
86
87```text
881, 2, 3, 4
89```
90
91but not:
92
93```text
945, 6, 7, ...
95```
96
97That is why the attention head uses a lower-triangular matrix called `tril`.
98
99## Multi-head attention
100
101One attention head can learn one kind of relationship.
102
103Multiple heads run in parallel so the model can learn several relationships at once.
104
105For code-like text, different heads might learn rough signals such as:
106
107- nearby character spelling
108- indentation/newline patterns
109- quote/string patterns
110- names used earlier in the context
111
112We do not manually assign those roles. Training discovers useful attention behavior.
113
114## Transformer block
115
116A GPT block has two main parts:
117
118```text
119causal self-attention
120feed-forward MLP
121```
122
123In code, one block does roughly:
124
125```python
126x = x + self_attention(layer_norm(x))
127x = x + feed_forward(layer_norm(x))
128```
129
130The `+` parts are residual connections.
131
132## Residual connection
133
134A residual connection adds a layer's output back to its input.
135
136Instead of:
137
138```text
139x -> layer -> new_x
140```
141
142we do:
143
144```text
145x -> layer -> change
146x + change -> new_x
147```
148
149This makes optimization easier because each layer can learn a useful adjustment instead of needing to rewrite the whole representation.
150
151## LayerNorm
152
153Layer normalization keeps each token representation numerically well-behaved.
154
155It helps training stay stable, especially as we stack multiple Transformer blocks.
156
157## Feed-forward network
158
159Transformers still use MLPs.
160
161Attention mixes information between positions. The feed-forward network then processes each position independently with nonlinear layers.
162
163So the split is:
164
165```text
166attention: communicate across sequence positions
167MLP: process each position's representation
168```
169
170## Why GPT predicts at every position
171
172Given this input:
173
174```text
175local
176```
177
178The targets are shifted by one:
179
180```text
181input:  local
182target: ocal?
183```
184
185More generally, every position predicts the next token.
186
187The causal mask makes this valid because each position only uses earlier context.
188
189## Tiny model settings
190
191The script uses:
192
193```python
194BLOCK_SIZE = 64
195N_EMBD = 64
196N_HEAD = 4
197N_LAYER = 2
198```
199
200Meaning:
201
202- the model can see up to 64 previous characters
203- each token representation has 64 numbers
204- each Transformer block has 4 attention heads
205- there are 2 Transformer blocks
206
207This is tiny by modern standards, but it contains the same core architectural ideas as larger GPT models.
208
209## Expected behavior
210
211Because the corpus is very small, this model can overfit.
212
213That is okay for this milestone. The purpose is to understand the architecture and verify that training, generation, and checkpointing work with self-attention.