768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 004-tiny-gpt.md
4.4 KB
1# Tiny GPT-style Transformer 2 3This milestone introduces a very small GPT-style model. 4 5The model is still character-level and trained on the tiny Lua corpus, but the architecture is now Transformer-based. 6 7## Previous models 8 9Bigram: 10 11```text 12current character -> next character 13``` 14 15Fixed-context MLP: 16 17```text 18previous 32 characters -> flatten everything -> next character 19``` 20 21Tiny GPT: 22 23```text 24previous characters -> self-attention -> next-character predictions at every position 25``` 26 27## Token embeddings and position embeddings 28 29The model starts with token IDs. 30 31Each token ID gets a learned token embedding: 32 33```text 34what character is this? 35``` 36 37GPT also adds a position embedding: 38 39```text 40where is this character in the block? 41``` 42 43The input representation is: 44 45```text 46token embedding + position embedding 47``` 48 49This matters because order changes meaning. `local function` and `function local` contain similar characters, but in different positions. 50 51## Self-attention 52 53Self-attention lets each position gather information from other positions. 54 55For every token position, the model computes: 56 57- query: what am I looking for? 58- key: what information do I contain? 59- value: what information should I pass along? 60 61Attention compares queries to keys. Then it uses softmax to create attention weights. Finally, it mixes values using those weights. 62 63Conceptually: 64 65```text 66attention scores = query dot key 67attention weights = softmax(scores) 68output = weighted sum of values 69``` 70 71So self-attention means: 72 73```text 74for this position, choose which earlier positions are useful 75``` 76 77## Causal mask 78 79GPT predicts future tokens from past tokens. 80 81During training, the answer is present later in the sequence, so we must prevent cheating. 82 83The causal mask blocks each position from seeing future positions. 84 85For example, position 4 may see: 86 87```text 881, 2, 3, 4 89``` 90 91but not: 92 93```text 945, 6, 7, ... 95``` 96 97That is why the attention head uses a lower-triangular matrix called `tril`. 98 99## Multi-head attention 100 101One attention head can learn one kind of relationship. 102 103Multiple heads run in parallel so the model can learn several relationships at once. 104 105For code-like text, different heads might learn rough signals such as: 106 107- nearby character spelling 108- indentation/newline patterns 109- quote/string patterns 110- names used earlier in the context 111 112We do not manually assign those roles. Training discovers useful attention behavior. 113 114## Transformer block 115 116A GPT block has two main parts: 117 118```text 119causal self-attention 120feed-forward MLP 121``` 122 123In code, one block does roughly: 124 125```python 126x = x + self_attention(layer_norm(x)) 127x = x + feed_forward(layer_norm(x)) 128``` 129 130The `+` parts are residual connections. 131 132## Residual connection 133 134A residual connection adds a layer's output back to its input. 135 136Instead of: 137 138```text 139x -> layer -> new_x 140``` 141 142we do: 143 144```text 145x -> layer -> change 146x + change -> new_x 147``` 148 149This makes optimization easier because each layer can learn a useful adjustment instead of needing to rewrite the whole representation. 150 151## LayerNorm 152 153Layer normalization keeps each token representation numerically well-behaved. 154 155It helps training stay stable, especially as we stack multiple Transformer blocks. 156 157## Feed-forward network 158 159Transformers still use MLPs. 160 161Attention mixes information between positions. The feed-forward network then processes each position independently with nonlinear layers. 162 163So the split is: 164 165```text 166attention: communicate across sequence positions 167MLP: process each position's representation 168``` 169 170## Why GPT predicts at every position 171 172Given this input: 173 174```text 175local 176``` 177 178The targets are shifted by one: 179 180```text 181input: local 182target: ocal? 183``` 184 185More generally, every position predicts the next token. 186 187The causal mask makes this valid because each position only uses earlier context. 188 189## Tiny model settings 190 191The script uses: 192 193```python 194BLOCK_SIZE = 64 195N_EMBD = 64 196N_HEAD = 4 197N_LAYER = 2 198``` 199 200Meaning: 201 202- the model can see up to 64 previous characters 203- each token representation has 64 numbers 204- each Transformer block has 4 attention heads 205- there are 2 Transformer blocks 206 207This is tiny by modern standards, but it contains the same core architectural ideas as larger GPT models. 208 209## Expected behavior 210 211Because the corpus is very small, this model can overfit. 212 213That is okay for this milestone. The purpose is to understand the architecture and verify that training, generation, and checkpointing work with self-attention.