768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 003-fixed-context-mlp.md
3.5 KB
1# Fixed-context MLP language model 2 3The bigram model learned: 4 5```text 6current character -> next character 7``` 8 9The fixed-context MLP learns: 10 11```text 12previous N characters -> next character 13``` 14 15In our script, `N` is: 16 17```python 18CONTEXT_SIZE = 32 19``` 20 21So the model gets 32 characters of context before making one prediction. 22 23## Why this is smarter than bigram 24 25A bigram model cannot tell the difference between these situations: 26 27```text 28loca 29hea 30playe 31``` 32 33It only sees the final character. 34 35The MLP can see a small phrase-like window: 36 37```text 38local function count_items(items) 39``` 40 41That means it can learn short patterns such as: 42 43```lua 44local function 45print( 46self.health 47end 48``` 49 50It still does not understand Lua, but it has more evidence for each prediction. 51 52## Embeddings 53 54An embedding turns a token ID into a learned vector. 55 56A token ID is just a label: 57 58```text 5942 60``` 61 62A vector can store learned features: 63 64```text 65[0.12, -0.33, 0.91, ...] 66``` 67 68In the MLP script: 69 70```python 71self.token_embedding_table = nn.Embedding(vocab_size, embedding_size) 72``` 73 74With: 75 76```python 77EMBEDDING_SIZE = 32 78``` 79 80Each character gets 32 learned numbers. 81 82## Flattening the context 83 84For one example, the embeddings have this shape: 85 86```text 87context_size x embedding_size 88``` 89 90With our settings: 91 92```text 9332 x 32 94``` 95 96The MLP wants one long input vector, so we flatten it: 97 98```text 9932 x 32 -> 1024 100``` 101 102That flattened vector represents the whole fixed context window. 103 104## Hidden layer 105 106The model then uses a small neural network: 107 108```text 109flattened context -> Linear -> ReLU -> Linear -> logits 110``` 111 112The first `Linear` layer mixes information from all positions in the context. 113 114This is where the model can learn patterns like: 115 116```text 117if the context contains "functio", predict "n" 118``` 119 120## ReLU 121 122`ReLU` is an activation function. 123 124It turns negative numbers into zero and leaves positive numbers alone: 125 126```text 127ReLU(-2.0) = 0 128ReLU( 3.0) = 3 129``` 130 131Why use it? 132 133Without activation functions, stacking linear layers would still behave like one big linear layer. ReLU gives the model nonlinear behavior, which lets it learn more complex patterns. 134 135## Output logits 136 137The final layer returns one logit for each possible next character. 138 139If the vocabulary has 51 characters, the output has 51 logits: 140 141```text 142logit for newline 143logit for space 144logit for "a" 145... 146``` 147 148Cross-entropy loss compares those logits to the correct next character. 149 150## What changed from bigram training? 151 152Bigram training examples looked like: 153 154```text 155input: a 156target: b 157``` 158 159MLP examples look like: 160 161```text 162input: previous 32 characters 163target: next character 164``` 165 166The training loop is still the same rhythm: 167 168```python 169optimizer.zero_grad() 170loss.backward() 171optimizer.step() 172``` 173 174So the important learning point is: 175 176> The model architecture changed, but the basic training loop stayed almost identical. 177 178## Limitation 179 180The MLP has a fixed context size. 181 182If something important happened 100 characters ago, and `CONTEXT_SIZE = 32`, the model cannot see it. 183 184Also, flattening the context means the model has a separate learned use for each position. Position 1 and position 20 are treated as totally different input slots. 185 186GPT handles this more flexibly with self-attention. 187 188## What GPT will add later 189 190GPT still uses: 191 192```text 193token IDs -> embeddings -> neural network -> logits -> loss 194``` 195 196But its neural network is a Transformer. 197 198The big new idea will be **attention**: 199 200```text 201which previous tokens matter most for this prediction? 202``` 203 204That is why the learning ladder is: 205 206```text 207bigram -> fixed-context MLP -> tiny GPT 208```