xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 003-fixed-context-mlp.md

3.5 KB
1# Fixed-context MLP language model
2
3The bigram model learned:
4
5```text
6current character -> next character
7```
8
9The fixed-context MLP learns:
10
11```text
12previous N characters -> next character
13```
14
15In our script, `N` is:
16
17```python
18CONTEXT_SIZE = 32
19```
20
21So the model gets 32 characters of context before making one prediction.
22
23## Why this is smarter than bigram
24
25A bigram model cannot tell the difference between these situations:
26
27```text
28loca
29hea
30playe
31```
32
33It only sees the final character.
34
35The MLP can see a small phrase-like window:
36
37```text
38local function count_items(items)
39```
40
41That means it can learn short patterns such as:
42
43```lua
44local function
45print(
46self.health
47end
48```
49
50It still does not understand Lua, but it has more evidence for each prediction.
51
52## Embeddings
53
54An embedding turns a token ID into a learned vector.
55
56A token ID is just a label:
57
58```text
5942
60```
61
62A vector can store learned features:
63
64```text
65[0.12, -0.33, 0.91, ...]
66```
67
68In the MLP script:
69
70```python
71self.token_embedding_table = nn.Embedding(vocab_size, embedding_size)
72```
73
74With:
75
76```python
77EMBEDDING_SIZE = 32
78```
79
80Each character gets 32 learned numbers.
81
82## Flattening the context
83
84For one example, the embeddings have this shape:
85
86```text
87context_size x embedding_size
88```
89
90With our settings:
91
92```text
9332 x 32
94```
95
96The MLP wants one long input vector, so we flatten it:
97
98```text
9932 x 32 -> 1024
100```
101
102That flattened vector represents the whole fixed context window.
103
104## Hidden layer
105
106The model then uses a small neural network:
107
108```text
109flattened context -> Linear -> ReLU -> Linear -> logits
110```
111
112The first `Linear` layer mixes information from all positions in the context.
113
114This is where the model can learn patterns like:
115
116```text
117if the context contains "functio", predict "n"
118```
119
120## ReLU
121
122`ReLU` is an activation function.
123
124It turns negative numbers into zero and leaves positive numbers alone:
125
126```text
127ReLU(-2.0) = 0
128ReLU( 3.0) = 3
129```
130
131Why use it?
132
133Without activation functions, stacking linear layers would still behave like one big linear layer. ReLU gives the model nonlinear behavior, which lets it learn more complex patterns.
134
135## Output logits
136
137The final layer returns one logit for each possible next character.
138
139If the vocabulary has 51 characters, the output has 51 logits:
140
141```text
142logit for newline
143logit for space
144logit for "a"
145...
146```
147
148Cross-entropy loss compares those logits to the correct next character.
149
150## What changed from bigram training?
151
152Bigram training examples looked like:
153
154```text
155input:  a
156target: b
157```
158
159MLP examples look like:
160
161```text
162input:  previous 32 characters
163target: next character
164```
165
166The training loop is still the same rhythm:
167
168```python
169optimizer.zero_grad()
170loss.backward()
171optimizer.step()
172```
173
174So the important learning point is:
175
176> The model architecture changed, but the basic training loop stayed almost identical.
177
178## Limitation
179
180The MLP has a fixed context size.
181
182If something important happened 100 characters ago, and `CONTEXT_SIZE = 32`, the model cannot see it.
183
184Also, flattening the context means the model has a separate learned use for each position. Position 1 and position 20 are treated as totally different input slots.
185
186GPT handles this more flexibly with self-attention.
187
188## What GPT will add later
189
190GPT still uses:
191
192```text
193token IDs -> embeddings -> neural network -> logits -> loss
194```
195
196But its neural network is a Transformer.
197
198The big new idea will be **attention**:
199
200```text
201which previous tokens matter most for this prediction?
202```
203
204That is why the learning ladder is:
205
206```text
207bigram -> fixed-context MLP -> tiny GPT
208```