xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 002-bigram-pytorch.md

4.0 KB
1# Bigram model and first PyTorch training loop
2
3The next goal is to train a tiny model that predicts the next character.
4
5This is not GPT yet. It is a **bigram language model**:
6
7```text
8current token -> next token
9```
10
11If the text is:
12
13```text
14local
15```
16
17then the training pairs include:
18
19```text
20l -> o
21o -> c
22c -> a
23a -> l
24```
25
26Because it only sees one character of context, it cannot understand longer patterns. That limitation is useful: it lets us learn the training loop before learning Transformers.
27
28## Tensor
29
30A **tensor** is PyTorch's main data container.
31
32You can think of it as a number, list, matrix, or higher-dimensional grid of numbers:
33
34```text
35scalar:  7
36vector:  [1, 2, 3]
37matrix:  [[1, 2], [3, 4]]
383D:      a batch of matrices
39```
40
41Our token IDs become tensors because neural networks operate on numeric arrays, not Python strings.
42
43In the training script:
44
45```python
46train_data = torch.tensor(ids, dtype=torch.long)
47```
48
49`dtype=torch.long` means integer IDs. PyTorch embedding layers expect integer token IDs.
50
51## Batch
52
53Training on one example at a time is slow and noisy. A **batch** groups many examples together.
54
55Our batch shape is:
56
57```text
58batch_size x block_size
59```
60
61With:
62
63```python
64BATCH_SIZE = 32
65BLOCK_SIZE = 32
66```
67
68we feed the model 32 short chunks of 32 characters each.
69
70## Inputs and targets
71
72For next-character prediction, the target is the input shifted by one character.
73
74Example:
75
76```text
77input:  local functio
78target: ocal function
79```
80
81At every position, the model guesses the next character.
82
83## Parameters / weights
84
85A neural network contains trainable numbers called **parameters** or **weights**.
86
87In this bigram model, the trainable parameter is one table:
88
89```text
90vocab_size x vocab_size
91```
92
93Our tiny vocabulary has 51 characters, so the model has:
94
95```text
9651 x 51 = 2601 parameters
97```
98
99Each row means:
100
101```text
102if the current character is X, what scores should we give possible next characters?
103```
104
105## Logits
106
107**Logits** are raw prediction scores before they become probabilities.
108
109For one input token, the model outputs one score for every possible next token.
110
111If the vocabulary size is 51, each prediction has 51 logits.
112
113Higher logit = the model currently thinks that next token is more likely.
114
115## Softmax
116
117During generation, we turn logits into probabilities with **softmax**.
118
119Softmax converts arbitrary scores like:
120
121```text
122[2.0, 0.5, -1.0]
123```
124
125into probabilities that add up to 1.
126
127Then we sample from those probabilities to choose the next character.
128
129## Loss
130
131**Loss** is the number we want training to minimize.
132
133Here we use cross-entropy loss. It answers:
134
135```text
136How surprised was the model by the correct next token?
137```
138
139Lower is better.
140
141A randomly initialized model usually starts with loss near:
142
143```text
144ln(vocab_size)
145```
146
147For 51 characters, that is about 3.93.
148
149## Gradient
150
151A **gradient** tells each parameter which direction would reduce the loss.
152
153PyTorch computes gradients with:
154
155```python
156loss.backward()
157```
158
159This is backpropagation.
160
161## Optimizer
162
163An **optimizer** updates model parameters using gradients.
164
165The script uses:
166
167```python
168torch.optim.AdamW
169```
170
171The basic training rhythm is:
172
173```python
174optimizer.zero_grad()
175loss.backward()
176optimizer.step()
177```
178
179Meaning:
180
1811. clear old gradients
1822. compute new gradients from the current batch
1833. adjust parameters to reduce loss
184
185## Checkpoint
186
187A **checkpoint** is a saved training artifact.
188
189Our checkpoint includes:
190
191- model weights
192- vocabulary
193- hyperparameters
194- final train/validation loss
195
196It is written to:
197
198```text
199checkpoints/bigram_tiny.pt
200```
201
202This lets us reload the trained model later without retraining from scratch.
203
204## Why this model will generate weird text
205
206The bigram model only knows one-character transitions.
207
208It may learn that:
209
210```text
211l is often followed by o
212n is often followed by d
213```
214
215But it cannot know that `function` should end with `end`, or that parentheses should match. That requires longer context and a more powerful architecture.
216
217This is expected. The point is to learn the mechanics first.