768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 002-bigram-pytorch.md
4.0 KB
1# Bigram model and first PyTorch training loop 2 3The next goal is to train a tiny model that predicts the next character. 4 5This is not GPT yet. It is a **bigram language model**: 6 7```text 8current token -> next token 9``` 10 11If the text is: 12 13```text 14local 15``` 16 17then the training pairs include: 18 19```text 20l -> o 21o -> c 22c -> a 23a -> l 24``` 25 26Because it only sees one character of context, it cannot understand longer patterns. That limitation is useful: it lets us learn the training loop before learning Transformers. 27 28## Tensor 29 30A **tensor** is PyTorch's main data container. 31 32You can think of it as a number, list, matrix, or higher-dimensional grid of numbers: 33 34```text 35scalar: 7 36vector: [1, 2, 3] 37matrix: [[1, 2], [3, 4]] 383D: a batch of matrices 39``` 40 41Our token IDs become tensors because neural networks operate on numeric arrays, not Python strings. 42 43In the training script: 44 45```python 46train_data = torch.tensor(ids, dtype=torch.long) 47``` 48 49`dtype=torch.long` means integer IDs. PyTorch embedding layers expect integer token IDs. 50 51## Batch 52 53Training on one example at a time is slow and noisy. A **batch** groups many examples together. 54 55Our batch shape is: 56 57```text 58batch_size x block_size 59``` 60 61With: 62 63```python 64BATCH_SIZE = 32 65BLOCK_SIZE = 32 66``` 67 68we feed the model 32 short chunks of 32 characters each. 69 70## Inputs and targets 71 72For next-character prediction, the target is the input shifted by one character. 73 74Example: 75 76```text 77input: local functio 78target: ocal function 79``` 80 81At every position, the model guesses the next character. 82 83## Parameters / weights 84 85A neural network contains trainable numbers called **parameters** or **weights**. 86 87In this bigram model, the trainable parameter is one table: 88 89```text 90vocab_size x vocab_size 91``` 92 93Our tiny vocabulary has 51 characters, so the model has: 94 95```text 9651 x 51 = 2601 parameters 97``` 98 99Each row means: 100 101```text 102if the current character is X, what scores should we give possible next characters? 103``` 104 105## Logits 106 107**Logits** are raw prediction scores before they become probabilities. 108 109For one input token, the model outputs one score for every possible next token. 110 111If the vocabulary size is 51, each prediction has 51 logits. 112 113Higher logit = the model currently thinks that next token is more likely. 114 115## Softmax 116 117During generation, we turn logits into probabilities with **softmax**. 118 119Softmax converts arbitrary scores like: 120 121```text 122[2.0, 0.5, -1.0] 123``` 124 125into probabilities that add up to 1. 126 127Then we sample from those probabilities to choose the next character. 128 129## Loss 130 131**Loss** is the number we want training to minimize. 132 133Here we use cross-entropy loss. It answers: 134 135```text 136How surprised was the model by the correct next token? 137``` 138 139Lower is better. 140 141A randomly initialized model usually starts with loss near: 142 143```text 144ln(vocab_size) 145``` 146 147For 51 characters, that is about 3.93. 148 149## Gradient 150 151A **gradient** tells each parameter which direction would reduce the loss. 152 153PyTorch computes gradients with: 154 155```python 156loss.backward() 157``` 158 159This is backpropagation. 160 161## Optimizer 162 163An **optimizer** updates model parameters using gradients. 164 165The script uses: 166 167```python 168torch.optim.AdamW 169``` 170 171The basic training rhythm is: 172 173```python 174optimizer.zero_grad() 175loss.backward() 176optimizer.step() 177``` 178 179Meaning: 180 1811. clear old gradients 1822. compute new gradients from the current batch 1833. adjust parameters to reduce loss 184 185## Checkpoint 186 187A **checkpoint** is a saved training artifact. 188 189Our checkpoint includes: 190 191- model weights 192- vocabulary 193- hyperparameters 194- final train/validation loss 195 196It is written to: 197 198```text 199checkpoints/bigram_tiny.pt 200``` 201 202This lets us reload the trained model later without retraining from scratch. 203 204## Why this model will generate weird text 205 206The bigram model only knows one-character transitions. 207 208It may learn that: 209 210```text 211l is often followed by o 212n is often followed by d 213``` 214 215But it cannot know that `function` should end with `end`, or that parentheses should match. That requires longer context and a more powerful architecture. 216 217This is expected. The point is to learn the mechanics first.