768
La
learning-ai-lua-code-model / files
learning-ai-lua-code-model / notes / 001-first-dataset.md
1.2 KB
1# First dataset pipeline 2 3A language model cannot read text directly. First, we turn text into numbers. 4 5For this first milestone, we use a **character-level tokenizer**: 6 7- each unique character in the corpus gets an integer ID 8- encoding means `text -> list[int]` 9- decoding means `list[int] -> text` 10 11Example idea: 12 13```text 14"lua" -> [12, 20, 3] 15``` 16 17The exact numbers do not matter. What matters is that the same vocabulary is used consistently. 18 19## Why split into train and validation? 20 21The model learns from the **training** split. 22 23The **validation** split is held back so we can ask, "Does the model only memorize the training text, or is it learning patterns that also help on text it did not train on?" 24 25For the tiny first corpus, overfitting is expected and useful. It proves that the data path, model, loss, optimizer, checkpointing, and sampling all work before we scale up. 26 27## Files produced by the preparation script 28 29Running `prepare_tiny_corpus.py` creates: 30 31- `tiny_train.txt` / `tiny_val.txt`: readable text splits 32- `tiny_train_ids.txt` / `tiny_val_ids.txt`: token IDs for model training 33- `tiny_vocab.json`: the character vocabulary 34- `tiny_metadata.json`: counts and paths for quick inspection