xonecas.io
768
La

learning-ai-lua-code-model / files

learning-ai-lua-code-model / notes / 001-first-dataset.md

1.2 KB
1# First dataset pipeline
2
3A language model cannot read text directly. First, we turn text into numbers.
4
5For this first milestone, we use a **character-level tokenizer**:
6
7- each unique character in the corpus gets an integer ID
8- encoding means `text -> list[int]`
9- decoding means `list[int] -> text`
10
11Example idea:
12
13```text
14"lua" -> [12, 20, 3]
15```
16
17The exact numbers do not matter. What matters is that the same vocabulary is used consistently.
18
19## Why split into train and validation?
20
21The model learns from the **training** split.
22
23The **validation** split is held back so we can ask, "Does the model only memorize the training text, or is it learning patterns that also help on text it did not train on?"
24
25For the tiny first corpus, overfitting is expected and useful. It proves that the data path, model, loss, optimizer, checkpointing, and sampling all work before we scale up.
26
27## Files produced by the preparation script
28
29Running `prepare_tiny_corpus.py` creates:
30
31- `tiny_train.txt` / `tiny_val.txt`: readable text splits
32- `tiny_train_ids.txt` / `tiny_val_ids.txt`: token IDs for model training
33- `tiny_vocab.json`: the character vocabulary
34- `tiny_metadata.json`: counts and paths for quick inspection