xonecas.io
6533
Yo

yoctomarkdown / commit 3bb66a3

docs: add token API refactor TODO for cel-tui integration

xonecas2026-04-04 3bb66a34d0ebc80adaef6392f24d7fb8a5656767 parent c528e67

1 file changed

TODO.md+61−0
@@ -0,0 +1,61 @@
1+# yoctomarkdown — TODO
2+
3+## Refactor: expose ANSI-free token API
4+
5+**Motivation:** cel-tui's `Markdown` component needs structured tokens (not ANSI strings) to map markdown blocks to TUI nodes. The parser and renderer are currently interleaved — `parseLine` calls `theme.heading()`, `theme.bold()`, etc. inline. Split them so the parser produces tokens and the ANSI renderer becomes one consumer of those tokens.
6+
7+**Consumer:** `@cel-tui/components` — see `packages/components/src/markdown.ts` in `~/src/cel-tui`, which currently has a placeholder local tokenizer that duplicates the parsing logic.
8+
9+### Token types to expose
10+
11+```ts
12+type InlineSpan =
13+ | { type: "text"; content: string }
14+ | { type: "bold"; content: string }
15+ | { type: "italic"; content: string }
16+ | { type: "code"; content: string }
17+ | { type: "link"; text: string; url: string };
18+
19+type BlockToken =
20+ | { type: "heading"; depth: 1 | 2 | 3; spans: InlineSpan[] }
21+ | { type: "paragraph"; spans: InlineSpan[] }
22+ | { type: "code_block"; lang: string; content: string }
23+ | { type: "list_item"; ordered: boolean; index: number; spans: InlineSpan[] }
24+ | { type: "blockquote"; spans: InlineSpan[] }
25+ | { type: "hr" }
26+ | { type: "blank" };
27+```
28+
29+### API to expose
30+
31+```ts
32+// Batch — tokenize a complete string
33+function tokenize(input: string): BlockToken[];
34+
35+// Incremental — streaming chunk by chunk
36+interface TokenHighlighter {
37+ write(chunk: string | Uint8Array): BlockToken[];
38+ end(): BlockToken[];
39+}
40+function createTokenizer(): TokenHighlighter;
41+```
42+
43+### Refactor steps
44+
45+1. Extract `parseInline` to return `InlineSpan[]` instead of an ANSI string
46+2. Extract `parseLine` to return `BlockToken` instead of an ANSI string
47+3. Add `tokenize(input)` — splits into lines, calls parseLine, returns `BlockToken[]`
48+4. Add `createTokenizer()` — same streaming model as `createHighlighter` but returns tokens
49+5. Rewrite `createHighlighter` to use `createTokenizer` internally + theme rendering on top
50+6. Export token types and tokenizer functions from `src/index.ts`
51+7. The ANSI renderer (`createHighlighter`, `highlightSync`) keeps its existing API and deps (`yoctocolors`, `string-width`, `strip-ansi`) — unchanged behavior
52+8. The token API (`tokenize`, `createTokenizer`, types) has **zero dependencies** — pure parsing
53+
54+### Constraints
55+
56+- Existing `highlightSync` and `createHighlighter` APIs must not change (backward compatible)
57+- Streaming behavior must be preserved — `createTokenizer().write()` emits completed block tokens immediately, buffers only the current incomplete line
58+- Unclosed code blocks emit as `code_block` on `end()`
59+- Unclosed inline formatting falls back to plain `text` spans
60+- All existing tests must continue to pass
61+- Update spec.md to document the new token API