xonecas.io
6533
Yo

yoctomarkdown / files

yoctomarkdown / TODO.md

2.1 KB
1# yoctomarkdown — TODO
2
3## Refactor: expose ANSI-free token API
4
5**Motivation:** cel-tui's `Markdown` component needs structured tokens (not ANSI strings) to map markdown blocks to TUI nodes. The parser and renderer are currently interleaved — `parseLine` calls `theme.heading()`, `theme.bold()`, etc. inline. Split them so the parser produces tokens and the ANSI renderer becomes one consumer of those tokens.
6
7**Consumer:** `@cel-tui/components` — see `packages/components/src/markdown.ts` in `~/src/cel-tui`, which currently has a placeholder local tokenizer that duplicates the parsing logic.
8
9### Token types and API
10
11See [spec.md — Token API](spec.md#token-api) for the full type definitions (`InlineSpan`, `BlockToken`) and function signatures (`tokenize`, `createTokenizer`).
12
13### Refactor steps
14
15- [x] Extract `parseInline` to return `InlineSpan[]` instead of an ANSI string
16- [x] Extract `parseLine` to return `BlockToken` instead of an ANSI string
17- [x] Add `tokenize(input)` — splits into lines, calls parseLine, returns `BlockToken[]`
18- [x] Add `createTokenizer()` — same streaming model as `createHighlighter` but returns tokens
19- [x] Rewrite `createHighlighter` to use tokenizer's `parseLineToken` + theme rendering on top
20- [x] Export token types and tokenizer functions from `src/index.ts`
21- [x] The ANSI renderer (`createHighlighter`, `highlightSync`) keeps its existing API and deps (`yoctocolors`, `string-width`, `strip-ansi`) — unchanged behavior
22- [x] The token API (`tokenize`, `createTokenizer`, types) has **zero dependencies** — pure parsing (`src/tokenizer.ts`)
23
24### Constraints
25
26- [x] Existing `highlightSync` and `createHighlighter` APIs must not change (backward compatible)
27- [x] Streaming behavior must be preserved — `createTokenizer().write()` emits completed block tokens immediately, buffers only the current incomplete line
28- [x] Unclosed code blocks emit as `code_block` on `end()`
29- [x] Unclosed inline formatting falls back to plain `text` spans
30- [x] All existing tests must continue to pass (11 original + 39 new = 50 total)
31- [x] Update spec.md to document the new token API