6533
Yo
yoctomarkdown / files
yoctomarkdown / TODO.md
2.1 KB
1# yoctomarkdown — TODO 2 3## Refactor: expose ANSI-free token API 4 5**Motivation:** cel-tui's `Markdown` component needs structured tokens (not ANSI strings) to map markdown blocks to TUI nodes. The parser and renderer are currently interleaved — `parseLine` calls `theme.heading()`, `theme.bold()`, etc. inline. Split them so the parser produces tokens and the ANSI renderer becomes one consumer of those tokens. 6 7**Consumer:** `@cel-tui/components` — see `packages/components/src/markdown.ts` in `~/src/cel-tui`, which currently has a placeholder local tokenizer that duplicates the parsing logic. 8 9### Token types and API 10 11See [spec.md — Token API](spec.md#token-api) for the full type definitions (`InlineSpan`, `BlockToken`) and function signatures (`tokenize`, `createTokenizer`). 12 13### Refactor steps 14 15- [x] Extract `parseInline` to return `InlineSpan[]` instead of an ANSI string 16- [x] Extract `parseLine` to return `BlockToken` instead of an ANSI string 17- [x] Add `tokenize(input)` — splits into lines, calls parseLine, returns `BlockToken[]` 18- [x] Add `createTokenizer()` — same streaming model as `createHighlighter` but returns tokens 19- [x] Rewrite `createHighlighter` to use tokenizer's `parseLineToken` + theme rendering on top 20- [x] Export token types and tokenizer functions from `src/index.ts` 21- [x] The ANSI renderer (`createHighlighter`, `highlightSync`) keeps its existing API and deps (`yoctocolors`, `string-width`, `strip-ansi`) — unchanged behavior 22- [x] The token API (`tokenize`, `createTokenizer`, types) has **zero dependencies** — pure parsing (`src/tokenizer.ts`) 23 24### Constraints 25 26- [x] Existing `highlightSync` and `createHighlighter` APIs must not change (backward compatible) 27- [x] Streaming behavior must be preserved — `createTokenizer().write()` emits completed block tokens immediately, buffers only the current incomplete line 28- [x] Unclosed code blocks emit as `code_block` on `end()` 29- [x] Unclosed inline formatting falls back to plain `text` spans 30- [x] All existing tests must continue to pass (11 original + 39 new = 50 total) 31- [x] Update spec.md to document the new token API