xonecas.io
6533
Yo

yoctomarkdown / commit 05cb703

feat: expose ANSI-free token API

- Add src/tokenizer.ts with zero-dependency token parser
  - InlineSpan, BlockToken types
  - tokenize() batch API
  - createTokenizer() streaming API
- Rewrite createHighlighter to use parseLineToken + theme rendering
  (eliminates duplicated parsing logic)
- Add 39 tokenizer tests (50 total, all passing)
- Update spec.md with Token API section
- Clean up examples/streaming-test.ts (fix lint warnings, deduplicate)
- Update TODO.md with completion status

xonecas2026-04-04 05cb7039a62e5619f6850f9cf83160474a3b321a parent 3bb66a3

6 files changed

This diff is too big to show in full. The first 1000 lines are below, and every file is listed with its counts.

TODO.md+17−47
@@ -6,56 +6,26 @@
66
77 **Consumer:** `@cel-tui/components` — see `packages/components/src/markdown.ts` in `~/src/cel-tui`, which currently has a placeholder local tokenizer that duplicates the parsing logic.
88
9-### Token types to expose
10-
11-```ts
12-type InlineSpan =
13- | { type: "text"; content: string }
14- | { type: "bold"; content: string }
15- | { type: "italic"; content: string }
16- | { type: "code"; content: string }
17- | { type: "link"; text: string; url: string };
18-
19-type BlockToken =
20- | { type: "heading"; depth: 1 | 2 | 3; spans: InlineSpan[] }
21- | { type: "paragraph"; spans: InlineSpan[] }
22- | { type: "code_block"; lang: string; content: string }
23- | { type: "list_item"; ordered: boolean; index: number; spans: InlineSpan[] }
24- | { type: "blockquote"; spans: InlineSpan[] }
25- | { type: "hr" }
26- | { type: "blank" };
27-```
28-
29-### API to expose
30-
31-```ts
32-// Batch — tokenize a complete string
33-function tokenize(input: string): BlockToken[];
34-
35-// Incremental — streaming chunk by chunk
36-interface TokenHighlighter {
37- write(chunk: string | Uint8Array): BlockToken[];
38- end(): BlockToken[];
39-}
40-function createTokenizer(): TokenHighlighter;
41-```
9+### Token types and API
10+
11+See [spec.md — Token API](spec.md#token-api) for the full type definitions (`InlineSpan`, `BlockToken`) and function signatures (`tokenize`, `createTokenizer`).
4212
4313 ### Refactor steps
4414
45-1. Extract `parseInline` to return `InlineSpan[]` instead of an ANSI string
46-2. Extract `parseLine` to return `BlockToken` instead of an ANSI string
47-3. Add `tokenize(input)` — splits into lines, calls parseLine, returns `BlockToken[]`
48-4. Add `createTokenizer()` — same streaming model as `createHighlighter` but returns tokens
49-5. Rewrite `createHighlighter` to use `createTokenizer` internally + theme rendering on top
50-6. Export token types and tokenizer functions from `src/index.ts`
51-7. The ANSI renderer (`createHighlighter`, `highlightSync`) keeps its existing API and deps (`yoctocolors`, `string-width`, `strip-ansi`) — unchanged behavior
52-8. The token API (`tokenize`, `createTokenizer`, types) has **zero dependencies** — pure parsing
15+- [x] Extract `parseInline` to return `InlineSpan[]` instead of an ANSI string
16+- [x] Extract `parseLine` to return `BlockToken` instead of an ANSI string
17+- [x] Add `tokenize(input)` — splits into lines, calls parseLine, returns `BlockToken[]`
18+- [x] Add `createTokenizer()` — same streaming model as `createHighlighter` but returns tokens
19+- [x] Rewrite `createHighlighter` to use tokenizer's `parseLineToken` + theme rendering on top
20+- [x] Export token types and tokenizer functions from `src/index.ts`
21+- [x] The ANSI renderer (`createHighlighter`, `highlightSync`) keeps its existing API and deps (`yoctocolors`, `string-width`, `strip-ansi`) — unchanged behavior
22+- [x] The token API (`tokenize`, `createTokenizer`, types) has **zero dependencies** — pure parsing (`src/tokenizer.ts`)
5323
5424 ### Constraints
5525
56-- Existing `highlightSync` and `createHighlighter` APIs must not change (backward compatible)
57-- Streaming behavior must be preserved — `createTokenizer().write()` emits completed block tokens immediately, buffers only the current incomplete line
58-- Unclosed code blocks emit as `code_block` on `end()`
59-- Unclosed inline formatting falls back to plain `text` spans
60-- All existing tests must continue to pass
61-- Update spec.md to document the new token API
26+- [x] Existing `highlightSync` and `createHighlighter` APIs must not change (backward compatible)
27+- [x] Streaming behavior must be preserved — `createTokenizer().write()` emits completed block tokens immediately, buffers only the current incomplete line
28+- [x] Unclosed code blocks emit as `code_block` on `end()`
29+- [x] Unclosed inline formatting falls back to plain `text` spans
30+- [x] All existing tests must continue to pass (11 original + 39 new = 50 total)
31+- [x] Update spec.md to document the new token API
examples/streaming-test.ts+21−38
@@ -41,85 +41,68 @@ This is a longer paragraph that should cause terminal wrapping when the terminal
4141
4242 const delay = (ms: number) => new Promise((r) => setTimeout(r, ms));
4343
44-async function streamWithPrefix(prefix: string, text: string) {
44+async function streamChars(text: string, prefix = "") {
4545 const highlighter = createHighlighter({ theme: "truecolor" });
4646
47- // Write the prefix directly — it's not part of the highlighter output
48- process.stdout.write(prefix);
47+ if (prefix) process.stdout.write(prefix);
4948
50- // Stream character by character with a small delay to make it visible
51- for (let i = 0; i < text.length; i++) {
52- const output = highlighter.write(text[i]!);
49+ for (const ch of text) {
50+ const output = highlighter.write(ch);
5351 if (output) process.stdout.write(output);
5452 await delay(8);
5553 }
5654
57- // Flush remaining
5855 const final = highlighter.end();
5956 if (final) process.stdout.write(final);
60-
6157 process.stdout.write("\n");
6258 }
6359
6460 async function main() {
6561 // 1. Standard full-document streaming
66- const highlighter = createHighlighter({ theme: "truecolor" });
67-
68- for (let i = 0; i < markdown.length; i++) {
69- const output = highlighter.write(markdown[i]!);
70- if (output) process.stdout.write(output);
71- await delay(8);
72- }
73-
74- const final = highlighter.end();
75- if (final) process.stdout.write(final);
76- process.stdout.write("\n");
62+ await streamChars(markdown);
7763
7864 // 2. Inline prefix preservation — short line (should NOT erase prefix)
79- await streamWithPrefix("AI: ", "Here is some **bold** and *italic* text.");
65+ await streamChars("Here is some **bold** and *italic* text.", "AI: ");
8066
8167 // 3. Inline prefix preservation — long line that wraps
82- await streamWithPrefix(
83- "Assistant: ",
68+ await streamChars(
8469 "This is a much longer response that should wrap across multiple terminal lines to test the edge case where cursor-back might overshoot past the prefix on the first visual line.",
70+ "Assistant: ",
8571 );
8672
8773 // 4. Multiple prefixed responses in sequence
88- await streamWithPrefix("Q: ", "What is `yoctomarkdown`?");
89- await streamWithPrefix(
90- "A: ",
74+ await streamChars("What is `yoctomarkdown`?", "Q: ");
75+ await streamChars(
9176 "A **tiny** streaming markdown highlighter for the terminal.",
77+ "A: ",
9278 );
9379
94- // 5. Unicode width edge cases — would have been broken before string-width
80+ // 5. Unicode width edge cases
9581 // CJK wide characters (each takes 2 cells)
96- await streamWithPrefix(
97- "CJK: ",
98- "日本語のテスト — **太字** and `コード` inline.",
99- );
82+ await streamChars("日本語のテスト — **太字** and `コード` inline.", "CJK: ");
10083
10184 // Emoji sequences (ZWJ family = 1 grapheme, 2 cells; skin-toned emoji)
102- await streamWithPrefix(
103- "Emoji: ",
85+ await streamChars(
10486 "Family 👨‍👩‍👧‍👦 and thumbs 👍🏽 and flag 🇯🇵 with **bold** after.",
87+ "Emoji: ",
10588 );
10689
10790 // Ambiguous-width characters (★ ① ☆ ♨ — EAW category "A")
108- await streamWithPrefix(
109- "Ambiguous: ",
91+ await streamChars(
11092 "Stars ★☆★☆★ and circled ①②③④⑤ and `♨ hotspring` done.",
93+ "Ambiguous: ",
11194 );
11295
11396 // Combining marks (café with combining é = 4 visible chars)
114- await streamWithPrefix(
115- "Combining: ",
97+ await streamChars(
11698 "A cafe\u0301 with re\u0301sume\u0301 — **nai\u0308ve** text with `combining` marks.",
99+ "Combining: ",
117100 );
118101
119102 // Mixed: CJK + emoji + ambiguous + wrapping
120- await streamWithPrefix(
121- "Mixed: ",
103+ await streamChars(
122104 "漢字と★emoji👨‍👩‍👧‍👦とambiguous①②③が混ざった**長い文**で折り返しテスト。This line mixes scripts to stress-test `visibleLength` on wrapped redraws.",
105+ "Mixed: ",
123106 );
124107 }
125108
spec.md+49−0
@@ -160,6 +160,55 @@ Behavior:
160160 - `write()` returns rendered output for the provided chunk
161161 - `end()` flushes any remaining buffered text
162162
163+## Token API
164+
165+The library MUST also expose an ANSI-free token API with **zero runtime dependencies**.
166+
167+### Token types
168+
169+```ts
170+type InlineSpan =
171+ | { type: "text"; content: string }
172+ | { type: "bold"; content: string }
173+ | { type: "italic"; content: string }
174+ | { type: "code"; content: string }
175+ | { type: "link"; text: string; url: string };
176+
177+type BlockToken =
178+ | { type: "heading"; depth: 1 | 2 | 3; spans: InlineSpan[] }
179+ | { type: "paragraph"; spans: InlineSpan[] }
180+ | { type: "code_block"; lang: string; content: string }
181+ | { type: "list_item"; ordered: boolean; index: number; spans: InlineSpan[] }
182+ | { type: "blockquote"; spans: InlineSpan[] }
183+ | { type: "hr" }
184+ | { type: "blank" };
185+```
186+
187+### Functions
188+
189+```ts
190+tokenize(input: string): BlockToken[]
191+createTokenizer(): TokenHighlighter
192+```
193+
194+```ts
195+interface TokenHighlighter {
196+ write(chunk: string | Uint8Array): BlockToken[];
197+ end(): BlockToken[];
198+}
199+```
200+
201+Behavior:
202+
203+- `tokenize()` tokenizes a complete Markdown string into block tokens
204+- `createTokenizer()` returns an incremental tokenizer for streaming use
205+- `write()` returns completed block tokens for the provided chunk
206+- `end()` flushes any remaining buffered text and unclosed constructs
207+- Streaming `write()` emits completed block tokens immediately; only the current incomplete line is buffered
208+- Unclosed fenced code blocks emit as `code_block` on `end()`
209+- Unclosed inline formatting falls back to plain `text` spans
210+- Inline code has priority over other inline formatting
211+
163212 ## CLI
164213
165214 The CLI command is:
src/index.ts+103−93
@@ -1,6 +1,10 @@
11 import stripAnsi from "strip-ansi";
22 import stringWidth from "string-width";
33 import { type Theme, themes } from "./themes";
4+import { type InlineSpan, parseLineToken } from "./tokenizer";
5+
6+export type { InlineSpan, BlockToken, TokenHighlighter } from "./tokenizer";
7+export { tokenize, createTokenizer } from "./tokenizer";
48
59 export interface Options {
610 theme?: "16" | "256" | "truecolor" | "minimal" | "none" | Theme;
@@ -36,6 +40,97 @@ function resolveTheme(options?: Options): Theme {
3640 return t as Theme;
3741 }
3842
43+// ── Token → ANSI rendering ──────────────────────────────────────────
44+
45+function renderSpans(spans: InlineSpan[], theme: Theme): string {
46+ let result = "";
47+ for (const span of spans) {
48+ switch (span.type) {
49+ case "text":
50+ result += span.content;
51+ break;
52+ case "bold":
53+ result += theme.bold(`**${span.content}**`);
54+ break;
55+ case "italic":
56+ result += theme.italic(`*${span.content}*`);
57+ break;
58+ case "code":
59+ result += theme.code(`\`${span.content}\``);
60+ break;
61+ case "link":
62+ result += theme.link(`[${span.text}](${span.url})`);
63+ break;
64+ }
65+ }
66+ return result;
67+}
68+
69+function renderLine(
70+ line: string,
71+ inCodeBlock: boolean,
72+ isPartial: boolean,
73+ theme: Theme,
74+): { rendered: string; newCodeBlockState: boolean } {
75+ // Inside a code block — render as code
76+ if (inCodeBlock) {
77+ const closing = line.trim().startsWith("```");
78+ return {
79+ rendered: theme.codeBlock(line),
80+ newCodeBlockState: !(closing && !isPartial),
81+ };
82+ }
83+
84+ // Code block opening fence
85+ if (line.trim().startsWith("```")) {
86+ return {
87+ rendered: theme.codeBlock(line),
88+ newCodeBlockState: !isPartial,
89+ };
90+ }
91+
92+ // For non-code lines, tokenize and render
93+ const token = parseLineToken(line, false);
94+ let rendered: string;
95+ switch (token.type) {
96+ case "hr":
97+ rendered = theme.hr(line);
98+ break;
99+ case "heading": {
100+ const hashes = "#".repeat(token.depth);
101+ rendered = theme.heading(`${hashes} ${renderSpans(token.spans, theme)}`);
102+ break;
103+ }
104+ case "blockquote": {
105+ const bqMatch = line.match(/^(>\s+)/);
106+ rendered =
107+ theme.blockquote(bqMatch?.[1] ?? "> ") +
108+ renderSpans(token.spans, theme);
109+ break;
110+ }
111+ case "list_item": {
112+ const markerMatch = line.match(/^((?:[-*]|\d+\.)\s+)/);
113+ rendered =
114+ theme.listMarker(markerMatch?.[1] ?? "") +
115+ renderSpans(token.spans, theme);
116+ break;
117+ }
118+ case "blank":
119+ rendered = "";
120+ break;
121+ case "paragraph":
122+ rendered = renderSpans(token.spans, theme);
123+ break;
124+ default:
125+ rendered = line;
126+ break;
127+ }
128+
129+ return { rendered, newCodeBlockState: false };
130+}
131+
132+// ── Public API ──────────────────────────────────────────────────────
133+
39134 export function highlightSync(input: string, options?: Options): string {
40135 const highlighter = createHighlighter(options);
41136 const out = highlighter.write(input);
@@ -49,100 +144,15 @@ export function createHighlighter(options?: Options): Highlighter {
49144 let buffer = "";
50145 let inCodeBlock = false;
51146
52- function parseInline(text: string): string {
53- let result = "";
54- let i = 0;
55- while (i < text.length) {
56- if (text.startsWith("`", i) && !text.startsWith("``", i)) {
57- const end = text.indexOf("`", i + 1);
58- if (end !== -1) {
59- result += theme.code(`\`${text.slice(i + 1, end)}\``);
60- i = end + 1;
61- continue;
62- }
63- }
64-
65- if (text.startsWith("**", i)) {
66- const end = text.indexOf("**", i + 2);
67- if (end !== -1) {
68- result += theme.bold(`**${text.slice(i + 2, end)}**`);
69- i = end + 2;
70- continue;
71- }
72- }
73-
74- if (text.startsWith("*", i) && !text.startsWith("**", i)) {
75- const end = text.indexOf("*", i + 1);
76- if (end !== -1 && !text.startsWith("**", end)) {
77- result += theme.italic(`*${text.slice(i + 1, end)}*`);
78- i = end + 1;
79- continue;
80- }
81- }
82-
83- if (text.startsWith("[", i)) {
84- const bracketEnd = text.indexOf("]", i + 1);
85- if (bracketEnd !== -1 && text.charAt(bracketEnd + 1) === "(") {
86- const parenEnd = text.indexOf(")", bracketEnd + 2);
87- if (parenEnd !== -1) {
88- result += theme.link(
89- `[${text.slice(i + 1, bracketEnd)}](${text.slice(bracketEnd + 2, parenEnd)})`,
90- );
91- i = parenEnd + 1;
92- continue;
93- }
94- }
95- }
96-
97- result += text[i];
98- i++;
99- }
100-
101- return result;
102- }
103-
104147 function parseLine(line: string, isPartial: boolean): string {
105- if (inCodeBlock) {
106- if (line.trim().startsWith("```")) {
107- if (!isPartial) inCodeBlock = false;
108- return theme.codeBlock(line);
109- }
110- return theme.codeBlock(line);
111- }
112-
113- if (line.trim().startsWith("```")) {
114- if (!isPartial) inCodeBlock = true;
115- return theme.codeBlock(line);
116- }
117-
118- const hrMatch = line.trim().match(/^(?:-{3,}|\*{3,}|_{3,})$/);
119- if (hrMatch) {
120- return theme.hr(line);
121- }
122-
123- const headingMatch = line.match(/^(#{1,3})\s+(.*)$/);
124- if (headingMatch) {
125- const hashes = headingMatch[1] ?? "";
126- const text = headingMatch[2] ?? "";
127- return theme.heading(`${hashes} ${parseInline(text)}`);
128- }
129-
130- const bqMatch = line.match(/^(>\s+)(.*)$/);
131- if (bqMatch) {
132- return theme.blockquote(bqMatch[1] ?? "") + parseInline(bqMatch[2] ?? "");
133- }
134-
135- const ulMatch = line.match(/^([-*]\s+)(.*)$/);
136- if (ulMatch) {
137- return theme.listMarker(ulMatch[1] ?? "") + parseInline(ulMatch[2] ?? "");
138- }
139-
140- const olMatch = line.match(/^(\d+\.\s+)(.*)$/);
141- if (olMatch) {
142- return theme.listMarker(olMatch[1] ?? "") + parseInline(olMatch[2] ?? "");
143- }
144-
145- return parseInline(line);
148+ const { rendered, newCodeBlockState } = renderLine(
149+ line,
150+ inCodeBlock,
151+ isPartial,
152+ theme,
153+ );
154+ inCodeBlock = newCodeBlockState;
155+ return rendered;
146156 }
147157
148158 let previousPartialRendered = "";
src/tokenizer.ts+260−0
@@ -0,0 +1,260 @@
1+// Pure token-based Markdown parser — zero dependencies.
2+
3+export type InlineSpan =
4+ | { type: "text"; content: string }
5+ | { type: "bold"; content: string }
6+ | { type: "italic"; content: string }
7+ | { type: "code"; content: string }
8+ | { type: "link"; text: string; url: string };
9+
10+export type BlockToken =
11+ | { type: "heading"; depth: 1 | 2 | 3; spans: InlineSpan[] }
12+ | { type: "paragraph"; spans: InlineSpan[] }
13+ | { type: "code_block"; lang: string; content: string }
14+ | { type: "list_item"; ordered: boolean; index: number; spans: InlineSpan[] }
15+ | { type: "blockquote"; spans: InlineSpan[] }
16+ | { type: "hr" }
17+ | { type: "blank" };
18+
19+export interface TokenHighlighter {
20+ write(chunk: string | Uint8Array): BlockToken[];
21+ end(): BlockToken[];
22+}
23+
24+// ── Inline parsing ───────────────────────────────────────────────────
25+
26+function parseInlineTokens(text: string): InlineSpan[] {
27+ const spans: InlineSpan[] = [];
28+ let i = 0;
29+ let plainStart = i;
30+
31+ function flushPlain() {
32+ if (plainStart < i) {
33+ spans.push({ type: "text", content: text.slice(plainStart, i) });
34+ }
35+ }
36+
37+ while (i < text.length) {
38+ // Inline code (highest priority)
39+ if (text[i] === "`" && text[i + 1] !== "`") {
40+ const end = text.indexOf("`", i + 1);
41+ if (end !== -1) {
42+ flushPlain();
43+ spans.push({ type: "code", content: text.slice(i + 1, end) });
44+ i = end + 1;
45+ plainStart = i;
46+ continue;
47+ }
48+ }
49+
50+ // Bold **text**
51+ if (text[i] === "*" && text[i + 1] === "*") {
52+ const end = text.indexOf("**", i + 2);
53+ if (end !== -1) {
54+ flushPlain();
55+ spans.push({ type: "bold", content: text.slice(i + 2, end) });
56+ i = end + 2;
57+ plainStart = i;
58+ continue;
59+ }
60+ }
61+
62+ // Italic *text* (not preceded by another *)
63+ if (text[i] === "*" && text[i + 1] !== "*") {
64+ const end = text.indexOf("*", i + 1);
65+ if (end !== -1 && text[end + 1] !== "*") {
66+ flushPlain();
67+ spans.push({ type: "italic", content: text.slice(i + 1, end) });
68+ i = end + 1;
69+ plainStart = i;
70+ continue;
71+ }
72+ }
73+
74+ // Link [text](url)
75+ if (text[i] === "[") {
76+ const bracketEnd = text.indexOf("]", i + 1);
77+ if (bracketEnd !== -1 && text[bracketEnd + 1] === "(") {
78+ const parenEnd = text.indexOf(")", bracketEnd + 2);
79+ if (parenEnd !== -1) {
80+ flushPlain();
81+ spans.push({
82+ type: "link",
83+ text: text.slice(i + 1, bracketEnd),
84+ url: text.slice(bracketEnd + 2, parenEnd),
85+ });
86+ i = parenEnd + 1;
87+ plainStart = i;
88+ continue;
89+ }
90+ }
91+ }
92+
93+ i++;
94+ }
95+
96+ // Remaining plain text
97+ if (plainStart < text.length) {
98+ spans.push({ type: "text", content: text.slice(plainStart) });
99+ }
100+
101+ return spans;
102+}
103+
104+// ── Block parsing ────────────────────────────────────────────────────
105+
106+export function parseLineToken(line: string, inCodeBlock: boolean): BlockToken {
107+ if (inCodeBlock) {
108+ // This function doesn't handle code block accumulation — that's
109+ // done at the higher level. This shouldn't be called for lines
110+ // inside a code block. Kept as a safety fallback.
111+ return { type: "paragraph", spans: [{ type: "text", content: line }] };
112+ }
113+
114+ // Fenced code block start — handled at higher level
115+ if (line.trim().startsWith("```")) {
116+ // Shouldn't normally reach here; the higher-level handles code blocks
117+ return { type: "paragraph", spans: [{ type: "text", content: line }] };
118+ }
119+
120+ // Horizontal rule
121+ if (/^(?:-{3,}|\*{3,}|_{3,})$/.test(line.trim())) {
122+ return { type: "hr" };
123+ }
124+
125+ // Heading
126+ const headingMatch = line.match(/^(#{1,3})\s+(.*)$/);
127+ if (headingMatch) {
128+ const depth = (headingMatch[1]?.length ?? 1) as 1 | 2 | 3;
129+ const text = headingMatch[2] ?? "";
130+ return { type: "heading", depth, spans: parseInlineTokens(text) };
131+ }
132+
133+ // Blockquote
134+ const bqMatch = line.match(/^>\s+(.*)$/);
135+ if (bqMatch) {
136+ return { type: "blockquote", spans: parseInlineTokens(bqMatch[1] ?? "") };
137+ }
138+
139+ // Unordered list
140+ const ulMatch = line.match(/^[-*]\s+(.*)$/);
141+ if (ulMatch) {
142+ return {
143+ type: "list_item",
144+ ordered: false,
145+ index: 0,
146+ spans: parseInlineTokens(ulMatch[1] ?? ""),
147+ };
148+ }
149+
150+ // Ordered list
151+ const olMatch = line.match(/^(\d+)\.\s+(.*)$/);
152+ if (olMatch) {
153+ return {
154+ type: "list_item",
155+ ordered: true,
156+ index: parseInt(olMatch[1] ?? "0", 10),
157+ spans: parseInlineTokens(olMatch[2] ?? ""),
158+ };
159+ }
160+
161+ // Blank line
162+ if (line.trim() === "") {
163+ return { type: "blank" };
164+ }
165+
166+ // Plain paragraph
167+ return { type: "paragraph", spans: parseInlineTokens(line) };
168+}
169+
170+// ── Batch tokenizer ─────────────────────────────────────────────────
171+
172+export function tokenize(input: string): BlockToken[] {
173+ const tokenizer = createTokenizer();
174+ const tokens = tokenizer.write(input);
175+ tokens.push(...tokenizer.end());
176+ return tokens;
177+}
178+
179+// ── Streaming tokenizer ─────────────────────────────────────────────
180+
181+export function createTokenizer(): TokenHighlighter {
182+ let buffer = "";
183+ let inCodeBlock = false;
184+ let codeLang = "";
185+ let codeLines: string[] = [];
186+
187+ function processLine(line: string): BlockToken | null {
188+ // Inside a code block
189+ if (inCodeBlock) {
190+ if (line.trim().startsWith("```")) {
191+ inCodeBlock = false;
192+ const token: BlockToken = {
193+ type: "code_block",
194+ lang: codeLang,
195+ content: codeLines.join("\n"),
196+ };
197+ codeLang = "";
198+ codeLines = [];
199+ return token;
200+ }
201+ codeLines.push(line);
202+ return null;
203+ }
204+
205+ // Start a code block
206+ if (line.trim().startsWith("```")) {
207+ inCodeBlock = true;
208+ const langMatch = line.trim().match(/^```(\S*)$/);
209+ codeLang = langMatch?.[1] ?? "";
210+ codeLines = [];
211+ return null;
212+ }
213+
214+ return parseLineToken(line, false);
215+ }
216+
217+ return {
218+ write(chunk: string | Uint8Array): BlockToken[] {
219+ const text =
220+ typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk);
221+ buffer += text;
222+
223+ const lines = buffer.split("\n");
224+ buffer = lines.pop() ?? "";
225+
226+ const tokens: BlockToken[] = [];
227+ for (const line of lines) {
228+ const token = processLine(line);
229+ if (token) tokens.push(token);
230+ }
231+
232+ return tokens;
233+ },
234+
235+ end(): BlockToken[] {
236+ const tokens: BlockToken[] = [];
237+
238+ // Flush remaining buffer
239+ if (buffer.length > 0) {
240+ const token = processLine(buffer);
241+ if (token) tokens.push(token);
242+ buffer = "";
243+ }
244+
245+ // Flush unclosed code block
246+ if (inCodeBlock) {
247+ tokens.push({
248+ type: "code_block",
249+ lang: codeLang,
250+ content: codeLines.join("\n"),
251+ });
252+ inCodeBlock = false;
253+ codeLang = "";
254+ codeLines = [];
255+ }
256+
257+ return tokens;
258+ },
259+ };
260+}
tests/tokenizer.test.ts+513−0
@@ -0,0 +1,513 @@
1+import { test, expect, describe } from "bun:test";
2+import { tokenize, createTokenizer, type BlockToken } from "../src/index";
3+
4+// ── Inline span parsing ──────────────────────────────────────────────
5+
6+describe("tokenize: inline spans", () => {
7+ test("plain text", () => {
8+ const tokens = tokenize("Hello world");
9+ expect(tokens).toEqual([
10+ { type: "paragraph", spans: [{ type: "text", content: "Hello world" }] },
11+ ]);
12+ });
13+
14+ test("bold", () => {
15+ const tokens = tokenize("some **bold** text");
16+ expect(tokens).toEqual([
17+ {
18+ type: "paragraph",
19+ spans: [
20+ { type: "text", content: "some " },
21+ { type: "bold", content: "bold" },
22+ { type: "text", content: " text" },
23+ ],
24+ },
25+ ]);
26+ });
27+
28+ test("italic", () => {
29+ const tokens = tokenize("some *italic* text");
30+ expect(tokens).toEqual([
31+ {
32+ type: "paragraph",
33+ spans: [
34+ { type: "text", content: "some " },
35+ { type: "italic", content: "italic" },
36+ { type: "text", content: " text" },
37+ ],
38+ },
39+ ]);
40+ });
41+
42+ test("inline code", () => {
43+ const tokens = tokenize("use `foo()` here");
44+ expect(tokens).toEqual([
45+ {
46+ type: "paragraph",
47+ spans: [
48+ { type: "text", content: "use " },
49+ { type: "code", content: "foo()" },
50+ { type: "text", content: " here" },
51+ ],
52+ },
53+ ]);
54+ });
55+
56+ test("link", () => {
57+ const tokens = tokenize("click [here](https://example.com) now");
58+ expect(tokens).toEqual([
59+ {
60+ type: "paragraph",
61+ spans: [
62+ { type: "text", content: "click " },
63+ { type: "link", text: "here", url: "https://example.com" },
64+ { type: "text", content: " now" },
65+ ],
66+ },
67+ ]);
68+ });
69+
70+ test("mixed inline formatting", () => {
71+ const tokens = tokenize("**bold** and *italic* and `code`");
72+ expect(tokens).toEqual([
73+ {
74+ type: "paragraph",
75+ spans: [
76+ { type: "bold", content: "bold" },
77+ { type: "text", content: " and " },
78+ { type: "italic", content: "italic" },
79+ { type: "text", content: " and " },
80+ { type: "code", content: "code" },
81+ ],
82+ },
83+ ]);
84+ });
85+
86+ test("unclosed bold falls back to text", () => {
87+ const tokens = tokenize("some **unclosed bold");
88+ expect(tokens).toEqual([
89+ {
90+ type: "paragraph",
91+ spans: [{ type: "text", content: "some **unclosed bold" }],
92+ },
93+ ]);
94+ });
95+
96+ test("unclosed italic falls back to text", () => {
97+ const tokens = tokenize("some *unclosed italic");
98+ expect(tokens).toEqual([
99+ {
100+ type: "paragraph",
101+ spans: [{ type: "text", content: "some *unclosed italic" }],
102+ },
103+ ]);
104+ });
105+
106+ test("unclosed inline code falls back to text", () => {
107+ const tokens = tokenize("some `unclosed code");
108+ expect(tokens).toEqual([
109+ {
110+ type: "paragraph",
111+ spans: [{ type: "text", content: "some `unclosed code" }],
112+ },
113+ ]);
114+ });
115+
116+ test("inline code takes priority over bold", () => {
117+ const tokens = tokenize("`**not bold**`");
118+ expect(tokens).toEqual([
119+ {
120+ type: "paragraph",
121+ spans: [{ type: "code", content: "**not bold**" }],
122+ },
123+ ]);
124+ });
125+});
126+
127+// ── Block-level tokens ───────────────────────────────────────────────
128+
129+describe("tokenize: block tokens", () => {
130+ test("heading 1", () => {
131+ const tokens = tokenize("# Hello");
132+ expect(tokens).toEqual([
133+ {
134+ type: "heading",
135+ depth: 1,
136+ spans: [{ type: "text", content: "Hello" }],
137+ },
138+ ]);
139+ });
140+
141+ test("heading 2", () => {
142+ const tokens = tokenize("## Sub");
143+ expect(tokens).toEqual([
144+ {
145+ type: "heading",
146+ depth: 2,
147+ spans: [{ type: "text", content: "Sub" }],
148+ },
149+ ]);
150+ });
151+
152+ test("heading 3", () => {
153+ const tokens = tokenize("### Third");
154+ expect(tokens).toEqual([
155+ {
156+ type: "heading",
157+ depth: 3,
158+ spans: [{ type: "text", content: "Third" }],
159+ },
160+ ]);
161+ });
162+
163+ test("heading with inline formatting", () => {
164+ const tokens = tokenize("# **Bold** heading");
165+ expect(tokens).toEqual([
166+ {
167+ type: "heading",
168+ depth: 1,
169+ spans: [
170+ { type: "bold", content: "Bold" },
171+ { type: "text", content: " heading" },
172+ ],
173+ },
174+ ]);
175+ });
176+
177+ test("fenced code block", () => {
178+ const tokens = tokenize("```js\nconst x = 1;\n```");
179+ expect(tokens).toEqual([
180+ { type: "code_block", lang: "js", content: "const x = 1;" },
181+ ]);
182+ });
183+
184+ test("fenced code block without language", () => {
185+ const tokens = tokenize("```\nhello\n```");
186+ expect(tokens).toEqual([
187+ { type: "code_block", lang: "", content: "hello" },
188+ ]);
189+ });
190+
191+ test("fenced code block with multiple lines", () => {
192+ const tokens = tokenize("```\nline1\nline2\nline3\n```");
193+ expect(tokens).toEqual([
194+ { type: "code_block", lang: "", content: "line1\nline2\nline3" },
195+ ]);
196+ });
197+
198+ test("unclosed code block emits as code_block on end", () => {
199+ const tokens = tokenize("```\nunclosed code");
200+ expect(tokens).toEqual([
201+ { type: "code_block", lang: "", content: "unclosed code" },
202+ ]);
203+ });
204+
205+ test("unordered list item with dash", () => {
206+ const tokens = tokenize("- Item one");
207+ expect(tokens).toEqual([
208+ {
209+ type: "list_item",
210+ ordered: false,
211+ index: 0,
212+ spans: [{ type: "text", content: "Item one" }],
213+ },
214+ ]);
215+ });
216+
217+ test("unordered list item with asterisk", () => {
218+ const tokens = tokenize("* Item two");
219+ expect(tokens).toEqual([
220+ {
221+ type: "list_item",
222+ ordered: false,
223+ index: 0,
224+ spans: [{ type: "text", content: "Item two" }],
225+ },
226+ ]);
227+ });
228+
229+ test("ordered list item", () => {
230+ const tokens = tokenize("1. First");
231+ expect(tokens).toEqual([
232+ {
233+ type: "list_item",
234+ ordered: true,
235+ index: 1,
236+ spans: [{ type: "text", content: "First" }],
237+ },
238+ ]);
239+ });
240+
241+ test("ordered list preserves index", () => {
242+ const tokens = tokenize("3. Third");
243+ expect(tokens).toEqual([
244+ {
245+ type: "list_item",
246+ ordered: true,
247+ index: 3,
248+ spans: [{ type: "text", content: "Third" }],
249+ },
250+ ]);
251+ });
252+
253+ test("blockquote", () => {
254+ const tokens = tokenize("> quoted text");
255+ expect(tokens).toEqual([
256+ {
257+ type: "blockquote",
258+ spans: [{ type: "text", content: "quoted text" }],
259+ },
260+ ]);
261+ });
262+
263+ test("blockquote with inline formatting", () => {
264+ const tokens = tokenize("> **bold** quote");
265+ expect(tokens).toEqual([
266+ {
267+ type: "blockquote",
268+ spans: [
269+ { type: "bold", content: "bold" },
270+ { type: "text", content: " quote" },
271+ ],
272+ },
273+ ]);
274+ });
275+
276+ test("horizontal rule ---", () => {
277+ const tokens = tokenize("---");
278+ expect(tokens).toEqual([{ type: "hr" }]);
279+ });
280+
281+ test("horizontal rule ***", () => {
282+ const tokens = tokenize("***");
283+ expect(tokens).toEqual([{ type: "hr" }]);
Cut short: this commit's diff is over 1000 lines.