@@ -1,557 +0,0 @@
1-# PROGRESS
2-
3-## Context
4-
5-We are unhappy with the current state of syntax highlighting in our stack.
6-
7-The problem does **not** appear to be a `cel-tui` limitation. The real issue is the syntax-highlighting ecosystem underneath it: most libraries are either renderer-first, HTML-first, theme-first, editor-specific, or not designed well for streamed / incremental tokenization.
8-
9-That matters for us because we want highlighting that is:
10-
11-- modern
12-- fast
13-- correct
14-- stream-friendly
15-- renderer-agnostic
16-- usable from Bun
17-- suitable for TUIs, editors, markdown renderers, and other apps
18-
19-## Why this came up
20-
21-We already tried `shiki-stream` in the past and did not get good results with Bun. That might have been integration pain, but it also exposed a bigger issue: the ecosystem does not have a clear, boring, low-level library that treats **streamed tokenization** as the primary job.
22-
23-So the idea is to make a new library in this workspace that solves that problem directly.
24-
25-## What we looked at
26-
27-We did a quick comparison of the current options that are adjacent to this problem.
28-
29-### TextMate / Shiki side
30-
31-#### `shiki-stream`
32-
33-Pros:
34-
35-- closest existing library to the exact streamed-highlighting problem
36-- models the fact that future input can change earlier highlighting
37-- supports recall tokens for unstable output
38-
39-Cons:
40-
41-- still shaped around the Shiki stack
42-- themed-token oriented instead of being a clean base tokenizer
43-- feels more like a stream renderer helper than a foundational token engine
44-- Bun compatibility / ergonomics were not great for us in practice
45-
46-#### `shiki`
47-
48-Pros:
49-
50-- high quality highlighting
51-- mature ecosystem
52-- has `GrammarState`, `getLastGrammarState`, and token APIs
53-- portable in principle and documented as Bun-compatible
54-
55-Cons:
56-
57-- API is still strongly renderer / theme oriented
58-- not ideal if what we really want is raw tokenization and stream state
59-- high-level abstraction is great for output, less great as a base engine
60-
61-#### `vscode-textmate`
62-
63-Pros:
64-
65-- probably the best low-level TextMate baseline
66-- already exposes the right mental model for streaming:
67- - tokenize a line
68- - keep the state stack
69- - continue from there
70-- cleaner tokenizer primitive than Shiki for this use case
71-
72-Cons:
73-
74-- low-level and raw
75-- manual grammar loading is annoying
76-- Oniguruma / WASM setup adds friction
77-- not a modern, batteries-included stream-token API
78-
79-#### `starry-night`
80-
81-Pros:
82-
83-- high quality GitHub-style output
84-- good reference point for output quality
85-- AST/HAST-first instead of HTML-string-only
86-
87-Cons:
88-
89-- not really a streaming token engine
90-- more output-oriented than tokenizer-oriented
91-
92-### Regex / classic highlighters
93-
94-#### `highlight.js`, `lowlight`, `Prism`, `refractor`
95-
96-Pros:
97-
98-- easy to adopt
99-- portable
100-- some provide AST/token structures instead of raw HTML
101-
102-Cons:
103-
104-- correctness ceiling is lower
105-- not compelling if the core complaint is poor highlighting quality
106-- generally not built around streamed, stateful tokenization
107-
108-These are useful libraries, but they do not look like the right foundation for a new "fast and correct streamed highlighting" core.
109-
110-### Parser / incremental side
111-
112-#### `web-tree-sitter`
113-
114-Pros:
115-
116-- real incremental parsing
117-- much better long-term story for editable documents
118-- can produce structured ranges from syntax queries
119-- strongest architecture for true incremental correctness
120-
121-Cons:
122-
123-- more plumbing
124-- more assets and setup
125-- grammar/query quality varies by language
126-- less turnkey as a standalone highlighting engine in JS/Bun apps
127-
128-#### `CodeMirror` / `Lezer`
129-
130-Pros:
131-
132-- clean incremental design
133-- strong API ideas around syntax tags and highlighting
134-- useful reference for architecture
135-
136-Cons:
137-
138-- smaller grammar ecosystem than TextMate
139-- not the obvious drop-in base for broad language coverage
140-
141-#### `Monaco`
142-
143-Pros:
144-
145-- good reference API for line tokenization with carry-forward state
146-- proves the usefulness of a tokenizer interface shaped around `line -> tokens + endState`
147-
148-Cons:
149-
150-- not the base library we want to depend on
151-- editor-focused rather than tokenizer-first
152-
153-## Ecosystem problem summary
154-
155-Right now the ecosystem feels broken in a very specific way:
156-
157-- **High-quality stacks** are usually tied to heavy theming / HTML / editor assumptions.
158-- **Simple stacks** are usually easier to use but weaker in correctness.
159-- **Incremental stacks** exist, but often require much more setup and language-specific plumbing.
160-- **Streaming support** is usually an add-on, not the core abstraction.
161-- **Renderer-agnostic token streams** are surprisingly rare.
162-- **Bun-friendly, low-friction APIs** are not the norm.
163-
164-There is no obvious library that is simultaneously:
165-
166-- token-first
167-- stream-aware
168-- renderer-agnostic
169-- Bun-friendly
170-- high-quality across many languages
171-- suitable for both append-only streams and more general document workflows
172-
173-## Why we want a new base library
174-
175-The goal is **not** to build another syntax highlighter that only outputs HTML or only serves `cel-tui`.
176-
177-The goal is to build a base library that covers the wider use case:
178-
179-- TUIs
180-- editors
181-- markdown renderers
182-- streaming LLM/code output
183-- static renderers
184-- diff viewers
185-- any app that needs syntax-aware token ranges without being forced into one renderer
186-
187-That base layer should:
188-
189-- expose a clean token/range API
190-- make tokenizer state explicit
191-- handle unstable streaming output in an explicit way
192- - for example: stable chunks, recalls, invalidations, or similar
193-- avoid HTML assumptions
194-- avoid hard-wiring theme concerns into tokenization
195-- work well in Bun
196-- be useful even outside this repo
197-
198-## Direction
199-
200-The current thinking is:
201-
202-1. `vscode-textmate` is the strongest baseline if we want to stay in the TextMate world initially.
203-2. `shiki-stream` is useful as a comparison point for stream semantics, especially recalls.
204-3. `web-tree-sitter` is the strongest long-term alternative if we want true incremental parsing instead of line/state tokenization.
205-
206-That suggests a practical plan:
207-
208-- start with a clean, low-level, stream-aware tokenizer core
209-- keep rendering and theming as separate layers
210-- design the API so the backend can be TextMate-first
211-- keep room for other backends later if needed
212-
213-## Name
214-
215-The new library is called **`clew`** and the workspace package name is **`@cel-tui/clew`**.
216-
217-The name comes from the maze metaphor: parsing input is like solving a maze, and a **clew** is the guiding thread used to find the path through a labyrinth.
218-
219-## API decisions so far
220-
221-We decided to keep the public API intentionally simple.
222-
223-Primary entrypoint:
224-
225-```ts
226-clew(content, options) => ClewStream | ClewOutput
227-```
228-
229-Current direction:
230-
231-- **streaming is the default**
232-- sync mode is opt-in via `stream: false`
233-- sync mode should return final structured tokens
234-- stream mode should support very fine-grained input, including per-character writes
235-- stream mode should expose:
236- - `onChunk`
237- - `onCorrection`
238- - `write`
239- - `snapshot`
240- - `end`
241-
242-Default behavior:
243-
244-- `stream: true`
245-- `stability: "eager"`
246-
247-We explicitly chose **`eager` as the default** because this library is stream-first.
248-
249-## Correction model
250-
251-One important API decision is that corrections should be **range-based**, not token-count-based.
252-
253-In other words, corrections should say:
254-
255-> replace source range `from..to` with these tokens
256-
257-This is a better fit for a base library than a renderer-specific recall model, and it keeps the API general enough for:
258-
259-- TUIs
260-- editors
261-- HTML renderers
262-- markdown renderers
263-- diff tools
264-- per-character streaming consumers
265-
266-## Initial scope
267-
268-To get moving, `clew` started with **TypeScript only**.
269-
270-That initial goal is complete.
271-
272-We now have a real stream-first tokenizer package, a real consumer in
273-`SyntaxHighlight`, and several non-TypeScript languages integrated.
274-
275-## Desired properties for v0
276-
277-A first version should aim for:
278-
279-- append-only streaming done well
280-- explicit tokenizer state
281-- renderer-agnostic token events or token ranges
282-- eager streaming by default, with corrections when needed
283-- Bun-first developer experience
284-- enough generality that `cel-tui` is just the first consumer, not the whole point
285-
286-## Architecture now
287-
288-The current shape is intentionally simple:
289-
290-- `packages/clew/` is a real workspace package
291-- `clew(content, options)` is still the single public entrypoint
292-- the stream layer is generic and shared across languages
293-- language support lives behind a small internal registry
294-- each language module owns:
295- - `tokenize(content)`
296- - optional `stableBoundary(content, ended)`
297-- `SyntaxHighlight` is now a **`clew` renderer only**
298-- old `lowlight` / `lextide` fallback logic has been removed from `SyntaxHighlight`
299-- the TypeScript / JavaScript path now starts with the TypeScript lexical classifier and layers AST-based semantic overlays on top
300-
301-Current `clew` source layout:
302-
303-- `src/model.ts`
304-- `src/shared.ts`
305-- `src/registry.ts`
306-- `src/stream.ts`
307-- `src/languages/typescript.ts`
308-- `src/languages/python.ts`
309-- `src/languages/bash.ts`
310-- `src/languages/json.ts`
311-- `src/languages/markdown.ts`
312-
313-This is the structure new languages should follow.
314-
315-## Scope model
316-
317-We also cleaned up the theme contract.
318-
319-`SyntaxHighlight` themes no longer target backend-specific class names.
320-They now target canonical `clew` scopes directly.
321-
322-Examples:
323-
324-- `keyword`
325-- `string`
326-- `property`
327-- `link`
328-- `variable`
329-- `command`
330-- `builtin`
331-- `markup.heading`
332-- `meta.substitution.command`
333-- `meta.substitution.arithmetic`
334-- `string.heredoc`
335-
336-That is a much better long-term contract than `hljs-*` or other backend-shaped names.
337-
338-## Supported languages today
339-
340-`clew` currently supports:
341-
342-### TypeScript / JavaScript ids
343-
344-- `typescript`
345-- `ts`
346-- `tsx`
347-- `mts`
348-- `cts`
349-- `javascript`
350-- `js`
351-- `jsx`
352-- `mjs`
353-- `cjs`
354-
355-### Python
356-
357-- `python`
358-- `py`
359-
360-### Bash
361-
362-- `bash`
363-
364-This is intentionally **Bash only**.
365-
366-Not supported:
367-
368-- `sh`
369-- `zsh`
370-- `fish`
371-- `shell`
372-
373-### JSON
374-
375-- `json`
376-
377-### Markdown
378-
379-- `markdown`
380-
381-## Testing direction
382-
383-We now have a clearer testing split.
384-
385-### 1. Tokenizer correctness tests (`packages/clew/src/index.test.ts`)
386-
387-These verify:
388-
389-- token text joins back to the original source exactly
390-- tokens fully cover the source with no gaps or overlaps
391-- normalized token types/scopes for supported constructs
392-- supported-language registry behavior
393-
394-### 2. Streaming correctness tests (`packages/clew/src/index.test.ts`)
395-
396-These verify:
397-
398-- eager append-only chunk emission
399-- correction behavior when later input changes earlier tokenization
400-- stable mode boundary behavior
401-- final output equality across different chunk boundaries
402-- JSON correction behavior for late-closing strings
403-- Markdown final-output stability across streamed chunk boundaries
404-
405-### 3. Rendering correctness tests (`packages/components/src/syntax-highlight.test.ts`)
406-
407-These verify:
408-
409-- unknown-language plain-text fallback
410-- theme-to-scope mapping
411-- line preservation and wrapping behavior
412-- append-only stability through the component
413-- TypeScript semantic scope rendering
414-- Python decorator / builtin / property rendering
415-- Bash rendering cases like heredocs and substitutions
416-- JSON property/value styling
417-- Markdown heading, list, code, and link styling
418-
419-This is the testing model we should keep extending as new languages land.
420-
421-## Current status
422-
423-We are well past the planning-only phase.
424-
425-What exists now:
426-
427-- `PROGRESS.md` documents the problem, decisions, and current direction
428-- `packages/clew/` exists as a real workspace package
429-- `packages/clew/README.md` documents the current architecture and supported languages
430-- `packages/clew/src/index.ts` exposes the public API
431-- `packages/clew/src/registry.ts` provides language lookup
432-- `packages/clew/src/stream.ts` implements the shared streaming / diff / correction layer
433-- `packages/clew/src/languages/typescript.ts` implements the TypeScript / JavaScript tokenizer path with AST-based semantic overlays
434-- `packages/clew/src/languages/python.ts` implements the Python tokenizer path
435-- `packages/clew/src/languages/bash.ts` implements the Bash tokenizer path
436-- `packages/clew/src/languages/json.ts` implements the JSON tokenizer path
437-- `packages/clew/src/languages/markdown.ts` implements the Markdown tokenizer path
438-- `SyntaxHighlight` in `packages/components/src/syntax-highlight.ts` now uses **only `clew`**
439-- `@cel-tui/components` no longer depends on `lextide`
440-- `examples/syntax-highlight.ts` is now a component-focused demo built around the real public API
441-- the example currently cycles real TypeScript, JavaScript, Python, Bash, JSON, and Markdown samples through `SyntaxHighlight`
442-
443-TypeScript / JavaScript support currently covers a richer semantic slice:
444-
445-- type aliases, interfaces, classes, enums, and type parameters
446-- primitive type keywords and type references
447-- function declarations, methods, and arrow-function bindings
448-- object properties and property access
449-- decorators
450-- parameters
451-- booleans and null
452-- bigint literals
453-
454-Python support currently covers a meaningful first slice:
455-
456-- decorators
457-- class and def declarations
458-- builtin calls
459-- property access
460-- booleans
461-- type-ish names like builtins and class names
462-
463-Bash support currently covers a meaningful first slice:
464-
465-- keywords / control flow
466-- builtins vs command words
467-- assignments
468-- single and double quoted strings
469-- variable expansion
470-- command substitution
471-- arithmetic substitution
472-- heredocs
473-
474-JSON support currently covers:
475-
476-- object keys vs string values
477-- booleans and null
478-- numbers
479-- arrays and objects
480-- punctuation / structural tokens
481-
482-Markdown support currently covers:
483-
484-- headings
485-- list markers
486-- blockquote markers
487-- inline code spans
488-- fenced code blocks
489-- links
490-- horizontal rules
491-- escapes
492-
493-Verified in this session:
494-
495-- `bun test`
496-- `bun run typecheck`
497-- `bun run check`
498-- `bun run format`
499-
500-## Example status
501-
502-The current `examples/syntax-highlight.ts` is intentionally a **component usage example**.
503-
504-It does **not** use tokenizer internals directly.
505-It simply streams append-only content into:
506-
507-```ts
508-SyntaxHighlight(content, language, { theme });
509-```
510-
511-The demo now includes real samples for:
512-
513-- TypeScript
514-- JavaScript
515-- Python
516-- Bash
517-- JSON
518-- Markdown
519-
520-It also lets you switch themes and rotate through those samples without reaching
521-into tokenizer internals.
522-
523-That makes it a much better verification target for real users of the component.
524-
525-## What is still missing
526-
527-Important remaining work:
528-
529-- broader language coverage beyond TypeScript / JavaScript, Python, Bash, JSON, and Markdown
530-- stronger corpus / fixture coverage against real language manuals and specs
531-- deeper Bash coverage for more edge cases
532-- a clearer long-term decision on which future backends should be tokenizer-only, parser-backed, TextMate-backed, or mixed
533-
534-## Next session plan
535-
536-The next session should focus on preparing the language pipeline for more backends without regressing correctness.
537-
538-Recommended next steps:
539-
540-1. add richer per-language corpus / fixture suites
541- - especially for Bash constructs from the Bash manual
542- - and for TypeScript / JavaScript edge cases
543-2. deepen Bash coverage before adding aliases or adjacent shells
544- - Bash is Bash; keep it precise
545-3. improve normalized scope semantics where the current output is too shallow
546-4. choose the next language and add it through the same registry pattern
547-5. keep using `SyntaxHighlight` as the primary end-to-end consumer while `clew` grows
548-
549-The workflow still looks like this:
550-
551-- change `clew`
552-- observe the result through `SyntaxHighlight`
553-- verify tokenizer tests
554-- verify rendering tests
555-- verify the example
556-
557-That keeps the work grounded in a real consumer and makes regressions obvious early.