@@ -0,0 +1,427 @@
1+# lextide
2+
3+TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`.
4+
5+## Product definition
6+
7+`lextide` is a **drop-in replacement for `lowlight`** for its public highlighting API and output shape.
8+
9+That means:
10+
11+- same core API shape
12+- same output model: HAST, not HTML
13+- same CSS class convention by default (`hljs-`)
14+- same general registry model: languages, aliases, auto-detect, list/check registration
15+- additive streaming API for incremental highlighting
16+
17+This is a clean rewrite, not a wrapper around `highlight.js`.
18+
19+## Scope
20+
21+### In scope
22+
23+- HAST output compatible with `lowlight`
24+- sync highlighting for full strings
25+- streaming highlighting for incremental input
26+- language registry + alias registry
27+- `highlightAuto` for registered languages
28+- initial built-in languages:
29+ - bash
30+ - javascript
31+ - typescript
32+ - python
33+- performance-first implementation
34+- modern TypeScript codebase and types
35+
36+### Out of scope
37+
38+- HTML serialization
39+- broad language coverage in v0
40+- Prism compatibility
41+- browser UI/rendering helpers as core API
42+
43+## Compatibility target
44+
45+`lextide` should match `lowlight` where consumers touch it.
46+
47+### Package-level exports
48+
49+`lextide` exports:
50+
51+- `createLowlight`
52+- `common`
53+- `all`
54+- `LanguageFn`
55+- `Options`
56+- `AutoOptions`
57+
58+### `createLowlight()` instance API
59+
60+Returned object must expose:
61+
62+- `highlight(language, value, options?)`
63+- `highlightAuto(value, options?)`
64+- `listLanguages()`
65+- `register(name, grammar)`
66+- `register(grammars)`
67+- `registerAlias(language, alias)`
68+- `registerAlias(aliases)`
69+- `registered(aliasOrName)`
70+
71+Additive API:
72+
73+- `stream(language, options?)` → `HighlightStream`
74+
75+Existing `lowlight` consumers should be able to swap imports to `lextide` and keep using HAST output the same way for supported languages.
76+
77+## HAST contract
78+
79+Sync and streaming final output must produce HAST roots with same shape as `lowlight`:
80+
81+```ts
82+interface Root {
83+ type: "root";
84+ children: ElementContent[];
85+ data?: {
86+ language?: string;
87+ relevance?: number;
88+ };
89+}
90+```
91+
92+### Node rules
93+
94+- root node is always `{type: 'root'}`
95+- highlighted scopes are emitted as `element` nodes with:
96+ - `tagName: 'span'`
97+ - `properties.className: string[]`
98+- plain text is emitted as `text` nodes
99+- adjacent text nodes are merged
100+- source text is preserved exactly; no whitespace normalization
101+- no HTML string is produced internally or publicly
102+
103+### Class-name compatibility
104+
105+Default class prefix is `hljs-`.
106+
107+Scope names are serialized like `lowlight` / current `highlight.js` emitter behavior:
108+
109+- first class segment gets configured prefix
110+- later dotted segments get underscore suffixes
111+
112+Example:
113+
114+- scope name: `title.function`
115+- class list: `['hljs-title', 'function_']`
116+
117+This matters for CSS compatibility with existing `lowlight` / `highlight.js` themes.
118+
119+## Sync API behavior
120+
121+### `highlight(language, value, options?)`
122+
123+Behavior:
124+
125+- throws if `language` is unknown
126+- error message should match `lowlight` style:
127+ - `Unknown language: \`<name>\` is not registered`
128+- never throws for malformed source code; highlighting is best-effort
129+- returns HAST root
130+- `root.data.language` is set to chosen language
131+- `root.data.relevance` is numeric
132+
133+### `highlightAuto(value, options?)`
134+
135+Behavior:
136+
137+- checks registered languages, or `options.subset` if provided
138+- returns best match by relevance
139+- returns empty root with `data.language === undefined` and `data.relevance === 0` when no language matches
140+- same HAST contract as `highlight`
141+
142+### `Options`
143+
144+```ts
145+interface Options {
146+ prefix?: string | null | undefined;
147+}
148+```
149+
150+Prefix semantics:
151+
152+- `undefined` or omitted → default `hljs-`
153+- `null` → no prefix (scope name used as-is)
154+- `string` → custom prefix
155+
156+### `AutoOptions`
157+
158+```ts
159+interface AutoOptions extends Options {
160+ subset?: readonly string[] | null | undefined;
161+}
162+```
163+
164+## Streaming API
165+
166+Streaming is additive. It does not replace sync `highlight()`.
167+
168+### Goals
169+
170+- support incremental code input
171+- emit useful partial results before full source is complete
172+- preserve exact final parity with sync highlighting
173+- minimize recomputation and allocations
174+- support context-sensitive grammars without lying to downstream consumers
175+
176+### Model
177+
178+`shiki-stream` has right core idea: keep output stable where possible, optionally allow recall for unstable tail.
179+
180+`lextide` should use same principle, but output **HAST fragments**, not themed tokens.
181+
182+### Proposed API
183+
184+```ts
185+interface StreamOptions extends Options {
186+ allowRecalls?: boolean | null | undefined;
187+}
188+
189+interface StreamUpdate {
190+ recall: number;
191+ stable: ElementContent[];
192+ unstable: ElementContent[];
193+}
194+
195+interface HighlightStream {
196+ write(chunk: string): StreamUpdate;
197+ end(chunk?: string): StreamUpdate;
198+ snapshot(): Root;
199+}
200+```
201+
202+### Semantics
203+
204+- `write(chunk)` consumes incremental source text
205+- `end(chunk?)` finalizes stream and flushes buffered tail
206+- `snapshot()` returns current best-effort full HAST root
207+- stream is language-specific; caller chooses language up front
208+- `snapshot()` always returns a valid HAST root; it is safe to render at any point
209+- after `end()`, final assembled result must equal sync `highlight(language, fullSource)`
210+
211+### Stable vs unstable output
212+
213+Default streaming behavior should be conservative:
214+
215+- only finalized HAST fragments are emitted as `stable`
216+- unfinished trailing region stays buffered internally until it becomes stable or stream ends
217+- `unstable` is empty unless recall mode is enabled
218+
219+With `allowRecalls: true`:
220+
221+- `unstable` contains current best-effort trailing fragment(s)
222+- `recall` tells consumer how many previously emitted unstable root children to discard
223+- consumer applies update as:
224+ 1. remove last `recall` children from previously rendered unstable tail
225+ 2. append `stable`
226+ 3. append `unstable`
227+
228+This gives low-latency live highlighting without pretending unstable output is final.
229+
230+### Stability boundary
231+
232+v0 stability boundary is line-oriented:
233+
234+- completed lines become stable as soon as grammar state for that line is finalized
235+- current trailing line remains unstable until next newline or `end()`
236+
237+Important detail:
238+
239+- multi-line constructs must still work
240+- grammar state carries across lines
241+- line-oriented stability is only output policy, not grammar limitation
242+
243+## Language system
244+
245+`lextide` ships native language definitions. v0 does not need to consume `highlight.js` grammars.
246+
247+### `LanguageFn`
248+
249+Keep public export name `LanguageFn` for lowlight-style familiarity, but back it with `lextide`'s own typed grammar definition API.
250+
251+Requirements:
252+
253+- resumable grammar state for streaming
254+- nested scopes
255+- relevance scoring for auto-detect
256+- no HTML emission
257+- no dependence on regex-only implementation details
258+
259+### Grammar definition shape
260+
261+```ts
262+interface GrammarRule {
263+ match: RegExp | ((input: string, pos: number) => number | null);
264+ scope?: string;
265+ push?: string; // named state to push onto stack
266+ pop?: boolean; // pop current state
267+ children?: GrammarRule[];
268+ relevance?: number; // per-rule relevance contribution
269+}
270+
271+interface GrammarDefinition {
272+ name: string;
273+ aliases?: string[];
274+ relevance?: number; // base relevance for auto-detect
275+ states: Record<string, GrammarRule[]>;
276+ startState: string;
277+}
278+
279+type LanguageFn = () => GrammarDefinition;
280+```
281+
282+Grammar execution is a state stack:
283+
284+- engine starts in `startState`
285+- rules are tried in order within the current state
286+- `push` enters a named state; `pop` exits back to parent
287+- `children` define inline sub-rules active while a parent rule's scope is open
288+- grammar state (stack + position) is serializable for streaming resumption
289+
290+Relevance scoring:
291+
292+- each grammar has a base `relevance` (default 0)
293+- individual rules can contribute additional relevance when matched
294+- `highlightAuto` sums relevance across all matched rules and picks the highest-scoring language
295+
296+### Initial built-ins
297+
298+v0 built-ins:
299+
300+- bash
301+- javascript
302+- typescript
303+- python
304+
305+Notes:
306+
307+- javascript and typescript should share as much implementation as practical
308+- `all` and `common` can be identical in v0 while language coverage is still small
309+
310+## Internal architecture
311+
312+Implementation should split into three layers.
313+
314+### 1. Core highlighter engine
315+
316+Responsible for:
317+
318+- scanning source incrementally
319+- maintaining language state
320+- opening scopes
321+- closing scopes
322+- producing text segments
323+- scoring relevance
324+
325+Output should be an internal event stream or span stream, not HAST-specific logic.
326+
327+### 2. HAST builder
328+
329+Responsible for:
330+
331+- converting internal events into lowlight-compatible HAST
332+- merging adjacent text nodes
333+- building `className` arrays
334+- attaching `data.language` and `data.relevance`
335+
336+Same builder should be used by:
337+
338+- sync highlighting
339+- streaming stable fragments
340+- streaming snapshots
341+
342+### 3. Registry layer
343+
344+Responsible for:
345+
346+- language registration
347+- alias resolution
348+- built-in maps (`all`, `common`)
349+- `highlightAuto` traversal over registered languages
350+
351+## Performance requirements
352+
353+Performance is first-class.
354+
355+v0 requirements:
356+
357+- no HTML intermediate representation
358+- single-pass highlighting for sync path where possible
359+- streaming path reprocesses only buffered unstable tail, not full prior source
360+- grammar state is resumable across chunks
361+- minimize string slicing and node churn
362+- avoid per-character object allocation in hot path
363+
364+## Behavioral invariants
365+
366+These must hold:
367+
368+- `highlight()` and streamed `end()` produce same final HAST for same input
369+- aliases resolve the same way across sync and stream paths
370+- prefix option affects sync and stream paths the same way
371+- malformed code does not throw
372+- unknown language does throw
373+- adjacent text nodes are not duplicated by chunk boundaries
374+- chunk boundaries do not change final output
375+
376+## Testing plan
377+
378+### Compatibility tests
379+
380+For each built-in language:
381+
382+- fixture-based HAST tests for representative code samples
383+- CSS class compatibility tests
384+- alias tests
385+- prefix option tests
386+- `highlightAuto` tests
387+
388+### Streaming tests
389+
390+- same source split at many chunk boundaries yields same final HAST
391+- line-by-line input
392+- one-character-at-a-time input
393+- multiline strings/comments across chunk boundaries
394+- `allowRecalls` update behavior
395+- `snapshot()` matches current assembled state
396+
397+### Error and edge tests
398+
399+- unknown language throws exact error message
400+- empty input
401+- empty registry
402+- subset filtering for `highlightAuto`
403+- alias lookup and registration behavior
404+
405+### Performance tests
406+
407+Bench against:
408+
409+- `lowlight`
410+- `shiki-stream`-style incremental workloads for streaming mode
411+
412+Focus on:
413+
414+- total time
415+- allocations
416+- chunked update cost
417+
418+## v0 definition of done
419+
420+`lextide` v0 is done when it has:
421+
422+- public `lowlight`-compatible sync API
423+- HAST output parity for supported languages
424+- built-in bash/javascript/typescript/python grammars
425+- streaming API with stable mode and recall mode
426+- parity tests for sync and stream final output
427+- benchmark coverage showing replacement is competitive enough to justify switch