xonecas.io
719
Le

lextide / files

lextide / spec.md

10.7 KB
1# lextide
2
3TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`.
4
5## Product definition
6
7`lextide` is a **drop-in replacement for `lowlight`** for its public highlighting API and output shape.
8
9That means:
10
11- same core API shape
12- same output model: HAST, not HTML
13- same CSS class convention by default (`hljs-`)
14- same general registry model: languages, aliases, auto-detect, list/check registration
15- additive streaming API for incremental highlighting
16
17This is a clean rewrite, not a wrapper around `highlight.js`.
18
19## Scope
20
21### In scope
22
23- HAST output compatible with `lowlight`
24- sync highlighting for full strings
25- streaming highlighting for incremental input
26- language registry + alias registry
27- `highlightAuto` for registered languages
28- initial built-in languages:
29  - bash
30  - javascript
31  - typescript
32  - python
33  - markdown
34- performance-first implementation
35- modern TypeScript codebase and types
36
37### Out of scope
38
39- HTML serialization
40- broad language coverage in v0
41- Prism compatibility
42- browser UI/rendering helpers as core API
43
44## Compatibility target
45
46`lextide` should match `lowlight` where consumers touch it.
47
48### Package-level exports
49
50`lextide` exports:
51
52- `createLowlight`
53- `common`
54- `all`
55- `LanguageFn`
56- `Options`
57- `AutoOptions`
58
59### `createLowlight()` instance API
60
61Returned object must expose:
62
63- `highlight(language, value, options?)`
64- `highlightAuto(value, options?)`
65- `listLanguages()`
66- `register(name, grammar)`
67- `register(grammars)`
68- `registerAlias(language, alias)`
69- `registerAlias(aliases)`
70- `registered(aliasOrName)`
71
72Additive API:
73
74- `stream(language, options?)` → `HighlightStream`
75
76Existing `lowlight` consumers should be able to swap imports to `lextide` and keep using HAST output the same way for supported languages.
77
78## HAST contract
79
80Sync and streaming final output must produce HAST roots with same shape as `lowlight`:
81
82```ts
83interface Root {
84  type: "root";
85  children: ElementContent[];
86  data?: {
87    language?: string;
88    relevance?: number;
89  };
90}
91```
92
93### Node rules
94
95- root node is always `{type: 'root'}`
96- highlighted scopes are emitted as `element` nodes with:
97  - `tagName: 'span'`
98  - `properties.className: string[]`
99- plain text is emitted as `text` nodes
100- adjacent text nodes are merged
101- source text is preserved exactly; no whitespace normalization
102- no HTML string is produced internally or publicly
103
104### Class-name compatibility
105
106Default class prefix is `hljs-`.
107
108Scope names are serialized like `lowlight` / current `highlight.js` emitter behavior:
109
110- first class segment gets configured prefix
111- later dotted segments get underscore suffixes
112
113Example:
114
115- scope name: `title.function`
116- class list: `['hljs-title', 'function_']`
117
118This matters for CSS compatibility with existing `lowlight` / `highlight.js` themes.
119
120## Sync API behavior
121
122### `highlight(language, value, options?)`
123
124Behavior:
125
126- throws if `language` is unknown
127- error message should match `lowlight` style:
128  - `Unknown language: \`<name>\` is not registered`
129- never throws for malformed source code; highlighting is best-effort
130- returns HAST root
131- `root.data.language` is set to chosen language
132- `root.data.relevance` is numeric
133
134### `highlightAuto(value, options?)`
135
136Behavior:
137
138- checks registered languages, or `options.subset` if provided
139- returns best match by relevance
140- returns empty root with `data.language === undefined` and `data.relevance === 0` when no language matches
141- same HAST contract as `highlight`
142
143### `Options`
144
145```ts
146interface Options {
147  prefix?: string | null | undefined;
148}
149```
150
151Prefix semantics:
152
153- `undefined` or omitted → default `hljs-`
154- `null` → no prefix (scope name used as-is)
155- `string` → custom prefix
156
157### `AutoOptions`
158
159```ts
160interface AutoOptions extends Options {
161  subset?: readonly string[] | null | undefined;
162}
163```
164
165## Streaming API
166
167Streaming is additive. It does not replace sync `highlight()`.
168
169### Goals
170
171- support incremental code input
172- emit useful partial results before full source is complete
173- preserve exact final parity with sync highlighting
174- minimize recomputation and allocations
175- support context-sensitive grammars without lying to downstream consumers
176
177### Model
178
179`shiki-stream` has right core idea: keep output stable where possible, optionally allow recall for unstable tail.
180
181`lextide` should use same principle, but output **HAST fragments**, not themed tokens.
182
183### Proposed API
184
185```ts
186interface StreamOptions extends Options {
187  allowRecalls?: boolean | null | undefined;
188}
189
190interface StreamUpdate {
191  recall: number;
192  stable: ElementContent[];
193  unstable: ElementContent[];
194}
195
196interface HighlightStream {
197  write(chunk: string): StreamUpdate;
198  end(chunk?: string): StreamUpdate;
199  snapshot(): Root;
200}
201```
202
203### Semantics
204
205- `write(chunk)` consumes incremental source text
206- `end(chunk?)` finalizes stream and flushes buffered tail
207- `snapshot()` returns current best-effort full HAST root
208- stream is language-specific; caller chooses language up front
209- `snapshot()` always returns a valid HAST root; it is safe to render at any point
210- after `end()`, final assembled result must equal sync `highlight(language, fullSource)`
211
212### Stable vs unstable output
213
214Default streaming behavior should be conservative:
215
216- only finalized HAST fragments are emitted as `stable`
217- unfinished trailing region stays buffered internally until it becomes stable or stream ends
218- `unstable` is empty unless recall mode is enabled
219
220With `allowRecalls: true`:
221
222- `unstable` contains current best-effort trailing fragment(s)
223- `recall` tells consumer how many previously emitted unstable root children to discard
224- consumer applies update as:
225  1. remove last `recall` children from previously rendered unstable tail
226  2. append `stable`
227  3. append `unstable`
228
229This gives low-latency live highlighting without pretending unstable output is final.
230
231### Stability boundary
232
233v0 stability boundary is line-oriented:
234
235- completed lines become stable as soon as grammar state for that line is finalized
236- current trailing line remains unstable until next newline or `end()`
237
238Important detail:
239
240- multi-line constructs must still work
241- grammar state carries across lines
242- line-oriented stability is only output policy, not grammar limitation
243
244## Language system
245
246`lextide` ships native language definitions. v0 does not need to consume `highlight.js` grammars.
247
248### `LanguageFn`
249
250Keep public export name `LanguageFn` for lowlight-style familiarity, but back it with `lextide`'s own typed grammar definition API.
251
252Requirements:
253
254- resumable grammar state for streaming
255- nested scopes
256- relevance scoring for auto-detect
257- no HTML emission
258- no dependence on regex-only implementation details
259
260### Grammar definition shape
261
262```ts
263interface GrammarRule {
264  match: RegExp | ((input: string, pos: number) => number | null);
265  scope?: string;
266  push?: string; // named state to push onto stack
267  pop?: boolean; // pop current state
268  children?: GrammarRule[];
269  relevance?: number; // per-rule relevance contribution
270}
271
272interface GrammarDefinition {
273  name: string;
274  aliases?: string[];
275  relevance?: number; // base relevance for auto-detect
276  states: Record<string, GrammarRule[]>;
277  startState: string;
278}
279
280type LanguageFn = () => GrammarDefinition;
281```
282
283Grammar execution is a state stack:
284
285- engine starts in `startState`
286- rules are tried in order within the current state
287- `push` enters a named state; `pop` exits back to parent
288- `children` define inline sub-rules active while a parent rule's scope is open
289- grammar state (stack + position) is serializable for streaming resumption
290
291Relevance scoring:
292
293- each grammar has a base `relevance` (default 0)
294- individual rules can contribute additional relevance when matched
295- `highlightAuto` sums relevance across all matched rules and picks the highest-scoring language
296
297### Initial built-ins
298
299v0 built-ins:
300
301- bash
302- javascript
303- typescript
304- python
305- markdown
306
307Notes:
308
309- javascript and typescript should share as much implementation as practical
310- `all` and `common` can be identical in v0 while language coverage is still small
311
312## Internal architecture
313
314Implementation should split into three layers.
315
316### 1. Core highlighter engine
317
318Responsible for:
319
320- scanning source incrementally
321- maintaining language state
322- opening scopes
323- closing scopes
324- producing text segments
325- scoring relevance
326
327Output should be an internal event stream or span stream, not HAST-specific logic.
328
329### 2. HAST builder
330
331Responsible for:
332
333- converting internal events into lowlight-compatible HAST
334- merging adjacent text nodes
335- building `className` arrays
336- attaching `data.language` and `data.relevance`
337
338Same builder should be used by:
339
340- sync highlighting
341- streaming stable fragments
342- streaming snapshots
343
344### 3. Registry layer
345
346Responsible for:
347
348- language registration
349- alias resolution
350- built-in maps (`all`, `common`)
351- `highlightAuto` traversal over registered languages
352
353## Performance requirements
354
355Performance is first-class.
356
357v0 requirements:
358
359- no HTML intermediate representation
360- single-pass highlighting for sync path where possible
361- streaming path reprocesses only buffered unstable tail, not full prior source
362- grammar state is resumable across chunks
363- minimize string slicing and node churn
364- avoid per-character object allocation in hot path
365
366## Behavioral invariants
367
368These must hold:
369
370- `highlight()` and streamed `end()` produce same final HAST for same input
371- aliases resolve the same way across sync and stream paths
372- prefix option affects sync and stream paths the same way
373- malformed code does not throw
374- unknown language does throw
375- adjacent text nodes are not duplicated by chunk boundaries
376- chunk boundaries do not change final output
377
378## Testing plan
379
380### Compatibility tests
381
382For each built-in language:
383
384- fixture-based HAST tests for representative code samples
385- CSS class compatibility tests
386- alias tests
387- prefix option tests
388- `highlightAuto` tests
389
390### Streaming tests
391
392- same source split at many chunk boundaries yields same final HAST
393- line-by-line input
394- one-character-at-a-time input
395- multiline strings/comments across chunk boundaries
396- `allowRecalls` update behavior
397- `snapshot()` matches current assembled state
398
399### Error and edge tests
400
401- unknown language throws exact error message
402- empty input
403- empty registry
404- subset filtering for `highlightAuto`
405- alias lookup and registration behavior
406
407### Performance tests
408
409Bench against:
410
411- `lowlight`
412- `shiki-stream`-style incremental workloads for streaming mode
413
414Focus on:
415
416- total time
417- allocations
418- chunked update cost
419
420## v0 definition of done
421
422`lextide` v0 is done when it has:
423
424- public `lowlight`-compatible sync API
425- HAST output parity for supported languages
426- built-in bash/javascript/typescript/python/markdown grammars
427- streaming API with stable mode and recall mode
428- parity tests for sync and stream final output
429- benchmark coverage showing replacement is competitive enough to justify switch