719
Le
lextide / files
lextide / spec.md
10.7 KB
1# lextide 2 3TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`. 4 5## Product definition 6 7`lextide` is a **drop-in replacement for `lowlight`** for its public highlighting API and output shape. 8 9That means: 10 11- same core API shape 12- same output model: HAST, not HTML 13- same CSS class convention by default (`hljs-`) 14- same general registry model: languages, aliases, auto-detect, list/check registration 15- additive streaming API for incremental highlighting 16 17This is a clean rewrite, not a wrapper around `highlight.js`. 18 19## Scope 20 21### In scope 22 23- HAST output compatible with `lowlight` 24- sync highlighting for full strings 25- streaming highlighting for incremental input 26- language registry + alias registry 27- `highlightAuto` for registered languages 28- initial built-in languages: 29 - bash 30 - javascript 31 - typescript 32 - python 33 - markdown 34- performance-first implementation 35- modern TypeScript codebase and types 36 37### Out of scope 38 39- HTML serialization 40- broad language coverage in v0 41- Prism compatibility 42- browser UI/rendering helpers as core API 43 44## Compatibility target 45 46`lextide` should match `lowlight` where consumers touch it. 47 48### Package-level exports 49 50`lextide` exports: 51 52- `createLowlight` 53- `common` 54- `all` 55- `LanguageFn` 56- `Options` 57- `AutoOptions` 58 59### `createLowlight()` instance API 60 61Returned object must expose: 62 63- `highlight(language, value, options?)` 64- `highlightAuto(value, options?)` 65- `listLanguages()` 66- `register(name, grammar)` 67- `register(grammars)` 68- `registerAlias(language, alias)` 69- `registerAlias(aliases)` 70- `registered(aliasOrName)` 71 72Additive API: 73 74- `stream(language, options?)` → `HighlightStream` 75 76Existing `lowlight` consumers should be able to swap imports to `lextide` and keep using HAST output the same way for supported languages. 77 78## HAST contract 79 80Sync and streaming final output must produce HAST roots with same shape as `lowlight`: 81 82```ts 83interface Root { 84 type: "root"; 85 children: ElementContent[]; 86 data?: { 87 language?: string; 88 relevance?: number; 89 }; 90} 91``` 92 93### Node rules 94 95- root node is always `{type: 'root'}` 96- highlighted scopes are emitted as `element` nodes with: 97 - `tagName: 'span'` 98 - `properties.className: string[]` 99- plain text is emitted as `text` nodes 100- adjacent text nodes are merged 101- source text is preserved exactly; no whitespace normalization 102- no HTML string is produced internally or publicly 103 104### Class-name compatibility 105 106Default class prefix is `hljs-`. 107 108Scope names are serialized like `lowlight` / current `highlight.js` emitter behavior: 109 110- first class segment gets configured prefix 111- later dotted segments get underscore suffixes 112 113Example: 114 115- scope name: `title.function` 116- class list: `['hljs-title', 'function_']` 117 118This matters for CSS compatibility with existing `lowlight` / `highlight.js` themes. 119 120## Sync API behavior 121 122### `highlight(language, value, options?)` 123 124Behavior: 125 126- throws if `language` is unknown 127- error message should match `lowlight` style: 128 - `Unknown language: \`<name>\` is not registered` 129- never throws for malformed source code; highlighting is best-effort 130- returns HAST root 131- `root.data.language` is set to chosen language 132- `root.data.relevance` is numeric 133 134### `highlightAuto(value, options?)` 135 136Behavior: 137 138- checks registered languages, or `options.subset` if provided 139- returns best match by relevance 140- returns empty root with `data.language === undefined` and `data.relevance === 0` when no language matches 141- same HAST contract as `highlight` 142 143### `Options` 144 145```ts 146interface Options { 147 prefix?: string | null | undefined; 148} 149``` 150 151Prefix semantics: 152 153- `undefined` or omitted → default `hljs-` 154- `null` → no prefix (scope name used as-is) 155- `string` → custom prefix 156 157### `AutoOptions` 158 159```ts 160interface AutoOptions extends Options { 161 subset?: readonly string[] | null | undefined; 162} 163``` 164 165## Streaming API 166 167Streaming is additive. It does not replace sync `highlight()`. 168 169### Goals 170 171- support incremental code input 172- emit useful partial results before full source is complete 173- preserve exact final parity with sync highlighting 174- minimize recomputation and allocations 175- support context-sensitive grammars without lying to downstream consumers 176 177### Model 178 179`shiki-stream` has right core idea: keep output stable where possible, optionally allow recall for unstable tail. 180 181`lextide` should use same principle, but output **HAST fragments**, not themed tokens. 182 183### Proposed API 184 185```ts 186interface StreamOptions extends Options { 187 allowRecalls?: boolean | null | undefined; 188} 189 190interface StreamUpdate { 191 recall: number; 192 stable: ElementContent[]; 193 unstable: ElementContent[]; 194} 195 196interface HighlightStream { 197 write(chunk: string): StreamUpdate; 198 end(chunk?: string): StreamUpdate; 199 snapshot(): Root; 200} 201``` 202 203### Semantics 204 205- `write(chunk)` consumes incremental source text 206- `end(chunk?)` finalizes stream and flushes buffered tail 207- `snapshot()` returns current best-effort full HAST root 208- stream is language-specific; caller chooses language up front 209- `snapshot()` always returns a valid HAST root; it is safe to render at any point 210- after `end()`, final assembled result must equal sync `highlight(language, fullSource)` 211 212### Stable vs unstable output 213 214Default streaming behavior should be conservative: 215 216- only finalized HAST fragments are emitted as `stable` 217- unfinished trailing region stays buffered internally until it becomes stable or stream ends 218- `unstable` is empty unless recall mode is enabled 219 220With `allowRecalls: true`: 221 222- `unstable` contains current best-effort trailing fragment(s) 223- `recall` tells consumer how many previously emitted unstable root children to discard 224- consumer applies update as: 225 1. remove last `recall` children from previously rendered unstable tail 226 2. append `stable` 227 3. append `unstable` 228 229This gives low-latency live highlighting without pretending unstable output is final. 230 231### Stability boundary 232 233v0 stability boundary is line-oriented: 234 235- completed lines become stable as soon as grammar state for that line is finalized 236- current trailing line remains unstable until next newline or `end()` 237 238Important detail: 239 240- multi-line constructs must still work 241- grammar state carries across lines 242- line-oriented stability is only output policy, not grammar limitation 243 244## Language system 245 246`lextide` ships native language definitions. v0 does not need to consume `highlight.js` grammars. 247 248### `LanguageFn` 249 250Keep public export name `LanguageFn` for lowlight-style familiarity, but back it with `lextide`'s own typed grammar definition API. 251 252Requirements: 253 254- resumable grammar state for streaming 255- nested scopes 256- relevance scoring for auto-detect 257- no HTML emission 258- no dependence on regex-only implementation details 259 260### Grammar definition shape 261 262```ts 263interface GrammarRule { 264 match: RegExp | ((input: string, pos: number) => number | null); 265 scope?: string; 266 push?: string; // named state to push onto stack 267 pop?: boolean; // pop current state 268 children?: GrammarRule[]; 269 relevance?: number; // per-rule relevance contribution 270} 271 272interface GrammarDefinition { 273 name: string; 274 aliases?: string[]; 275 relevance?: number; // base relevance for auto-detect 276 states: Record<string, GrammarRule[]>; 277 startState: string; 278} 279 280type LanguageFn = () => GrammarDefinition; 281``` 282 283Grammar execution is a state stack: 284 285- engine starts in `startState` 286- rules are tried in order within the current state 287- `push` enters a named state; `pop` exits back to parent 288- `children` define inline sub-rules active while a parent rule's scope is open 289- grammar state (stack + position) is serializable for streaming resumption 290 291Relevance scoring: 292 293- each grammar has a base `relevance` (default 0) 294- individual rules can contribute additional relevance when matched 295- `highlightAuto` sums relevance across all matched rules and picks the highest-scoring language 296 297### Initial built-ins 298 299v0 built-ins: 300 301- bash 302- javascript 303- typescript 304- python 305- markdown 306 307Notes: 308 309- javascript and typescript should share as much implementation as practical 310- `all` and `common` can be identical in v0 while language coverage is still small 311 312## Internal architecture 313 314Implementation should split into three layers. 315 316### 1. Core highlighter engine 317 318Responsible for: 319 320- scanning source incrementally 321- maintaining language state 322- opening scopes 323- closing scopes 324- producing text segments 325- scoring relevance 326 327Output should be an internal event stream or span stream, not HAST-specific logic. 328 329### 2. HAST builder 330 331Responsible for: 332 333- converting internal events into lowlight-compatible HAST 334- merging adjacent text nodes 335- building `className` arrays 336- attaching `data.language` and `data.relevance` 337 338Same builder should be used by: 339 340- sync highlighting 341- streaming stable fragments 342- streaming snapshots 343 344### 3. Registry layer 345 346Responsible for: 347 348- language registration 349- alias resolution 350- built-in maps (`all`, `common`) 351- `highlightAuto` traversal over registered languages 352 353## Performance requirements 354 355Performance is first-class. 356 357v0 requirements: 358 359- no HTML intermediate representation 360- single-pass highlighting for sync path where possible 361- streaming path reprocesses only buffered unstable tail, not full prior source 362- grammar state is resumable across chunks 363- minimize string slicing and node churn 364- avoid per-character object allocation in hot path 365 366## Behavioral invariants 367 368These must hold: 369 370- `highlight()` and streamed `end()` produce same final HAST for same input 371- aliases resolve the same way across sync and stream paths 372- prefix option affects sync and stream paths the same way 373- malformed code does not throw 374- unknown language does throw 375- adjacent text nodes are not duplicated by chunk boundaries 376- chunk boundaries do not change final output 377 378## Testing plan 379 380### Compatibility tests 381 382For each built-in language: 383 384- fixture-based HAST tests for representative code samples 385- CSS class compatibility tests 386- alias tests 387- prefix option tests 388- `highlightAuto` tests 389 390### Streaming tests 391 392- same source split at many chunk boundaries yields same final HAST 393- line-by-line input 394- one-character-at-a-time input 395- multiline strings/comments across chunk boundaries 396- `allowRecalls` update behavior 397- `snapshot()` matches current assembled state 398 399### Error and edge tests 400 401- unknown language throws exact error message 402- empty input 403- empty registry 404- subset filtering for `highlightAuto` 405- alias lookup and registration behavior 406 407### Performance tests 408 409Bench against: 410 411- `lowlight` 412- `shiki-stream`-style incremental workloads for streaming mode 413 414Focus on: 415 416- total time 417- allocations 418- chunked update cost 419 420## v0 definition of done 421 422`lextide` v0 is done when it has: 423 424- public `lowlight`-compatible sync API 425- HAST output parity for supported languages 426- built-in bash/javascript/typescript/python/markdown grammars 427- streaming API with stable mode and recall mode 428- parity tests for sync and stream final output 429- benchmark coverage showing replacement is competitive enough to justify switch