xonecas.io
719
Le

lextide / commit 83c1cb0

chore: bootstrap lextide tooling

xonecas2026-04-11 83c1cb01ca850ce6bb2a069fca851f41dd642a79

10 files changed

.gitignore+3−0
@@ -0,0 +1,3 @@
1+node_modules/
2+coverage/
3+dist/
AGENTS.md+170−0
@@ -0,0 +1,170 @@
1+# AGENTS.md
2+
3+## Project
4+
5+- Name: `lextide`
6+- Goal: clean rewrite of `lowlight`
7+- Core requirement: **drop-in replacement for `lowlight`**
8+- Output contract: **HAST**, not HTML
9+- Big extra feature: **streaming support**
10+
11+## Read first
12+
13+Before doing anything, read:
14+
15+1. `spec.md`
16+2. `TODO.md`
17+
18+`spec.md` is source of truth for product and API direction.
19+
20+## Current repo state
21+
22+- Repo has Bun/TypeScript/Biome/Lefthook scaffold
23+- Git repo is initialized
24+- Implementation is still near-zero; `spec.md` is still source of truth
25+- Keep bootstrap changes small and aligned with planned public API
26+
27+## Non-negotiables
28+
29+- Treat `lextide` as replacement for `lowlight`, not as generic lexer experiment
30+- HAST is public contract
31+- Keep `lowlight`-compatible API shape for supported features
32+- Streaming is required, not optional nice-to-have
33+- Do not drift into HTML renderer work
34+
35+## Compatibility target
36+
37+Match `lowlight` where users touch it:
38+
39+- `createLowlight`
40+- `common`
41+- `all`
42+- `highlight(language, value, options?)`
43+- `highlightAuto(value, options?)`
44+- `listLanguages()`
45+- `register(name, grammar)` / `register(grammars)`
46+- `registerAlias(language, alias)` / `registerAlias(aliases)`
47+- `registered(aliasOrName)`
48+
49+Additive API allowed:
50+
51+- `stream(language, options?)`
52+
53+## Output shape
54+
55+Return HAST roots like `lowlight`:
56+
57+- root node with `type: 'root'`
58+- text nodes for plain text
59+- `span` element nodes for scopes
60+- `properties.className` arrays
61+- `root.data.language`
62+- `root.data.relevance`
63+
64+Default class prefix:
65+
66+- `hljs-`
67+
68+## v0 language scope
69+
70+Start small:
71+
72+- bash
73+- javascript
74+- typescript
75+- python
76+
77+`all` and `common` can be same at first.
78+
79+## Streaming direction
80+
81+Streaming should follow stable/unstable model inspired by `shiki-stream`:
82+
83+- stable output by default
84+- optional recall mode for unstable tail
85+- final streamed result must match sync `highlight()` output
86+- line-oriented stability boundary is acceptable for v0
87+- grammar state must survive across chunks
88+
89+## Planning guidance
90+
91+User does not want quiz-style planning.
92+
93+Good:
94+
95+- inspect repo
96+- inspect relevant upstream behavior when needed
97+- make concrete proposal
98+- implement or update spec directly
99+
100+Bad:
101+
102+- turning obvious requirements into a questionnaire
103+- reframing project away from drop-in `lowlight` replacement
104+- over-indexing on abstract lexer theory
105+
106+When requirements are clear, act on them.
107+
108+## Tooling
109+
110+- ESM-only
111+- Bun for package management, scripts, and tests
112+- Biome for formatting, linting, and import sorting
113+- Lefthook for pre-commit checks
114+- TypeScript with strict mode
115+
116+## Scripts
117+
118+- `bun run format` — apply Biome formatting and import sorting
119+- `bun run lint` — run Biome lint
120+- `bun run typecheck` — run `tsc --noEmit`
121+- `bun run test` — run Bun test runner
122+- `bun run check` — run Biome check, typecheck, and tests
123+
124+## Workflow
125+
126+- run `bun install` after clone; `prepare` installs Lefthook
127+- use TDD by default: write or update test first, confirm it fails, implement minimal fix, then rerun targeted test
128+- prefer focused verification while iterating; do not spam `bun run check` after every small edit
129+- run `bun run format` before commit or handoff
130+- remember pre-commit already runs Biome check, typecheck, and tests automatically on commit
131+- run `bun run check` for final manual verification when needed, not as repetitive loop during development
132+- when committing code, if the user didn't explicitly ask you to, always review the diff with the user before committing.
133+
134+## Spec ambiguities
135+
136+When a requirement is unclear or missing from `spec.md`:
137+
138+- if the answer is obvious from `lowlight` behavior, match `lowlight` and note it
139+- if it's a judgment call, make the call, implement it, and flag it to the user
140+- do not block on questions that have a reasonable default
141+
142+## Working style
143+
144+- Keep scope tight
145+- Prefer boring, explicit TypeScript
146+- Reuse spec language and project terminology
147+- Follow the three-layer architecture from `spec.md` (core engine → HAST builder → registry)
148+- Verify claims against actual `lowlight` behavior when compatibility matters
149+- If researching streaming behavior, check `shiki-stream`
150+
151+## If starting implementation
152+
153+First useful steps:
154+
155+1. define public types and `createLowlight()` API
156+2. build internal HAST builder
157+3. build registry + aliases
158+4. add sync path for one language
159+5. add parity tests with Bun test
160+6. add streaming API
161+7. expand languages
162+
163+## Session bootstrap checklist
164+
165+- read `spec.md`
166+- read this file
167+- inspect repo files, especially `package.json`, `tsconfig.json`, `biome.json`, and `lefthook.yml`
168+- restate task in one or two concrete lines
169+- do work
170+- verify only what actually exists
TODO.md+21−0
@@ -0,0 +1,21 @@
1+# TODO
2+
3+- [x] scaffold package, with git, typescript, and bun for tests and package management.
4+- [x] Add minimal bootstrap files.
5+- [x] Install biomejs for formatting and linting. Ensure organizeImports is on and our complexity config matches:
6+
7+ ```json
8+ "complexity": {
9+ "noExcessiveCognitiveComplexity": {
10+ "level": "error",
11+ "options": {
12+ "maxAllowedComplexity": 15
13+ }
14+ },
15+ "noExcessiveNestedTestSuites": "error"
16+ }
17+ ```
18+
19+- [x] Use left-hook to install pre-commit hook that checks test, biome format, biome lint and typescript types.
20+- [x] Update AGENTS.md with the new scripts and process. Correct anything that is outdated or not specified.
21+- [x] Update TODO.md with your progress
biome.json+24−0
@@ -0,0 +1,24 @@
1+{
2+ "$schema": "https://biomejs.dev/schemas/2.4.11/schema.json",
3+ "assist": {
4+ "actions": {
5+ "source": {
6+ "organizeImports": "on"
7+ }
8+ }
9+ },
10+ "linter": {
11+ "enabled": true,
12+ "rules": {
13+ "complexity": {
14+ "noExcessiveCognitiveComplexity": {
15+ "level": "error",
16+ "options": {
17+ "maxAllowedComplexity": 15
18+ }
19+ },
20+ "noExcessiveNestedTestSuites": "error"
21+ }
22+ }
23+ }
24+}
bun.lock+60−0
@@ -0,0 +1,60 @@
1+{
2+ "lockfileVersion": 1,
3+ "configVersion": 1,
4+ "workspaces": {
5+ "": {
6+ "name": "lextide",
7+ "devDependencies": {
8+ "@biomejs/biome": "^2.4.11",
9+ "lefthook": "^2.1.5",
10+ "prettier": "^3.8.2",
11+ "typescript": "^6.0.2",
12+ },
13+ },
14+ },
15+ "packages": {
16+ "@biomejs/biome": ["@biomejs/biome@2.4.11", "", { "optionalDependencies": { "@biomejs/cli-darwin-arm64": "2.4.11", "@biomejs/cli-darwin-x64": "2.4.11", "@biomejs/cli-linux-arm64": "2.4.11", "@biomejs/cli-linux-arm64-musl": "2.4.11", "@biomejs/cli-linux-x64": "2.4.11", "@biomejs/cli-linux-x64-musl": "2.4.11", "@biomejs/cli-win32-arm64": "2.4.11", "@biomejs/cli-win32-x64": "2.4.11" }, "bin": { "biome": "bin/biome" } }, "sha512-nWxHX8tf3Opb/qRgZpBbsTOqOodkbrkJ7S+JxJAruxOReaDPPmPuLBAGQ8vigyUgo0Q …
17+
18+ "@biomejs/cli-darwin-arm64": ["@biomejs/cli-darwin-arm64@2.4.11", "", { "os": "darwin", "cpu": "arm64" }, "sha512-wOt+ed+L2dgZanWyL6i29qlXMc088N11optzpo10peayObBaAshbTcxKUchzEMp9QSY8rh5h6VfAFE3WTS1rqg=="],
19+
20+ "@biomejs/cli-darwin-x64": ["@biomejs/cli-darwin-x64@2.4.11", "", { "os": "darwin", "cpu": "x64" }, "sha512-gZ6zR8XmZlExfi/Pz/PffmdpWOQ8Qhy7oBztgkR8/ylSRyLwfRPSadmiVCV8WQ8PoJ2MWUy2fgID9zmtgUUJmw=="],
21+
22+ "@biomejs/cli-linux-arm64": ["@biomejs/cli-linux-arm64@2.4.11", "", { "os": "linux", "cpu": "arm64" }, "sha512-avdJaEElXrKceK0va9FkJ4P5ci3N01TGkc6ni3P8l3BElqbOz42Wg2IyX3gbh0ZLEd4HVKEIrmuVu/AMuSeFFA=="],
23+
24+ "@biomejs/cli-linux-arm64-musl": ["@biomejs/cli-linux-arm64-musl@2.4.11", "", { "os": "linux", "cpu": "arm64" }, "sha512-+Sbo1OAmlegtdwqFE8iOxFIWLh1B3OEgsuZfBpyyN/kWuqZ8dx9ZEes6zVnDMo+zRHF2wLynRVhoQmV7ohxl2Q=="],
25+
26+ "@biomejs/cli-linux-x64": ["@biomejs/cli-linux-x64@2.4.11", "", { "os": "linux", "cpu": "x64" }, "sha512-TagWV0iomp5LnEnxWFg4nQO+e52Fow349vaX0Q/PIcX6Zhk4GGBgp3qqZ8PVkpC+cuehRctMf3+6+FgQ8jCEFQ=="],
27+
28+ "@biomejs/cli-linux-x64-musl": ["@biomejs/cli-linux-x64-musl@2.4.11", "", { "os": "linux", "cpu": "x64" }, "sha512-bexd2IklK7ZgPhrz6jXzpIL6dEAH9MlJU1xGTrypx+FICxrXUp4CqtwfiuoDKse+UlgAlWtzML3jrMqeEAHEhA=="],
29+
30+ "@biomejs/cli-win32-arm64": ["@biomejs/cli-win32-arm64@2.4.11", "", { "os": "win32", "cpu": "arm64" }, "sha512-RJhaTnY8byzxDt4bDVb7AFPHkPcjOPK3xBip4ZRTrN3TEfyhjLRm3r3mqknqydgVTB74XG8l4jMLwEACEeihVg=="],
31+
32+ "@biomejs/cli-win32-x64": ["@biomejs/cli-win32-x64@2.4.11", "", { "os": "win32", "cpu": "x64" }, "sha512-A8D3JM/00C2KQgUV3oj8Ba15EHEYwebAGCy5Sf9GAjr5Y3+kJIYOiESoqRDeuRZueuMdCsbLZIUqmPhpYXJE9A=="],
33+
34+ "lefthook": ["lefthook@2.1.5", "", { "optionalDependencies": { "lefthook-darwin-arm64": "2.1.5", "lefthook-darwin-x64": "2.1.5", "lefthook-freebsd-arm64": "2.1.5", "lefthook-freebsd-x64": "2.1.5", "lefthook-linux-arm64": "2.1.5", "lefthook-linux-x64": "2.1.5", "lefthook-openbsd-arm64": "2.1.5", "lefthook-openbsd-x64": "2.1.5", "lefthook-windows-arm64": "2.1.5", "lefthook-windows-x64": "2.1.5" }, "bin": { "lefthook": "bin/index.js" } }, "sha512-yB9IFWurFllusbPZqvG0EavTmpNXPya2MuO7Li7YT78xAj3u …
35+
36+ "lefthook-darwin-arm64": ["lefthook-darwin-arm64@2.1.5", "", { "os": "darwin", "cpu": "arm64" }, "sha512-VITTaw8PxxyE26gkZ8UcwIa5ZrWnKNRGLeeSrqri40cQdXvLTEoMq2tjjw7eiL9UcB0waRReDdzydevy9GOPUQ=="],
37+
38+ "lefthook-darwin-x64": ["lefthook-darwin-x64@2.1.5", "", { "os": "darwin", "cpu": "x64" }, "sha512-AvtjYiW0BSGHBGrdvL313seUymrW9FxI+6JJwJ+ZSaa2sH81etrTB0wAwlH1L9VfFwK9+gWvatZBvLfF3L4fPw=="],
39+
40+ "lefthook-freebsd-arm64": ["lefthook-freebsd-arm64@2.1.5", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-mXjJwe8jKGWGiBYUxfQY1ab3Nn5NhafqT9q3KJz8m5joGGQj4JD0cbWxF1nVBLBWsDGbWZRZunTCMGcIScT2bQ=="],
41+
42+ "lefthook-freebsd-x64": ["lefthook-freebsd-x64@2.1.5", "", { "os": "freebsd", "cpu": "x64" }, "sha512-exD69dCjc1K45BxatDPGoH4NmEvgLKPm4kJLOWn1fTeHRKZwWiFPwnjknEoG2OemlCDHmCU++5X40kMEG0WBlA=="],
43+
44+ "lefthook-linux-arm64": ["lefthook-linux-arm64@2.1.5", "", { "os": "linux", "cpu": "arm64" }, "sha512-57TDKC5ewWpsCLZQKIJMHumFEObYKVundmPpiWhX491hINRZYYOL/26yrnVnNcidThRzTiTC+HLcuplLcaXtbA=="],
45+
46+ "lefthook-linux-x64": ["lefthook-linux-x64@2.1.5", "", { "os": "linux", "cpu": "x64" }, "sha512-bqK3LrAB5l5YaCaoHk6qRWlITrGWzP4FbwRxA31elbxjd0wgNWZ2Sn3zEfSEcxz442g7/PPkEwqqsTx0kSFzpg=="],
47+
48+ "lefthook-openbsd-arm64": ["lefthook-openbsd-arm64@2.1.5", "", { "os": "openbsd", "cpu": "arm64" }, "sha512-5aSwK7vV3A6t0w9PnxCMiVjQlcvopBP50BtmnnLnNJyAYHnFbZ0Baq5M0WkE9IsUkWSux0fe6fd0jDkuG711MA=="],
49+
50+ "lefthook-openbsd-x64": ["lefthook-openbsd-x64@2.1.5", "", { "os": "openbsd", "cpu": "x64" }, "sha512-Y+pPdDuENJ8qWnUgL02xxhpjblc0WnwXvWGfqnl3WZrAgHzQpwx3G6469RID/wlNVdHYAlw3a8UkFSMYsTzXvA=="],
51+
52+ "lefthook-windows-arm64": ["lefthook-windows-arm64@2.1.5", "", { "os": "win32", "cpu": "arm64" }, "sha512-2PlcFBjTzJaMufw0c28kfhB/0zmaRCU0TRPPsil/HU2YNOExod4upPGLk9qjgsOmb2YVWFz6zq6u7+D1yqmzTQ=="],
53+
54+ "lefthook-windows-x64": ["lefthook-windows-x64@2.1.5", "", { "os": "win32", "cpu": "x64" }, "sha512-yiAh8qxml6uqy10jDxOdN9fOQpyLxBFY1fgCEAhn7sVJYmJKRhjqSBwZX6LG5MQjzr29KStrIdw7TR3lf3rT7Q=="],
55+
56+ "prettier": ["prettier@3.8.2", "", { "bin": { "prettier": "bin/prettier.cjs" } }, "sha512-8c3mgTe0ASwWAJK+78dpviD+A8EqhndQPUBpNUIPt6+xWlIigCwfN01lWr9MAede4uqXGTEKeQWTvzb3vjia0Q=="],
57+
58+ "typescript": ["typescript@6.0.2", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ=="],
59+ }
60+}
lefthook.yml+8−0
@@ -0,0 +1,8 @@
1+pre-commit:
2+ commands:
3+ biome:
4+ run: bunx @biomejs/biome check .
5+ typecheck:
6+ run: bun run typecheck
7+ test:
8+ run: bun run test
package.json+22−0
@@ -0,0 +1,22 @@
1+{
2+ "name": "lextide",
3+ "version": "0.0.0",
4+ "description": "TypeScript-first, HAST-first, streaming-capable rewrite of lowlight.",
5+ "type": "module",
6+ "packageManager": "bun@1.3.9",
7+ "scripts": {
8+ "test": "bun test --pass-with-no-tests",
9+ "format": "biome check --write .",
10+ "format:docs": "prettier --write \"**/*.md\"",
11+ "lint": "biome lint .",
12+ "typecheck": "tsc --noEmit",
13+ "check": "biome check . && bun run typecheck && bun run test",
14+ "prepare": "lefthook install"
15+ },
16+ "devDependencies": {
17+ "@biomejs/biome": "^2.4.11",
18+ "lefthook": "^2.1.5",
19+ "prettier": "^3.8.2",
20+ "typescript": "^6.0.2"
21+ }
22+}
spec.md+427−0
@@ -0,0 +1,427 @@
1+# lextide
2+
3+TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`.
4+
5+## Product definition
6+
7+`lextide` is a **drop-in replacement for `lowlight`** for its public highlighting API and output shape.
8+
9+That means:
10+
11+- same core API shape
12+- same output model: HAST, not HTML
13+- same CSS class convention by default (`hljs-`)
14+- same general registry model: languages, aliases, auto-detect, list/check registration
15+- additive streaming API for incremental highlighting
16+
17+This is a clean rewrite, not a wrapper around `highlight.js`.
18+
19+## Scope
20+
21+### In scope
22+
23+- HAST output compatible with `lowlight`
24+- sync highlighting for full strings
25+- streaming highlighting for incremental input
26+- language registry + alias registry
27+- `highlightAuto` for registered languages
28+- initial built-in languages:
29+ - bash
30+ - javascript
31+ - typescript
32+ - python
33+- performance-first implementation
34+- modern TypeScript codebase and types
35+
36+### Out of scope
37+
38+- HTML serialization
39+- broad language coverage in v0
40+- Prism compatibility
41+- browser UI/rendering helpers as core API
42+
43+## Compatibility target
44+
45+`lextide` should match `lowlight` where consumers touch it.
46+
47+### Package-level exports
48+
49+`lextide` exports:
50+
51+- `createLowlight`
52+- `common`
53+- `all`
54+- `LanguageFn`
55+- `Options`
56+- `AutoOptions`
57+
58+### `createLowlight()` instance API
59+
60+Returned object must expose:
61+
62+- `highlight(language, value, options?)`
63+- `highlightAuto(value, options?)`
64+- `listLanguages()`
65+- `register(name, grammar)`
66+- `register(grammars)`
67+- `registerAlias(language, alias)`
68+- `registerAlias(aliases)`
69+- `registered(aliasOrName)`
70+
71+Additive API:
72+
73+- `stream(language, options?)` → `HighlightStream`
74+
75+Existing `lowlight` consumers should be able to swap imports to `lextide` and keep using HAST output the same way for supported languages.
76+
77+## HAST contract
78+
79+Sync and streaming final output must produce HAST roots with same shape as `lowlight`:
80+
81+```ts
82+interface Root {
83+ type: "root";
84+ children: ElementContent[];
85+ data?: {
86+ language?: string;
87+ relevance?: number;
88+ };
89+}
90+```
91+
92+### Node rules
93+
94+- root node is always `{type: 'root'}`
95+- highlighted scopes are emitted as `element` nodes with:
96+ - `tagName: 'span'`
97+ - `properties.className: string[]`
98+- plain text is emitted as `text` nodes
99+- adjacent text nodes are merged
100+- source text is preserved exactly; no whitespace normalization
101+- no HTML string is produced internally or publicly
102+
103+### Class-name compatibility
104+
105+Default class prefix is `hljs-`.
106+
107+Scope names are serialized like `lowlight` / current `highlight.js` emitter behavior:
108+
109+- first class segment gets configured prefix
110+- later dotted segments get underscore suffixes
111+
112+Example:
113+
114+- scope name: `title.function`
115+- class list: `['hljs-title', 'function_']`
116+
117+This matters for CSS compatibility with existing `lowlight` / `highlight.js` themes.
118+
119+## Sync API behavior
120+
121+### `highlight(language, value, options?)`
122+
123+Behavior:
124+
125+- throws if `language` is unknown
126+- error message should match `lowlight` style:
127+ - `Unknown language: \`<name>\` is not registered`
128+- never throws for malformed source code; highlighting is best-effort
129+- returns HAST root
130+- `root.data.language` is set to chosen language
131+- `root.data.relevance` is numeric
132+
133+### `highlightAuto(value, options?)`
134+
135+Behavior:
136+
137+- checks registered languages, or `options.subset` if provided
138+- returns best match by relevance
139+- returns empty root with `data.language === undefined` and `data.relevance === 0` when no language matches
140+- same HAST contract as `highlight`
141+
142+### `Options`
143+
144+```ts
145+interface Options {
146+ prefix?: string | null | undefined;
147+}
148+```
149+
150+Prefix semantics:
151+
152+- `undefined` or omitted → default `hljs-`
153+- `null` → no prefix (scope name used as-is)
154+- `string` → custom prefix
155+
156+### `AutoOptions`
157+
158+```ts
159+interface AutoOptions extends Options {
160+ subset?: readonly string[] | null | undefined;
161+}
162+```
163+
164+## Streaming API
165+
166+Streaming is additive. It does not replace sync `highlight()`.
167+
168+### Goals
169+
170+- support incremental code input
171+- emit useful partial results before full source is complete
172+- preserve exact final parity with sync highlighting
173+- minimize recomputation and allocations
174+- support context-sensitive grammars without lying to downstream consumers
175+
176+### Model
177+
178+`shiki-stream` has right core idea: keep output stable where possible, optionally allow recall for unstable tail.
179+
180+`lextide` should use same principle, but output **HAST fragments**, not themed tokens.
181+
182+### Proposed API
183+
184+```ts
185+interface StreamOptions extends Options {
186+ allowRecalls?: boolean | null | undefined;
187+}
188+
189+interface StreamUpdate {
190+ recall: number;
191+ stable: ElementContent[];
192+ unstable: ElementContent[];
193+}
194+
195+interface HighlightStream {
196+ write(chunk: string): StreamUpdate;
197+ end(chunk?: string): StreamUpdate;
198+ snapshot(): Root;
199+}
200+```
201+
202+### Semantics
203+
204+- `write(chunk)` consumes incremental source text
205+- `end(chunk?)` finalizes stream and flushes buffered tail
206+- `snapshot()` returns current best-effort full HAST root
207+- stream is language-specific; caller chooses language up front
208+- `snapshot()` always returns a valid HAST root; it is safe to render at any point
209+- after `end()`, final assembled result must equal sync `highlight(language, fullSource)`
210+
211+### Stable vs unstable output
212+
213+Default streaming behavior should be conservative:
214+
215+- only finalized HAST fragments are emitted as `stable`
216+- unfinished trailing region stays buffered internally until it becomes stable or stream ends
217+- `unstable` is empty unless recall mode is enabled
218+
219+With `allowRecalls: true`:
220+
221+- `unstable` contains current best-effort trailing fragment(s)
222+- `recall` tells consumer how many previously emitted unstable root children to discard
223+- consumer applies update as:
224+ 1. remove last `recall` children from previously rendered unstable tail
225+ 2. append `stable`
226+ 3. append `unstable`
227+
228+This gives low-latency live highlighting without pretending unstable output is final.
229+
230+### Stability boundary
231+
232+v0 stability boundary is line-oriented:
233+
234+- completed lines become stable as soon as grammar state for that line is finalized
235+- current trailing line remains unstable until next newline or `end()`
236+
237+Important detail:
238+
239+- multi-line constructs must still work
240+- grammar state carries across lines
241+- line-oriented stability is only output policy, not grammar limitation
242+
243+## Language system
244+
245+`lextide` ships native language definitions. v0 does not need to consume `highlight.js` grammars.
246+
247+### `LanguageFn`
248+
249+Keep public export name `LanguageFn` for lowlight-style familiarity, but back it with `lextide`'s own typed grammar definition API.
250+
251+Requirements:
252+
253+- resumable grammar state for streaming
254+- nested scopes
255+- relevance scoring for auto-detect
256+- no HTML emission
257+- no dependence on regex-only implementation details
258+
259+### Grammar definition shape
260+
261+```ts
262+interface GrammarRule {
263+ match: RegExp | ((input: string, pos: number) => number | null);
264+ scope?: string;
265+ push?: string; // named state to push onto stack
266+ pop?: boolean; // pop current state
267+ children?: GrammarRule[];
268+ relevance?: number; // per-rule relevance contribution
269+}
270+
271+interface GrammarDefinition {
272+ name: string;
273+ aliases?: string[];
274+ relevance?: number; // base relevance for auto-detect
275+ states: Record<string, GrammarRule[]>;
276+ startState: string;
277+}
278+
279+type LanguageFn = () => GrammarDefinition;
280+```
281+
282+Grammar execution is a state stack:
283+
284+- engine starts in `startState`
285+- rules are tried in order within the current state
286+- `push` enters a named state; `pop` exits back to parent
287+- `children` define inline sub-rules active while a parent rule's scope is open
288+- grammar state (stack + position) is serializable for streaming resumption
289+
290+Relevance scoring:
291+
292+- each grammar has a base `relevance` (default 0)
293+- individual rules can contribute additional relevance when matched
294+- `highlightAuto` sums relevance across all matched rules and picks the highest-scoring language
295+
296+### Initial built-ins
297+
298+v0 built-ins:
299+
300+- bash
301+- javascript
302+- typescript
303+- python
304+
305+Notes:
306+
307+- javascript and typescript should share as much implementation as practical
308+- `all` and `common` can be identical in v0 while language coverage is still small
309+
310+## Internal architecture
311+
312+Implementation should split into three layers.
313+
314+### 1. Core highlighter engine
315+
316+Responsible for:
317+
318+- scanning source incrementally
319+- maintaining language state
320+- opening scopes
321+- closing scopes
322+- producing text segments
323+- scoring relevance
324+
325+Output should be an internal event stream or span stream, not HAST-specific logic.
326+
327+### 2. HAST builder
328+
329+Responsible for:
330+
331+- converting internal events into lowlight-compatible HAST
332+- merging adjacent text nodes
333+- building `className` arrays
334+- attaching `data.language` and `data.relevance`
335+
336+Same builder should be used by:
337+
338+- sync highlighting
339+- streaming stable fragments
340+- streaming snapshots
341+
342+### 3. Registry layer
343+
344+Responsible for:
345+
346+- language registration
347+- alias resolution
348+- built-in maps (`all`, `common`)
349+- `highlightAuto` traversal over registered languages
350+
351+## Performance requirements
352+
353+Performance is first-class.
354+
355+v0 requirements:
356+
357+- no HTML intermediate representation
358+- single-pass highlighting for sync path where possible
359+- streaming path reprocesses only buffered unstable tail, not full prior source
360+- grammar state is resumable across chunks
361+- minimize string slicing and node churn
362+- avoid per-character object allocation in hot path
363+
364+## Behavioral invariants
365+
366+These must hold:
367+
368+- `highlight()` and streamed `end()` produce same final HAST for same input
369+- aliases resolve the same way across sync and stream paths
370+- prefix option affects sync and stream paths the same way
371+- malformed code does not throw
372+- unknown language does throw
373+- adjacent text nodes are not duplicated by chunk boundaries
374+- chunk boundaries do not change final output
375+
376+## Testing plan
377+
378+### Compatibility tests
379+
380+For each built-in language:
381+
382+- fixture-based HAST tests for representative code samples
383+- CSS class compatibility tests
384+- alias tests
385+- prefix option tests
386+- `highlightAuto` tests
387+
388+### Streaming tests
389+
390+- same source split at many chunk boundaries yields same final HAST
391+- line-by-line input
392+- one-character-at-a-time input
393+- multiline strings/comments across chunk boundaries
394+- `allowRecalls` update behavior
395+- `snapshot()` matches current assembled state
396+
397+### Error and edge tests
398+
399+- unknown language throws exact error message
400+- empty input
401+- empty registry
402+- subset filtering for `highlightAuto`
403+- alias lookup and registration behavior
404+
405+### Performance tests
406+
407+Bench against:
408+
409+- `lowlight`
410+- `shiki-stream`-style incremental workloads for streaming mode
411+
412+Focus on:
413+
414+- total time
415+- allocations
416+- chunked update cost
417+
418+## v0 definition of done
419+
420+`lextide` v0 is done when it has:
421+
422+- public `lowlight`-compatible sync API
423+- HAST output parity for supported languages
424+- built-in bash/javascript/typescript/python grammars
425+- streaming API with stable mode and recall mode
426+- parity tests for sync and stream final output
427+- benchmark coverage showing replacement is competitive enough to justify switch
src/index.ts+1−0
@@ -0,0 +1 @@
1+export {};
tsconfig.json+12−0
@@ -0,0 +1,12 @@
1+{
2+ "compilerOptions": {
3+ "target": "ES2022",
4+ "lib": ["ES2022"],
5+ "module": "ESNext",
6+ "moduleResolution": "Bundler",
7+ "strict": true,
8+ "noEmit": true,
9+ "verbatimModuleSyntax": true
10+ },
11+ "include": ["src/**/*.ts", "test/**/*.ts"]
12+}