xonecas.io
719
Le

lextide / commit c131e83

feat: release lextide v0.1.0

xonecas2026-04-11 c131e83c4b879742a503347be48c59d8fa560a6a parent 83c1cb0

24 files changed

This diff is too big to show in full. The first 1000 lines are below, and every file is listed with its counts.

.github/workflows/publish.yml+49−0
@@ -0,0 +1,49 @@
1+name: publish
2+
3+on:
4+ push:
5+ tags:
6+ - "v*.*.*"
7+ workflow_dispatch:
8+
9+jobs:
10+ publish:
11+ runs-on: ubuntu-latest
12+ permissions:
13+ contents: read
14+ steps:
15+ - name: Checkout
16+ uses: actions/checkout@v4
17+ with:
18+ fetch-depth: 0
19+
20+ - name: Setup Bun
21+ uses: oven-sh/setup-bun@v2
22+ with:
23+ bun-version: 1.3.9
24+
25+ - name: Install deps
26+ run: bun install --frozen-lockfile
27+
28+ - name: Verify tag matches package version
29+ if: github.event_name == 'push'
30+ run: |
31+ VERSION="$(bun --print 'JSON.parse(await Bun.file("package.json").text()).version')"
32+ test "v${VERSION}" = "${GITHUB_REF_NAME}"
33+
34+ - name: Run checks
35+ run: bun run check
36+
37+ - name: Build package
38+ run: bun run build
39+
40+ - name: Dry-run publish
41+ run: bun publish --dry-run --access public
42+ env:
43+ NPM_CONFIG_TOKEN: ${{ secrets.NPM_TOKEN }}
44+
45+ - name: Publish to npm
46+ if: github.event_name == 'push'
47+ run: bun publish --access public
48+ env:
49+ NPM_CONFIG_TOKEN: ${{ secrets.NPM_TOKEN }}
AGENTS.md+29−12
@@ -21,8 +21,10 @@ Before doing anything, read:
2121
2222 - Repo has Bun/TypeScript/Biome/Lefthook scaffold
2323 - Git repo is initialized
24-- Implementation is still near-zero; `spec.md` is still source of truth
25-- Keep bootstrap changes small and aligned with planned public API
24+- Repo now has working v0 implementation for sync highlighting, auto-detect, registry/aliases, and streaming
25+- Built-in languages currently: bash, javascript, typescript, python
26+- Tests cover public lowlight-style contract plus streaming parity
27+- `spec.md` is still source of truth for future expansion and compatibility gaps
2628
2729 ## Non-negotiables
2830
@@ -115,11 +117,13 @@ When requirements are clear, act on them.
115117
116118 ## Scripts
117119
120+- `bun run build` — build publishable ESM output into `dist/` and emit declarations
118121 - `bun run format` — apply Biome formatting and import sorting
119122 - `bun run lint` — run Biome lint
120123 - `bun run typecheck` — run `tsc --noEmit`
121124 - `bun run test` — run Bun test runner
122125 - `bun run check` — run Biome check, typecheck, and tests
126+- `bun run pack:dry` — dry-run npm publish with Bun
123127
124128 ## Workflow
125129
@@ -129,7 +133,19 @@ When requirements are clear, act on them.
129133 - run `bun run format` before commit or handoff
130134 - remember pre-commit already runs Biome check, typecheck, and tests automatically on commit
131135 - run `bun run check` for final manual verification when needed, not as repetitive loop during development
132-- when committing code, if the user didn't explicitly ask you to, always review the diff with the user before committing.
136+- when user asks for release, publish, version bump, tag, or repo sync/push work, that counts as explicit permission to use git, create commits, create tags, and push them as needed for that release flow
137+- otherwise, when committing code, if the user didn't explicitly ask you to, always review the diff with the user before committing.
138+
139+## Release workflow
140+
141+- package version must use semver `X.Y.Z`
142+- before release, make sure `spec.md`, `README.md`, `AGENTS.md`, and `TODO.md` agree on current public API and scope
143+- build release artifacts with `bun run build`
144+- inspect publish payload with `bun run pack:dry`
145+- publish with `bun publish --access public`
146+- GitHub Actions workflow lives at `.github/workflows/publish.yml`
147+- tag-triggered publish expects `NPM_TOKEN` secret and tag name `v<package.json version>`
148+- release flow should sync local `main` with `origin/main`, create annotated tag, push tag, then let workflow publish
133149
134150 ## Spec ambiguities
135151
@@ -148,17 +164,18 @@ When a requirement is unclear or missing from `spec.md`:
148164 - Verify claims against actual `lowlight` behavior when compatibility matters
149165 - If researching streaming behavior, check `shiki-stream`
150166
151-## If starting implementation
167+## If extending implementation
168+
169+Current implementation shape:
152170
153-First useful steps:
171+1. `src/core/engine.ts` — generic scanner/state engine
172+2. `src/hast.ts` — HAST builder + clone helpers
173+3. `src/registry.ts` — lowlight-compatible API and registry
174+4. `src/stream.ts` — streaming API; stable mode keeps last root child buffered until future input proves it is closed
175+5. `src/languages/*.ts` — built-in language definitions
176+6. `test/lextide.test.ts` — sync/auto/stream contract coverage
154177
155-1. define public types and `createLowlight()` API
156-2. build internal HAST builder
157-3. build registry + aliases
158-4. add sync path for one language
159-5. add parity tests with Bun test
160-6. add streaming API
161-7. expand languages
178+When adding behavior, extend existing layers instead of bypassing them.
162179
163180 ## Session bootstrap checklist
164181
README.md+363−0
@@ -0,0 +1,363 @@
1+# lextide
2+
3+`lextide` = TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`.
4+
5+Goal simple:
6+
7+- drop-in replacement for `lowlight`
8+- return HAST, not HTML
9+- keep lowlight-style API
10+- add streaming
11+
12+## Status
13+
14+Current built-in languages:
15+
16+- bash
17+- javascript
18+- typescript
19+- python
20+
21+`common` and `all` same for now.
22+
23+## Install
24+
25+```sh
26+bun add lextide
27+# or
28+npm install lextide
29+# or
30+yarn add lextide
31+# or
32+pnpm add lextide
33+```
34+
35+## Quick start
36+
37+```ts
38+import { common, createLowlight } from "lextide";
39+
40+const lowlight = createLowlight(common);
41+const tree = lowlight.highlight("javascript", "const answer = 42");
42+
43+console.log(tree);
44+```
45+
46+Output shape:
47+
48+```ts
49+{
50+ type: "root",
51+ children: [
52+ {
53+ type: "element",
54+ tagName: "span",
55+ properties: { className: ["hljs-keyword"] },
56+ children: [{ type: "text", value: "const" }]
57+ },
58+ { type: "text", value: " answer = " },
59+ {
60+ type: "element",
61+ tagName: "span",
62+ properties: { className: ["hljs-number"] },
63+ children: [{ type: "text", value: "42" }]
64+ }
65+ ],
66+ data: {
67+ language: "javascript",
68+ relevance: 3
69+ }
70+}
71+```
72+
73+## Lowlight-compatible API
74+
75+```ts
76+import { all, common, createLowlight } from "lextide";
77+```
78+
79+Instance API:
80+
81+- `highlight(language, value, options?)`
82+- `highlightAuto(value, options?)`
83+- `listLanguages()`
84+- `register(name, grammar)`
85+- `register(grammars)`
86+- `registerAlias(language, alias)`
87+- `registerAlias(aliases)`
88+- `registered(aliasOrName)`
89+- `stream(language, options?)`
90+
91+## Built-ins
92+
93+```ts
94+import { common, createLowlight } from "lextide";
95+
96+const lowlight = createLowlight(common);
97+
98+console.log(lowlight.listLanguages());
99+// ["bash", "javascript", "typescript", "python"]
100+```
101+
102+Useful built-in aliases:
103+
104+- javascript: `js`, `jsx`, `mjs`, `cjs`
105+- typescript: `ts`, `tsx`, `mts`, `cts`
106+- python: `py`
107+- bash: `sh`, `shell`, `zsh`, `bash`
108+
109+## `highlight()`
110+
111+```ts
112+const tree = lowlight.highlight("python", "def greet():\n return 1\n");
113+```
114+
115+Rules:
116+
117+- throws on unknown language
118+- does best-effort highlight on malformed source
119+- sets `root.data.language`
120+- sets `root.data.relevance`
121+
122+Unknown language error:
123+
124+```ts
125+lowlight.highlight("wat", "...");
126+// Error: Unknown language: `wat` is not registered
127+```
128+
129+## `highlightAuto()`
130+
131+```ts
132+const tree = lowlight.highlightAuto("interface Person { name: string }");
133+```
134+
135+Optional subset:
136+
137+```ts
138+const tree = lowlight.highlightAuto(source, {
139+ subset: ["javascript", "typescript"],
140+});
141+```
142+
143+Rules:
144+
145+- checks registered languages, or `subset` when given
146+- returns best match by relevance
147+- returns empty root when nothing useful matches
148+
149+## Options
150+
151+```ts
152+interface Options {
153+ prefix?: string | null | undefined;
154+}
155+```
156+
157+Prefix behavior:
158+
159+- omitted or `undefined` -> `hljs-`
160+- `null` -> no prefix
161+- string -> custom prefix
162+
163+Example:
164+
165+```ts
166+lowlight.highlight("javascript", "const x = 1", { prefix: "code-" });
167+```
168+
169+## Streaming
170+
171+Streaming API give incremental updates.
172+
173+```ts
174+const stream = lowlight.stream("typescript");
175+
176+let children = [];
177+
178+children.push(...stream.write("interface Person {\n").stable);
179+children.push(...stream.write(" name: string\n").stable);
180+children.push(...stream.end("}\n").stable);
181+
182+console.log(stream.snapshot());
183+```
184+
185+API:
186+
187+```ts
188+interface StreamOptions extends Options {
189+ allowRecalls?: boolean | null | undefined;
190+}
191+
192+interface StreamUpdate {
193+ recall: number;
194+ stable: ElementContent[];
195+ unstable: ElementContent[];
196+}
197+
198+interface HighlightStream {
199+ write(chunk: string): StreamUpdate;
200+ end(chunk?: string): StreamUpdate;
201+ snapshot(): Root;
202+}
203+```
204+
205+### Stable mode
206+
207+Default mode conservative:
208+
209+- only stable nodes go to `stable`
210+- `unstable` empty
211+- final `end()` output matches sync `highlight()` output
212+
213+### Recall mode
214+
215+```ts
216+const stream = lowlight.stream("javascript", { allowRecalls: true });
217+```
218+
219+Now each update may include unstable tail:
220+
221+- `recall` = how many previously rendered unstable root children to remove
222+- append `stable`
223+- append `unstable`
224+
225+Consumer algorithm:
226+
227+```ts
228+const rendered: ElementContent[] = [];
229+
230+function apply(update: StreamUpdate) {
231+ if (update.recall > 0) {
232+ rendered.splice(rendered.length - update.recall, update.recall);
233+ }
234+
235+ rendered.push(...update.stable, ...update.unstable);
236+}
237+```
238+
239+## HAST contract
240+
241+`lextide` returns HAST-like roots:
242+
243+```ts
244+interface Root {
245+ type: "root";
246+ children: ElementContent[];
247+ data?: {
248+ language?: string;
249+ relevance?: number;
250+ };
251+}
252+```
253+
254+Node rules:
255+
256+- plain text = `{type: "text", value}`
257+- scopes = `span` elements
258+- classes live in `properties.className`
259+- adjacent text nodes merged
260+- source text preserved exactly
261+
262+Class serialization matches lowlight/highlight.js style:
263+
264+- first scope segment gets prefix
265+- later dotted segments get underscore suffixes
266+
267+Example:
268+
269+- scope: `title.function`
270+- classes: `['hljs-title', 'function_']`
271+
272+## Custom grammars
273+
274+You can register custom grammars.
275+
276+```ts
277+import { createLowlight, type LanguageFn } from "lextide";
278+
279+const toy: LanguageFn = () => ({
280+ name: "toy",
281+ relevance: 1,
282+ startState: "root",
283+ states: {
284+ root: [
285+ { match: /foo/, scope: "keyword", relevance: 2 },
286+ { match: /"[^"]*"/, scope: "string", relevance: 1 },
287+ ],
288+ },
289+});
290+
291+const lowlight = createLowlight();
292+lowlight.register("toy", toy);
293+lowlight.registerAlias("toy", ["toybox", "toy-script"]);
294+
295+console.log(lowlight.highlight("toybox", 'foo "bar"'));
296+```
297+
298+Public grammar types:
299+
300+```ts
301+interface GrammarRule {
302+ match: RegExp | ((input: string, pos: number) => number | null);
303+ scope?: string;
304+ push?: string;
305+ pop?: boolean;
306+ children?: GrammarRule[];
307+ relevance?: number;
308+}
309+
310+interface GrammarDefinition {
311+ name: string;
312+ aliases?: string[];
313+ relevance?: number;
314+ states: Record<string, GrammarRule[]>;
315+ startState: string;
316+}
317+
318+type LanguageFn = () => GrammarDefinition;
319+```
320+
321+## Render yourself
322+
323+`lextide` no render HTML.
324+It return HAST.
325+Use whatever renderer fit app.
326+
327+Examples:
328+
329+- rehype / unified pipeline
330+- custom React renderer
331+- custom HTML serializer
332+
333+## Why `lextide`
334+
335+- lowlight-style API
336+- HAST output only
337+- no HTML intermediate step
338+- streaming support
339+- strict TypeScript
340+- small built-in language set, focused on v0 quality
341+
342+## Current scope and caveats
343+
344+Current project target = solid v0 for builtin languages, not full `highlight.js` language coverage.
345+
346+Today:
347+
348+- builtins only: bash / javascript / typescript / python
349+- `common` and `all` same
350+- no HTML serializer included
351+- grammar parity work still ongoing for builtin languages
352+
353+## Dev
354+
355+```sh
356+bun run build
357+bun run format
358+bun run lint
359+bun run typecheck
360+bun run test
361+bun run check
362+bun run pack:dry
363+```
TODO.md+6−0
@@ -19,3 +19,9 @@
1919 - [x] Use left-hook to install pre-commit hook that checks test, biome format, biome lint and typescript types.
2020 - [x] Update AGENTS.md with the new scripts and process. Correct anything that is outdated or not specified.
2121 - [x] Update TODO.md with your progress
22+- [x] Implement tests to match spec and lowlight user contract
23+- [x] Implement the spec.
24+- [x] Review implementationg, fix bugs, ensure complete implementation.
25+- [x] Update AGENTS.md with the new scripts and process. Correct anything that is outdated or not specified.
26+- [x] Update TODO.md with your progress
27+- [ ] complete language parity for builtin languages
biome.json+3−0
@@ -20,5 +20,8 @@
2020 "noExcessiveNestedTestSuites": "error"
2121 }
2222 }
23+ },
24+ "files": {
25+ "includes": ["**", "!dist", "!node_modules"]
2326 }
2427 }
package.json+19−1
@@ -1,16 +1,34 @@
11 {
22 "name": "lextide",
3- "version": "0.0.0",
3+ "version": "0.1.0",
44 "description": "TypeScript-first, HAST-first, streaming-capable rewrite of lowlight.",
55 "type": "module",
66 "packageManager": "bun@1.3.9",
7+ "main": "./dist/index.js",
8+ "types": "./dist/index.d.ts",
9+ "exports": {
10+ ".": {
11+ "types": "./dist/index.d.ts",
12+ "import": "./dist/index.js"
13+ }
14+ },
15+ "files": [
16+ "dist",
17+ "README.md"
18+ ],
19+ "publishConfig": {
20+ "access": "public"
21+ },
722 "scripts": {
23+ "build": "rm -rf dist && bun build ./src/index.ts --outdir ./dist --target node --format esm && tsc -p tsconfig.build.json",
824 "test": "bun test --pass-with-no-tests",
925 "format": "biome check --write .",
1026 "format:docs": "prettier --write \"**/*.md\"",
1127 "lint": "biome lint .",
1228 "typecheck": "tsc --noEmit",
1329 "check": "biome check . && bun run typecheck && bun run test",
30+ "pack:dry": "bun publish --dry-run --access public",
31+ "prepublishOnly": "bun run build",
1432 "prepare": "lefthook install"
1533 },
1634 "devDependencies": {
src/builtins.ts+19−0
@@ -0,0 +1,19 @@
1+import { bash } from "./languages/bash";
2+import { javascript } from "./languages/javascript";
3+import { python } from "./languages/python";
4+import { typescript } from "./languages/typescript";
5+import type { LanguageFn } from "./types";
6+
7+export const common: Readonly<Record<string, LanguageFn>> = {
8+ bash,
9+ javascript,
10+ typescript,
11+ python,
12+};
13+
14+export const all: Readonly<Record<string, LanguageFn>> = {
15+ bash,
16+ javascript,
17+ typescript,
18+ python,
19+};
src/core/engine.ts+518−0
@@ -0,0 +1,518 @@
1+import type { GrammarDefinition } from "../types";
2+import {
3+ getInternalStates,
4+ type InternalMatcher,
5+ type InternalRule,
6+ type MatchResult,
7+} from "./grammar";
8+
9+const stickyExpressions = new WeakMap<RegExp, RegExp>();
10+
11+interface ResolvedRule {
12+ children?: InternalRule[] | undefined;
13+ length: number;
14+ pop: boolean;
15+ push?: string | undefined;
16+ pushScope?: string | null | undefined;
17+ relevance: number;
18+ scope?: string | undefined;
19+}
20+
21+export interface Segment {
22+ text: string;
23+ scopes: string[];
24+ start: number;
25+ end: number;
26+ stateAfter: string[];
27+}
28+
29+export interface ParseResult {
30+ segments: Segment[];
31+ relevance: number;
32+ state: string[];
33+}
34+
35+export interface SegmentGroup {
36+ segments: Segment[];
37+ end: number;
38+ stateAfter: string[];
39+}
40+
41+export function parseSource(
42+ grammar: GrammarDefinition,
43+ input: string,
44+ initialState: readonly string[] = [grammar.startState],
45+): ParseResult {
46+ const states = getInternalStates(grammar);
47+ const stack =
48+ initialState.length > 0 ? [...initialState] : [grammar.startState];
49+ const segments: Segment[] = [];
50+ let relevance = 0;
51+ let position = 0;
52+
53+ while (position < input.length) {
54+ const state = currentState(states, stack);
55+ const resolved = matchStateRules(state.rules, input, position);
56+
57+ if (!resolved) {
58+ position = consumePlainSegment(states, stack, segments, input, position);
59+ continue;
60+ }
61+
62+ position = consumeResolvedSegment(
63+ states,
64+ stack,
65+ segments,
66+ input,
67+ position,
68+ resolved,
69+ );
70+ relevance += resolved.relevance;
71+ }
72+
73+ return { segments, relevance, state: stack };
74+}
75+
76+export function groupSegmentsByRootChild(
77+ segments: readonly Segment[],
78+): SegmentGroup[] {
79+ if (segments.length === 0) {
80+ return [];
81+ }
82+
83+ const groups: SegmentGroup[] = [];
84+ let currentKey = rootChildKey(segments[0]);
85+ let currentSegments: Segment[] = [segments[0]];
86+
87+ for (let index = 1; index < segments.length; index += 1) {
88+ const segment = segments[index];
89+ const key = rootChildKey(segment);
90+
91+ if (key === currentKey) {
92+ currentSegments.push(segment);
93+ continue;
94+ }
95+
96+ groups.push(createGroup(currentSegments));
97+ currentSegments = [segment];
98+ currentKey = key;
99+ }
100+
101+ groups.push(createGroup(currentSegments));
102+
103+ return groups;
104+}
105+
106+function consumePlainSegment(
107+ states: ReturnType<typeof getInternalStates>,
108+ stack: readonly string[],
109+ segments: Segment[],
110+ input: string,
111+ position: number,
112+): number {
113+ const end = position + codePointLength(input, position);
114+
115+ pushSegment(
116+ segments,
117+ input.slice(position, end),
118+ activeScopes(states, stack),
119+ position,
120+ end,
121+ stack,
122+ );
123+
124+ return end;
125+}
126+
127+function consumeResolvedSegment(
128+ states: ReturnType<typeof getInternalStates>,
129+ stack: string[],
130+ segments: Segment[],
131+ input: string,
132+ position: number,
133+ resolved: ResolvedRule,
134+): number {
135+ const end = position + resolved.length;
136+ const text = input.slice(position, end);
137+ const scopes = activeScopes(
138+ states,
139+ stack,
140+ resolved.pushScope,
141+ resolved.scope,
142+ );
143+
144+ emitResolvedSegments(segments, text, scopes, position, stack, resolved);
145+ applyStateTransition(stack, resolved);
146+ updateTailState(segments, stack);
147+
148+ return end;
149+}
150+
151+function currentState(
152+ states: ReturnType<typeof getInternalStates>,
153+ stack: readonly string[],
154+) {
155+ const stateName = stack[stack.length - 1];
156+ const state = states[stateName];
157+
158+ if (!state) {
159+ throw new Error(`Unknown grammar state: ${stateName}`);
160+ }
161+
162+ return state;
163+}
164+
165+function emitResolvedSegments(
166+ segments: Segment[],
167+ text: string,
168+ scopes: readonly string[],
169+ position: number,
170+ stack: readonly string[],
171+ resolved: ResolvedRule,
172+): void {
173+ if (resolved.children && resolved.children.length > 0) {
174+ const childSegments = parseChildren(
175+ text,
176+ resolved.children,
177+ scopes,
178+ position,
179+ stack,
180+ );
181+
182+ for (const childSegment of childSegments) {
183+ pushSegment(
184+ segments,
185+ childSegment.text,
186+ childSegment.scopes,
187+ childSegment.start,
188+ childSegment.end,
189+ stack,
190+ );
191+ }
192+
193+ return;
194+ }
195+
196+ pushSegment(
197+ segments,
198+ text,
199+ [...scopes],
200+ position,
201+ position + resolved.length,
202+ stack,
203+ );
204+}
205+
206+function applyStateTransition(stack: string[], resolved: ResolvedRule): void {
207+ if (resolved.push) {
208+ stack.push(resolved.push);
209+ }
210+
211+ if (resolved.pop && stack.length > 1) {
212+ stack.pop();
213+ }
214+}
215+
216+function updateTailState(segments: Segment[], stack: readonly string[]): void {
217+ const tail = segments[segments.length - 1];
218+
219+ if (tail) {
220+ tail.stateAfter = [...stack];
221+ }
222+}
223+
224+function activeScopes(
225+ states: ReturnType<typeof getInternalStates>,
226+ stack: readonly string[],
227+ pushScope?: string | null,
228+ transientScope?: string,
229+): string[] {
230+ const scopes = stack
231+ .map((stateName) => states[stateName]?.defaultScope)
232+ .filter((scope): scope is string => typeof scope === "string");
233+
234+ if (typeof pushScope === "string") {
235+ scopes.push(pushScope);
236+ }
237+
238+ if (typeof transientScope === "string") {
239+ scopes.push(transientScope);
240+ }
241+
242+ return scopes;
243+}
244+
245+function codePointLength(input: string, position: number): number {
246+ const codePoint = input.codePointAt(position);
247+
248+ if (codePoint === undefined) {
249+ return 1;
250+ }
251+
252+ return codePoint > 0xffff ? 2 : 1;
253+}
254+
255+function createGroup(segments: Segment[]): SegmentGroup {
256+ const tail = segments[segments.length - 1];
257+
258+ return {
259+ segments,
260+ end: tail.end,
261+ stateAfter: tail.stateAfter,
262+ };
263+}
264+
265+function matchExpression(
266+ expression: RegExp,
267+ input: string,
268+ position: number,
269+): string | null {
270+ const sticky = getStickyExpression(expression);
271+ sticky.lastIndex = position;
272+ const match = sticky.exec(input);
273+
274+ if (!match || match[0].length === 0) {
275+ return null;
276+ }
277+
278+ return match[0];
279+}
280+
281+function matchStateRules(
282+ rules: readonly InternalRule[],
283+ input: string,
284+ position: number,
285+): ResolvedRule | null {
286+ for (const rule of rules) {
287+ const resolved = resolveRule(rule, input, position);
288+
289+ if (resolved) {
290+ return resolved;
291+ }
292+ }
293+
294+ return null;
295+}
296+
297+function parseChildren(
298+ input: string,
299+ rules: readonly InternalRule[],
300+ baseScopes: readonly string[],
301+ offset: number,
302+ stateAfter: readonly string[],
303+): Segment[] {
304+ const segments: Segment[] = [];
305+ let position = 0;
306+
307+ while (position < input.length) {
308+ const resolved = matchStateRules(rules, input, position);
309+
310+ if (!resolved) {
311+ const length = codePointLength(input, position);
312+ const end = position + length;
313+
314+ pushSegment(
315+ segments,
316+ input.slice(position, end),
317+ [...baseScopes],
318+ offset + position,
319+ offset + end,
320+ stateAfter,
321+ );
322+
323+ position = end;
324+ continue;
325+ }
326+
327+ const end = position + resolved.length;
328+ const text = input.slice(position, end);
329+ const scopes = [...baseScopes];
330+
331+ if (resolved.scope) {
332+ scopes.push(resolved.scope);
333+ }
334+
335+ if (resolved.children && resolved.children.length > 0) {
336+ const childSegments = parseChildren(
337+ text,
338+ resolved.children,
339+ scopes,
340+ offset + position,
341+ stateAfter,
342+ );
343+
344+ for (const childSegment of childSegments) {
345+ pushSegment(
346+ segments,
347+ childSegment.text,
348+ childSegment.scopes,
349+ childSegment.start,
350+ childSegment.end,
351+ stateAfter,
352+ );
353+ }
354+ } else {
355+ pushSegment(
356+ segments,
357+ text,
358+ scopes,
359+ offset + position,
360+ offset + end,
361+ stateAfter,
362+ );
363+ }
364+
365+ position = end;
366+ }
367+
368+ return segments;
369+}
370+
371+function pushSegment(
372+ segments: Segment[],
373+ text: string,
374+ scopes: string[],
375+ start: number,
376+ end: number,
377+ stateAfter: readonly string[],
378+): void {
379+ if (text.length === 0) {
380+ return;
381+ }
382+
383+ const tail = segments[segments.length - 1];
384+
385+ if (tail && tail.end === start && sameScopes(tail.scopes, scopes)) {
386+ tail.text += text;
387+ tail.end = end;
388+ tail.stateAfter = [...stateAfter];
389+ return;
390+ }
391+
392+ segments.push({
393+ text,
394+ scopes,
395+ start,
396+ end,
397+ stateAfter: [...stateAfter],
398+ });
399+}
400+
401+function resolveRule(
402+ rule: InternalRule,
403+ input: string,
404+ position: number,
405+): ResolvedRule | null {
406+ const matched = matchRule(rule, input, position);
407+
408+ if (!isValidMatch(matched)) {
409+ return null;
410+ }
411+
412+ if (typeof matched === "object" && matched !== null) {
413+ return buildResolvedRuleFromObject(rule, matched);
414+ }
415+
416+ return buildResolvedRuleFromLength(rule, matchLength(matched));
417+}
418+
419+function matchRule(
420+ rule: InternalRule,
421+ input: string,
422+ position: number,
423+): MatchResult | string {
424+ return rule.match instanceof RegExp
425+ ? matchExpression(rule.match, input, position)
426+ : resolveMatcher(rule.match, input, position);
427+}
428+
429+function resolveMatcher(
430+ matcher: InternalMatcher,
431+ input: string,
432+ position: number,
433+): MatchResult {
434+ return matcher(input, position);
Cut short: this commit's diff is over 1000 lines.
src/core/grammar.ts+74−0
Cut short: this commit's diff is over 1000 lines.
src/hast.ts+136−0
Cut short: this commit's diff is over 1000 lines.
src/index.ts+18−1
Cut short: this commit's diff is over 1000 lines.
src/languages/bash.ts+752−0
Cut short: this commit's diff is over 1000 lines.
src/languages/ecmascript.ts+404−0
Cut short: this commit's diff is over 1000 lines.
src/languages/helpers.ts+310−0
Cut short: this commit's diff is over 1000 lines.
src/languages/javascript.ts+51−0
Cut short: this commit's diff is over 1000 lines.
src/languages/python.ts+321−0
Cut short: this commit's diff is over 1000 lines.
src/languages/typescript.ts+83−0
Cut short: this commit's diff is over 1000 lines.
src/registry.ts+197−0
Cut short: this commit's diff is over 1000 lines.
src/stream.ts+133−0
Cut short: this commit's diff is over 1000 lines.
src/types.ts+83−0
Cut short: this commit's diff is over 1000 lines.
test/bun-test.d.ts+21−0
Cut short: this commit's diff is over 1000 lines.
test/lextide.test.ts+551−0
Cut short: this commit's diff is over 1000 lines.
tsconfig.build.json+11−0
Cut short: this commit's diff is over 1000 lines.
tsconfig.json+1−1
Cut short: this commit's diff is over 1000 lines.