xonecas.io
719
Le

lextide / files

lextide / README.md

6.8 KB
1# lextide
2
3`lextide` = TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`.
4
5Goal simple:
6
7- drop-in replacement for `lowlight`
8- return HAST, not HTML
9- keep lowlight-style API
10- add streaming
11
12## Status
13
14Current built-in languages:
15
16- bash
17- javascript
18- typescript
19- python
20- markdown
21
22`common` and `all` same for now.
23
24## Install
25
26```sh
27bun add lextide
28# or
29npm install lextide
30# or
31yarn add lextide
32# or
33pnpm add lextide
34```
35
36## Quick start
37
38```ts
39import { common, createLowlight } from "lextide";
40
41const lowlight = createLowlight(common);
42const tree = lowlight.highlight("javascript", "const answer = 42");
43
44console.log(tree);
45```
46
47Output shape:
48
49```ts
50{
51  type: "root",
52  children: [
53    {
54      type: "element",
55      tagName: "span",
56      properties: { className: ["hljs-keyword"] },
57      children: [{ type: "text", value: "const" }]
58    },
59    { type: "text", value: " answer = " },
60    {
61      type: "element",
62      tagName: "span",
63      properties: { className: ["hljs-number"] },
64      children: [{ type: "text", value: "42" }]
65    }
66  ],
67  data: {
68    language: "javascript",
69    relevance: 3
70  }
71}
72```
73
74## Lowlight-compatible API
75
76```ts
77import { all, common, createLowlight } from "lextide";
78```
79
80Instance API:
81
82- `highlight(language, value, options?)`
83- `highlightAuto(value, options?)`
84- `listLanguages()`
85- `register(name, grammar)`
86- `register(grammars)`
87- `registerAlias(language, alias)`
88- `registerAlias(aliases)`
89- `registered(aliasOrName)`
90- `stream(language, options?)`
91
92## Built-ins
93
94```ts
95import { common, createLowlight } from "lextide";
96
97const lowlight = createLowlight(common);
98
99console.log(lowlight.listLanguages());
100// ["bash", "javascript", "typescript", "python", "markdown"]
101```
102
103Useful built-in aliases:
104
105- javascript: `js`, `jsx`, `mjs`, `cjs`
106- typescript: `ts`, `tsx`, `mts`, `cts`
107- python: `py`
108- markdown: `md`, `mkdown`, `mkd`
109- bash: `sh`, `shell`, `zsh`, `bash`
110
111## `highlight()`
112
113```ts
114const tree = lowlight.highlight("python", "def greet():\n    return 1\n");
115```
116
117Rules:
118
119- throws on unknown language
120- does best-effort highlight on malformed source
121- sets `root.data.language`
122- sets `root.data.relevance`
123
124Unknown language error:
125
126```ts
127lowlight.highlight("wat", "...");
128// Error: Unknown language: `wat` is not registered
129```
130
131## `highlightAuto()`
132
133```ts
134const tree = lowlight.highlightAuto("interface Person { name: string }");
135```
136
137Optional subset:
138
139```ts
140const tree = lowlight.highlightAuto(source, {
141  subset: ["javascript", "typescript"],
142});
143```
144
145Rules:
146
147- checks registered languages, or `subset` when given
148- returns best match by relevance
149- returns empty root when nothing useful matches
150
151## Options
152
153```ts
154interface Options {
155  prefix?: string | null | undefined;
156}
157```
158
159Prefix behavior:
160
161- omitted or `undefined` -> `hljs-`
162- `null` -> no prefix
163- string -> custom prefix
164
165Example:
166
167```ts
168lowlight.highlight("javascript", "const x = 1", { prefix: "code-" });
169```
170
171## Streaming
172
173Streaming API give incremental updates.
174
175```ts
176const stream = lowlight.stream("typescript");
177
178let children = [];
179
180children.push(...stream.write("interface Person {\n").stable);
181children.push(...stream.write("  name: string\n").stable);
182children.push(...stream.end("}\n").stable);
183
184console.log(stream.snapshot());
185```
186
187API:
188
189```ts
190interface StreamOptions extends Options {
191  allowRecalls?: boolean | null | undefined;
192}
193
194interface StreamUpdate {
195  recall: number;
196  stable: ElementContent[];
197  unstable: ElementContent[];
198}
199
200interface HighlightStream {
201  write(chunk: string): StreamUpdate;
202  end(chunk?: string): StreamUpdate;
203  snapshot(): Root;
204}
205```
206
207### Stable mode
208
209Default mode conservative:
210
211- only stable nodes go to `stable`
212- `unstable` empty
213- final `end()` output matches sync `highlight()` output
214
215### Recall mode
216
217```ts
218const stream = lowlight.stream("javascript", { allowRecalls: true });
219```
220
221Now each update may include unstable tail:
222
223- `recall` = how many previously rendered unstable root children to remove
224- append `stable`
225- append `unstable`
226
227Consumer algorithm:
228
229```ts
230const rendered: ElementContent[] = [];
231
232function apply(update: StreamUpdate) {
233  if (update.recall > 0) {
234    rendered.splice(rendered.length - update.recall, update.recall);
235  }
236
237  rendered.push(...update.stable, ...update.unstable);
238}
239```
240
241## HAST contract
242
243`lextide` returns HAST-like roots:
244
245```ts
246interface Root {
247  type: "root";
248  children: ElementContent[];
249  data?: {
250    language?: string;
251    relevance?: number;
252  };
253}
254```
255
256Node rules:
257
258- plain text = `{type: "text", value}`
259- scopes = `span` elements
260- classes live in `properties.className`
261- adjacent text nodes merged
262- source text preserved exactly
263
264Class serialization matches lowlight/highlight.js style:
265
266- first scope segment gets prefix
267- later dotted segments get underscore suffixes
268
269Example:
270
271- scope: `title.function`
272- classes: `['hljs-title', 'function_']`
273
274## Custom grammars
275
276You can register custom grammars.
277
278```ts
279import { createLowlight, type LanguageFn } from "lextide";
280
281const toy: LanguageFn = () => ({
282  name: "toy",
283  relevance: 1,
284  startState: "root",
285  states: {
286    root: [
287      { match: /foo/, scope: "keyword", relevance: 2 },
288      { match: /"[^"]*"/, scope: "string", relevance: 1 },
289    ],
290  },
291});
292
293const lowlight = createLowlight();
294lowlight.register("toy", toy);
295lowlight.registerAlias("toy", ["toybox", "toy-script"]);
296
297console.log(lowlight.highlight("toybox", 'foo "bar"'));
298```
299
300Public grammar types:
301
302```ts
303interface GrammarRule {
304  match: RegExp | ((input: string, pos: number) => number | null);
305  scope?: string;
306  push?: string;
307  pop?: boolean;
308  children?: GrammarRule[];
309  relevance?: number;
310}
311
312interface GrammarDefinition {
313  name: string;
314  aliases?: string[];
315  relevance?: number;
316  states: Record<string, GrammarRule[]>;
317  startState: string;
318}
319
320type LanguageFn = () => GrammarDefinition;
321```
322
323## Render yourself
324
325`lextide` no render HTML.
326It return HAST.
327Use whatever renderer fit app.
328
329Examples:
330
331- rehype / unified pipeline
332- custom React renderer
333- custom HTML serializer
334
335## Why `lextide`
336
337- lowlight-style API
338- HAST output only
339- no HTML intermediate step
340- streaming support
341- strict TypeScript
342- small built-in language set, focused on v0 quality
343
344## Current scope and caveats
345
346Current project target = solid v0 for builtin languages, not full `highlight.js` language coverage.
347
348Today:
349
350- builtins only: bash / javascript / typescript / python / markdown
351- `common` and `all` same
352- no HTML serializer included
353- grammar parity work still ongoing for builtin languages
354
355## Dev
356
357```sh
358bun run build
359bun run format
360bun run lint
361bun run typecheck
362bun run test
363bun run check
364bun run pack:dry
365```
366
367## Release
368
369Local release flow:
370
371```sh
372git fetch origin
373git pull --ff-only origin main
374bun run check
375bun run build
376bun run pack:dry
377git tag -a v<version> -m "v<version>"
378git push origin main
379git push origin v<version>
380bun publish --access public
381```
382
383Replace `<version>` with the value in `package.json`.