719
Le
lextide / files
lextide / README.md
6.8 KB
1# lextide 2 3`lextide` = TypeScript-first, HAST-first, streaming-capable rewrite of `lowlight`. 4 5Goal simple: 6 7- drop-in replacement for `lowlight` 8- return HAST, not HTML 9- keep lowlight-style API 10- add streaming 11 12## Status 13 14Current built-in languages: 15 16- bash 17- javascript 18- typescript 19- python 20- markdown 21 22`common` and `all` same for now. 23 24## Install 25 26```sh 27bun add lextide 28# or 29npm install lextide 30# or 31yarn add lextide 32# or 33pnpm add lextide 34``` 35 36## Quick start 37 38```ts 39import { common, createLowlight } from "lextide"; 40 41const lowlight = createLowlight(common); 42const tree = lowlight.highlight("javascript", "const answer = 42"); 43 44console.log(tree); 45``` 46 47Output shape: 48 49```ts 50{ 51 type: "root", 52 children: [ 53 { 54 type: "element", 55 tagName: "span", 56 properties: { className: ["hljs-keyword"] }, 57 children: [{ type: "text", value: "const" }] 58 }, 59 { type: "text", value: " answer = " }, 60 { 61 type: "element", 62 tagName: "span", 63 properties: { className: ["hljs-number"] }, 64 children: [{ type: "text", value: "42" }] 65 } 66 ], 67 data: { 68 language: "javascript", 69 relevance: 3 70 } 71} 72``` 73 74## Lowlight-compatible API 75 76```ts 77import { all, common, createLowlight } from "lextide"; 78``` 79 80Instance API: 81 82- `highlight(language, value, options?)` 83- `highlightAuto(value, options?)` 84- `listLanguages()` 85- `register(name, grammar)` 86- `register(grammars)` 87- `registerAlias(language, alias)` 88- `registerAlias(aliases)` 89- `registered(aliasOrName)` 90- `stream(language, options?)` 91 92## Built-ins 93 94```ts 95import { common, createLowlight } from "lextide"; 96 97const lowlight = createLowlight(common); 98 99console.log(lowlight.listLanguages()); 100// ["bash", "javascript", "typescript", "python", "markdown"] 101``` 102 103Useful built-in aliases: 104 105- javascript: `js`, `jsx`, `mjs`, `cjs` 106- typescript: `ts`, `tsx`, `mts`, `cts` 107- python: `py` 108- markdown: `md`, `mkdown`, `mkd` 109- bash: `sh`, `shell`, `zsh`, `bash` 110 111## `highlight()` 112 113```ts 114const tree = lowlight.highlight("python", "def greet():\n return 1\n"); 115``` 116 117Rules: 118 119- throws on unknown language 120- does best-effort highlight on malformed source 121- sets `root.data.language` 122- sets `root.data.relevance` 123 124Unknown language error: 125 126```ts 127lowlight.highlight("wat", "..."); 128// Error: Unknown language: `wat` is not registered 129``` 130 131## `highlightAuto()` 132 133```ts 134const tree = lowlight.highlightAuto("interface Person { name: string }"); 135``` 136 137Optional subset: 138 139```ts 140const tree = lowlight.highlightAuto(source, { 141 subset: ["javascript", "typescript"], 142}); 143``` 144 145Rules: 146 147- checks registered languages, or `subset` when given 148- returns best match by relevance 149- returns empty root when nothing useful matches 150 151## Options 152 153```ts 154interface Options { 155 prefix?: string | null | undefined; 156} 157``` 158 159Prefix behavior: 160 161- omitted or `undefined` -> `hljs-` 162- `null` -> no prefix 163- string -> custom prefix 164 165Example: 166 167```ts 168lowlight.highlight("javascript", "const x = 1", { prefix: "code-" }); 169``` 170 171## Streaming 172 173Streaming API give incremental updates. 174 175```ts 176const stream = lowlight.stream("typescript"); 177 178let children = []; 179 180children.push(...stream.write("interface Person {\n").stable); 181children.push(...stream.write(" name: string\n").stable); 182children.push(...stream.end("}\n").stable); 183 184console.log(stream.snapshot()); 185``` 186 187API: 188 189```ts 190interface StreamOptions extends Options { 191 allowRecalls?: boolean | null | undefined; 192} 193 194interface StreamUpdate { 195 recall: number; 196 stable: ElementContent[]; 197 unstable: ElementContent[]; 198} 199 200interface HighlightStream { 201 write(chunk: string): StreamUpdate; 202 end(chunk?: string): StreamUpdate; 203 snapshot(): Root; 204} 205``` 206 207### Stable mode 208 209Default mode conservative: 210 211- only stable nodes go to `stable` 212- `unstable` empty 213- final `end()` output matches sync `highlight()` output 214 215### Recall mode 216 217```ts 218const stream = lowlight.stream("javascript", { allowRecalls: true }); 219``` 220 221Now each update may include unstable tail: 222 223- `recall` = how many previously rendered unstable root children to remove 224- append `stable` 225- append `unstable` 226 227Consumer algorithm: 228 229```ts 230const rendered: ElementContent[] = []; 231 232function apply(update: StreamUpdate) { 233 if (update.recall > 0) { 234 rendered.splice(rendered.length - update.recall, update.recall); 235 } 236 237 rendered.push(...update.stable, ...update.unstable); 238} 239``` 240 241## HAST contract 242 243`lextide` returns HAST-like roots: 244 245```ts 246interface Root { 247 type: "root"; 248 children: ElementContent[]; 249 data?: { 250 language?: string; 251 relevance?: number; 252 }; 253} 254``` 255 256Node rules: 257 258- plain text = `{type: "text", value}` 259- scopes = `span` elements 260- classes live in `properties.className` 261- adjacent text nodes merged 262- source text preserved exactly 263 264Class serialization matches lowlight/highlight.js style: 265 266- first scope segment gets prefix 267- later dotted segments get underscore suffixes 268 269Example: 270 271- scope: `title.function` 272- classes: `['hljs-title', 'function_']` 273 274## Custom grammars 275 276You can register custom grammars. 277 278```ts 279import { createLowlight, type LanguageFn } from "lextide"; 280 281const toy: LanguageFn = () => ({ 282 name: "toy", 283 relevance: 1, 284 startState: "root", 285 states: { 286 root: [ 287 { match: /foo/, scope: "keyword", relevance: 2 }, 288 { match: /"[^"]*"/, scope: "string", relevance: 1 }, 289 ], 290 }, 291}); 292 293const lowlight = createLowlight(); 294lowlight.register("toy", toy); 295lowlight.registerAlias("toy", ["toybox", "toy-script"]); 296 297console.log(lowlight.highlight("toybox", 'foo "bar"')); 298``` 299 300Public grammar types: 301 302```ts 303interface GrammarRule { 304 match: RegExp | ((input: string, pos: number) => number | null); 305 scope?: string; 306 push?: string; 307 pop?: boolean; 308 children?: GrammarRule[]; 309 relevance?: number; 310} 311 312interface GrammarDefinition { 313 name: string; 314 aliases?: string[]; 315 relevance?: number; 316 states: Record<string, GrammarRule[]>; 317 startState: string; 318} 319 320type LanguageFn = () => GrammarDefinition; 321``` 322 323## Render yourself 324 325`lextide` no render HTML. 326It return HAST. 327Use whatever renderer fit app. 328 329Examples: 330 331- rehype / unified pipeline 332- custom React renderer 333- custom HTML serializer 334 335## Why `lextide` 336 337- lowlight-style API 338- HAST output only 339- no HTML intermediate step 340- streaming support 341- strict TypeScript 342- small built-in language set, focused on v0 quality 343 344## Current scope and caveats 345 346Current project target = solid v0 for builtin languages, not full `highlight.js` language coverage. 347 348Today: 349 350- builtins only: bash / javascript / typescript / python / markdown 351- `common` and `all` same 352- no HTML serializer included 353- grammar parity work still ongoing for builtin languages 354 355## Dev 356 357```sh 358bun run build 359bun run format 360bun run lint 361bun run typecheck 362bun run test 363bun run check 364bun run pack:dry 365``` 366 367## Release 368 369Local release flow: 370 371```sh 372git fetch origin 373git pull --ff-only origin main 374bun run check 375bun run build 376bun run pack:dry 377git tag -a v<version> -m "v<version>" 378git push origin main 379git push origin v<version> 380bun publish --access public 381``` 382 383Replace `<version>` with the value in `package.json`.