xonecas.io
70160
Ce

cel-tui / commit a5c3855

refactor(components): use lextide for syntax highlighting

xonecas2026-04-11 a5c385558cfe550970ace1e39b6f0f55ed6b5dbe parent a687a5a

8 files changed

TODO.md+0−13
@@ -32,14 +32,6 @@ Remaining work, known bugs, and planned improvements.
3232
3333 - ❌ **Markdown heading inline styling** — Headings (`#`, `##`, `###`) still strip inline formatting to plain text. Since headings are short and single-line, this is low priority. Paragraphs, list items, and blockquotes now render inline formatting via wrapping HStack.
3434
35-## Syntax Highlighting
36-
37-- ❌ **Shell / terminal language highlighting needs a better backend** — `SyntaxHighlight` currently uses `lowlight` / `highlight.js`. Local checks in this repo show `bash`, `shell`, `shellsession`, and `zsh` are registered, but `fish` and `tmux` are not. Bash quality is also a known upstream weak spot: highlight.js issues call out heredocs, redirects, line continuations, and switch / prompt mis-highlighting, and maintainers explicitly describe bash as "a very simple grammar" that needs a cha …
38- 1. **Stay on lowlight + register extra grammars/aliases** — lowest migration cost and keeps the current synchronous model, but highlight.js pushes niche language support to third-party grammars and a quick npm search did not turn up obvious maintained `fish` / `tmux` packages.
39- 2. **Prototype Shiki backend** — TextMate / VS Code grammar ecosystem, documented `shellsession` and `ansi` support, and a JavaScript regex engine option that avoids WASM. Costs: async initialization, more dependency / bundle weight, and a new TextMate-scope-to-cel-style mapping layer.
40- 3. **Prototype starry-night or tree-sitter backend** — starry-night gives GitHub-style TextMate coverage but is async + WASM-heavy; tree-sitter has active bash / fish / tmux / zsh grammars, but would require building our own highlight-query + theme-mapping pipeline.
41- Fix: hide highlighter behind an internal adapter, add fixture-based comparison tests for `bash`, `shellsession`, `zsh`, `fish`, `tmux`, and `ansi`, then prototype Shiki first and only switch after measuring startup cost, memory, and token quality against lowlight on those fixtures.
42-
4335 ## Toolchain
4436
4537 - 🔧 **Biome formatter clashes with Prettier on generic type arguments** — When a chained `.method<TypeArgs>(longString)` call exceeds `lineWidth`, biome's formatter breaks at the function args `(` while prettier breaks at the type args `<`. They never converge. Current config has biome formatter enabled (`indentStyle: "space"`), which means both formatters fight on this pattern. Fix: either disable biome's formatter (set `formatter.enabled: false` — `organizeImports` still works as an err …
@@ -50,10 +42,5 @@ Remaining work, known bugs, and planned improvements.
5042
5143 - 💡 **Scrollbar styling** — Scrollbar characters (`┃`/`│` vertical, `━`/`─` horizontal) and colors are hardcoded in `paintScrollbar`. A `scrollbarStyle` prop on containers (or a global theme option) would allow customizing thumb/track characters and colors. Currently the only two visual elements the framework doesn't support styling.
5244
53-- 💡 **Textarea component** (`packages/components`) — A higher-level wrapper around TextInput that handles the autogrowing pattern out of the box. Props like `maxLines`, `minLines`, and `placeholder` would cover the common chat-input / form-field use case without requiring the `HStack + flex + maxHeight` boilerplate. Could also include optional line-count display and character limit.
54-
55-- 💡 **Editor component** (`packages/components`) — A TextInput with a line-number gutter, built as a component. Would compose an HStack with a fixed-width line-number column (styled, right-aligned) alongside a flex TextInput, keeping scroll synchronized. Useful for code/config editing use cases like the markdown editor example.
56-
57-- 💡 Additional example apps (text editor from spec reference example)
5845 - 💡 `overflow: "hidden"` as explicit prop (currently all containers clip by default, which matches the spec's default behavior, but the prop value is not checked)
5946 - 💡 Higher Kitty protocol levels — key-release events (level 2), associated text (level 3), and full event types (level 4) for advanced input patterns (games, physical key layout awareness)
bun.lock+3−9
@@ -21,7 +21,7 @@
2121 "dependencies": {
2222 "@cel-tui/core": "workspace:*",
2323 "@cel-tui/types": "workspace:*",
24- "lowlight": "^3.3.0",
24+ "lextide": "^0.1.0",
2525 "yoctomarkdown": "^0.1.0",
2626 },
2727 "peerDependencies": {
@@ -102,16 +102,10 @@
102102
103103 "bun-types": ["bun-types@1.3.12", "", { "dependencies": { "@types/node": "*" } }, "sha512-HqOLj5PoFajAQciOMRiIZGNoKxDJSr6qigAttOX40vJuSp6DN/CxWp9s3C1Xwm4oH7ybueITwiaOcWXoYVoRkA=="],
104104
105- "dequal": ["dequal@2.0.3", "", {}, "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA=="],
106-
107- "devlop": ["devlop@1.1.0", "", { "dependencies": { "dequal": "^2.0.0" } }, "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA=="],
108-
109105 "entities": ["entities@4.5.0", "", {}, "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw=="],
110106
111107 "get-east-asian-width": ["get-east-asian-width@1.5.0", "", {}, "sha512-CQ+bEO+Tva/qlmw24dCejulK5pMzVnUOFOijVogd3KQs07HnRIgp8TGipvCCRT06xeYEbpbgwaCxglFyiuIcmA=="],
112108
113- "highlight.js": ["highlight.js@11.11.1", "", {}, "sha512-Xwwo44whKBVCYoliBQwaPvtd/2tYFkRQtXDWj1nackaV2JPXx3L0+Jvd8/qCJ2p+ML0/XVkJ2q+Mr+UVdpJK5w=="],
114-
115109 "lefthook": ["lefthook@2.1.5", "", { "optionalDependencies": { "lefthook-darwin-arm64": "2.1.5", "lefthook-darwin-x64": "2.1.5", "lefthook-freebsd-arm64": "2.1.5", "lefthook-freebsd-x64": "2.1.5", "lefthook-linux-arm64": "2.1.5", "lefthook-linux-x64": "2.1.5", "lefthook-openbsd-arm64": "2.1.5", "lefthook-openbsd-x64": "2.1.5", "lefthook-windows-arm64": "2.1.5", "lefthook-windows-x64": "2.1.5" }, "bin": { "lefthook": "bin/index.js" } }, "sha512-yB9IFWurFllusbPZqvG0EavTmpNXPya2MuO7Li7YT78xAj3u …
116110
117111 "lefthook-darwin-arm64": ["lefthook-darwin-arm64@2.1.5", "", { "os": "darwin", "cpu": "arm64" }, "sha512-VITTaw8PxxyE26gkZ8UcwIa5ZrWnKNRGLeeSrqri40cQdXvLTEoMq2tjjw7eiL9UcB0waRReDdzydevy9GOPUQ=="],
@@ -134,9 +128,9 @@
134128
135129 "lefthook-windows-x64": ["lefthook-windows-x64@2.1.5", "", { "os": "win32", "cpu": "x64" }, "sha512-yiAh8qxml6uqy10jDxOdN9fOQpyLxBFY1fgCEAhn7sVJYmJKRhjqSBwZX6LG5MQjzr29KStrIdw7TR3lf3rT7Q=="],
136130
137- "linkify-it": ["linkify-it@5.0.0", "", { "dependencies": { "uc.micro": "^2.0.0" } }, "sha512-5aHCbzQRADcdP+ATqnDuhhJ/MRIqDkZX5pyjFHRRysS8vZ5AbqGEoFIb6pYHPZ+L/OC2Lc+xT8uHVVR5CAK/wQ=="],
131+ "lextide": ["lextide@0.1.0", "", {}, "sha512-i1fx9SzultDOIgVtFt/vBOeyFaGYCNLT6OcdqOLRfWL9p6aTQsq8JQ+8LpJ4T7znWNjfjLqUU6dK5g2kXhSq3A=="],
138132
139- "lowlight": ["lowlight@3.3.0", "", { "dependencies": { "@types/hast": "^3.0.0", "devlop": "^1.0.0", "highlight.js": "~11.11.0" } }, "sha512-0JNhgFoPvP6U6lE/UdVsSq99tn6DhjjpAj5MxG49ewd2mOBVtwWYIT8ClyABhq198aXXODMU6Ox8DrGy/CpTZQ=="],
133+ "linkify-it": ["linkify-it@5.0.0", "", { "dependencies": { "uc.micro": "^2.0.0" } }, "sha512-5aHCbzQRADcdP+ATqnDuhhJ/MRIqDkZX5pyjFHRRysS8vZ5AbqGEoFIb6pYHPZ+L/OC2Lc+xT8uHVVR5CAK/wQ=="],
140134
141135 "lunr": ["lunr@2.3.9", "", {}, "sha512-zTU3DaZaF3Rt9rhN3uBMGQD3dD2/vFQqnvZCDv4dl5iOzq2IZQqTxu90r4E5J+nP70J3ilqVCrbho2eWaeW8Ow=="],
142136
docs/skill/cel-tui/SKILL.md+2−2
@@ -258,7 +258,7 @@ VStack({ flex: 1, overflow: "scroll", padding: { x: 1 } }, [
258258 SyntaxHighlight(source, "javascript", { theme: "dark-plus" });
259259 ```
260260
261-`SyntaxHighlight(content, language, props?)` renders registered lowlight/highlight.js languages and aliases into cel-tui primitives. Highlighting is synchronous. Unknown language ids render plain text. The optional `theme` accepts the small built-in presets (`"default"`, `"dark-plus"`) or a best-effort token-color registration object.
261+`SyntaxHighlight(content, language, props?)` renders registered lextide languages and aliases into cel-tui primitives. The component stays synchronous to call, but append-only updates reuse lextide's streaming parser state. Unknown language ids render plain text. The optional `theme` accepts the small built-in presets (`"default"`, `"dark-plus"`) or a best-effort token-color registration object.
262262
263263 ## Gotchas
264264
@@ -278,7 +278,7 @@ SyntaxHighlight(source, "javascript", { theme: "dark-plus" });
278278 - **Kitty-first keyboard input** — the framework enables Kitty level 1 and gets full modifier fidelity when the host preserves it (`alt+x`, `ctrl+plus`, `shift+enter`, etc.). It also normalizes recoverable legacy encodings so common shortcuts keep working in tmux. For best results, use a Kitty-compatible terminal or `tmux` with `set -s extended-keys on`. On older legacy hosts, historically ambiguous collisions such as `ctrl+i` vs `tab` or `ctrl+m` vs `enter` cannot be recovered once the host c …
279279 - **tmux is good for keyboard-driven manual checks** — common `tmux send-keys` paths work for printable chars, `Tab`/`BTab`, `Enter`, `Escape`, arrows, and many `Ctrl+letter` shortcuts. Use exact raw-sequence injection only when you need to target a protocol-specific encoding. Mouse input remains unreliable in tmux and should be verified in a real terminal.
280280 - **Button limitations** — `Button` from `@cel-tui/components` does not forward container sizing props (`width`, `height`, `flex`, `minWidth`, etc.). It supports styling (`fgColor`, `bgColor`, `bold`, etc.), `focusStyle`, `focused`, `onFocus`, `onBlur`, `onKeyPress`, and `padding`. For full layout control, use `HStack` + `Text` directly.
281-- **SyntaxHighlight is append-optimized, not parser-state perfect** — append-only updates re-highlight a suffix window near the end instead of the whole block. Non-append edits still fall back to a full re-highlight. Unknown language ids render plain text.
281+- **SyntaxHighlight keeps streaming parser state only for append-only growth** — appended content reuses lextide's stream state, but non-append edits still reset and replay the full snippet. Unknown language ids render plain text.
282282
283283 ## Composing Components
284284
docs/skill/cel-tui/references/api.md+3−3
@@ -293,11 +293,11 @@ SyntaxHighlight(code, "javascript", { theme: "dark-plus" });
293293 ```
294294
295295 - Signature: `SyntaxHighlight(content, language, props?)`
296-- `language` accepts registered lowlight/highlight.js language ids and aliases
296+- `language` accepts registered lextide language ids and aliases
297297 - `props.theme` accepts the built-in presets (`"default"`, `"dark-plus"`) or a best-effort token theme registration object
298298 - Uses a terminal-friendly ANSI 16 fallback theme by default
299-- Highlighting is synchronous; unsupported languages render plain text
300-- Append-only updates reuse a small per-language cache by re-highlighting only a suffix window near the end of the block
299+- Highlighting is synchronous at the component boundary; unsupported languages render plain text
300+- Append-only updates reuse lextide stream state, while non-append edits reset and replay the full snippet
301301
302302 ## Theme
303303
examples/syntax-highlight.ts+11−7
@@ -2,7 +2,7 @@
22 * cel-tui: SyntaxHighlight demo
33 *
44 * Streams TypeScript and Bash code into SyntaxHighlight.
5- * Shows synchronous lowlight highlighting plus append-only incremental updates.
5+ * Shows lextide-backed highlighting with append-only incremental updates.
66 *
77 * Run: bun run examples/syntax-highlight.ts
88 */
@@ -117,19 +117,23 @@ const BASH_CODE = [
117117 "#!/usr/bin/env bash",
118118 "set -euo pipefail",
119119 "",
120- 'project_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)',
120+ "readonly script_path=${BASH_SOURCE[0]}",
121+ "script_dir=${script_path%/*}",
122+ "targets=(debug release)",
123+ "count=0",
121124 "",
122125 "build() {",
123126 " local target=${1:-debug}",
124- ' echo "==> building ${target} from ${project_root}"',
127+ ' printf \'==> building %s from %s\\n\' "$target" "$script_dir"',
128+ " ((count += 1))",
125129 " bun run typecheck",
126130 " bun test",
127131 "}",
128132 "",
129- "for target in debug release; do",
130- ' if [[ "$target" == "release" ]]; then',
131- ' build "$target"',
132- " fi",
133+ 'for target in "${targets[@]}"; do',
134+ ' case "$target" in',
135+ ' release) build "$target" ;;',
136+ " esac",
133137 "done",
134138 ].join("\n");
135139
packages/components/package.json+2−2
@@ -32,9 +32,9 @@
3232 "typescript"
3333 ],
3434 "dependencies": {
35- "@cel-tui/types": "workspace:*",
3635 "@cel-tui/core": "workspace:*",
37- "lowlight": "^3.3.0",
36+ "@cel-tui/types": "workspace:*",
37+ "lextide": "^0.1.0",
3838 "yoctomarkdown": "^0.1.0"
3939 },
4040 "peerDependencies": {
packages/components/src/syntax-highlight.test.ts+8−0
@@ -96,6 +96,14 @@ describe("SyntaxHighlight", () => {
9696 expect(firstToken.props.fgColor).toBeDefined();
9797 });
9898
99+ test("supports shell alias highlighting", () => {
100+ const node = render('if true; then echo "$HOME"; fi', "shell");
101+
102+ expect(findText(node, "if").props.fgColor).toBeDefined();
103+ expect(findText(node, "then").props.fgColor).toBeDefined();
104+ expect(findText(node, "fi").props.fgColor).toBeDefined();
105+ });
106+
99107 test("wraps plain fallback content by default", () => {
100108 const node = render(
101109 "alpha beta gamma delta epsilon",
packages/components/src/syntax-highlight.ts+56−74
@@ -1,11 +1,17 @@
11 import type { Color, ContainerNode, Node, StyleProps } from "@cel-tui/types";
22 import { HStack, Text, VStack } from "@cel-tui/core";
3-import { all, createLowlight } from "lowlight";
3+import {
4+ all,
5+ createLowlight,
6+ type Element,
7+ type ElementContent,
8+ type HighlightStream,
9+ type StreamUpdate,
10+} from "lextide";
411
512 const lowlight = createLowlight(all);
613
714 const DEFAULT_THEME_NAME = "cel-ansi16";
8-const APPEND_REHIGHLIGHT_WINDOW = 8_192;
915 const MAX_STATES_PER_KEY = 4;
1016
1117 const ANSI_SLOT_HEX: ReadonlyArray<readonly [Color, string]> = [
@@ -88,7 +94,7 @@ export interface SyntaxHighlightProps {
8894 *
8995 * Built-in presets currently include `"default"` and `"dark-plus"`.
9096 * Theme registration objects are applied as best-effort overrides onto the
91- * lowlight/highlight.js token categories used by this component.
97+ * lextide token categories used by this component.
9298 */
9399 theme?: SyntaxHighlightTheme;
94100 }
@@ -110,26 +116,15 @@ interface ResolvedSyntaxHighlightTheme {
110116 }
111117
112118 interface SyntaxHighlightState {
119+ children: HighlightNode[];
113120 lastContent: string;
114121 lastNode: ContainerNode | null;
115- lines: HighlightRun[][];
122+ stream: HighlightStream;
116123 theme: ResolvedSyntaxHighlightTheme;
117124 }
118125
119-interface HighlightTextNode {
120- type: "text";
121- value: string;
122-}
123-
124-interface HighlightElementNode {
125- type: "element";
126- children: HighlightNode[];
127- properties?: {
128- className?: string | readonly string[];
129- };
130-}
131-
132-type HighlightNode = HighlightElementNode | HighlightTextNode;
126+type HighlightElementNode = Element;
127+type HighlightNode = ElementContent;
133128
134129 const rgbCache = new Map<string, readonly [number, number, number]>();
135130 const colorSlotCache = new Map<string, Color>();
@@ -518,34 +513,43 @@ function findState(
518513 return best;
519514 }
520515
516+function createStream(language: string): HighlightStream {
517+ return lowlight.stream(language, { allowRecalls: true });
518+}
519+
521520 function createState(
521+ language: string,
522522 theme: ResolvedSyntaxHighlightTheme,
523523 ): SyntaxHighlightState {
524524 return {
525+ children: [],
525526 lastContent: "",
526527 lastNode: null,
527- lines: [[]],
528+ stream: createStream(language),
528529 theme,
529530 };
530531 }
531532
532-function countNewlines(input: string): number {
533- let count = 0;
534- for (let i = 0; i < input.length; i++) {
535- if (input.charCodeAt(i) === 10) {
536- count++;
537- }
538- }
539- return count;
533+function applyStreamUpdate(
534+ children: HighlightNode[],
535+ update: StreamUpdate,
536+): void {
537+ const nextLength = Math.max(0, children.length - update.recall);
538+ children.length = nextLength;
539+ children.push(...update.stable, ...update.unstable);
540540 }
541541
542-function findAppendWindowStart(
542+function resetState(
543+ state: SyntaxHighlightState,
544+ language: string,
543545 content: string,
544- previousLength: number,
545-): number {
546- const target = Math.max(0, previousLength - APPEND_REHIGHLIGHT_WINDOW);
547- const newline = content.lastIndexOf("\n", target);
548- return newline === -1 ? 0 : newline + 1;
546+): void {
547+ state.children = [];
548+ state.stream = createStream(language);
549+
550+ if (content.length > 0) {
551+ applyStreamUpdate(state.children, state.stream.write(content));
552+ }
549553 }
550554
551555 function pushRun(
@@ -567,11 +571,7 @@ function pushRun(
567571 }
568572
569573 function getClassNames(node: HighlightElementNode): readonly string[] {
570- const className = node.properties?.className;
571- if (Array.isArray(className)) {
572- return className;
573- }
574- return typeof className === "string" ? [className] : [];
574+ return node.properties.className;
575575 }
576576
577577 function resolveNodeStyle(
@@ -630,21 +630,13 @@ function pushHighlightNode(
630630 }
631631 }
632632
633-function highlightContentToLines(
634- content: string,
635- language: string,
633+function highlightChildrenToLines(
634+ children: readonly HighlightNode[],
636635 theme: ResolvedSyntaxHighlightTheme,
637636 ): HighlightRun[][] {
638- if (content.length === 0) {
639- return [[]];
640- }
641-
642- const root = lowlight.highlight(language, content) as {
643- children: HighlightNode[];
644- };
645637 const lines: HighlightRun[][] = [[]];
646638
647- for (const child of root.children) {
639+ for (const child of children) {
648640 pushHighlightNode(lines, child, theme.baseStyle, theme);
649641 }
650642
@@ -661,25 +653,14 @@ function updateState(
661653 }
662654
663655 if (!content.startsWith(state.lastContent)) {
664- state.lines = highlightContentToLines(content, language, state.theme);
665- state.lastContent = content;
666- state.lastNode = null;
667- return;
656+ resetState(state, language, content);
657+ } else {
658+ const chunk = content.slice(state.lastContent.length);
659+ if (chunk.length > 0) {
660+ applyStreamUpdate(state.children, state.stream.write(chunk));
661+ }
668662 }
669663
670- const windowStart = findAppendWindowStart(content, state.lastContent.length);
671- const prefixLineCount = Math.min(
672- countNewlines(content.slice(0, windowStart)),
673- state.lines.length,
674- );
675- const suffixLines = highlightContentToLines(
676- content.slice(windowStart),
677- language,
678- state.theme,
679- );
680-
681- state.lines.length = prefixLineCount;
682- state.lines.push(...suffixLines);
683664 state.lastContent = content;
684665 state.lastNode = null;
685666 }
@@ -717,24 +698,25 @@ function renderHighlightedState(state: SyntaxHighlightState): ContainerNode {
717698 return state.lastNode;
718699 }
719700
720- state.lastNode = VStack({}, state.lines.map(lineToNode));
701+ const lines = highlightChildrenToLines(state.children, state.theme);
702+ state.lastNode = VStack({}, lines.map(lineToNode));
721703 return state.lastNode;
722704 }
723705
724706 /**
725707 * Render syntax-highlighted code as cel-tui primitives.
726708 *
727- * Uses lowlight/highlight.js synchronously with a small append-only cache. When
728- * content grows by appending new text, only a suffix window near the end is
729- * re-highlighted and merged back into the cached lines. Non-append edits fall
730- * back to a full re-highlight of the snippet.
709+ * Uses lextide synchronously at the component boundary while keeping a
710+ * streaming parser state per language/theme cache entry. Append-only updates
711+ * apply lextide's recall/stable/unstable deltas, while non-append edits reset
712+ * the stream and replay the full snippet.
731713 *
732714 * Unknown languages render as plain text. The default theme is terminal-friendly
733- * and maps lowlight token classes onto cel's ANSI palette slots while leaving
715+ * and maps lextide token classes onto cel's ANSI palette slots while leaving
734716 * base text on terminal defaults.
735717 *
736718 * @param content - Source code to render.
737- * @param language - Registered highlight.js language id or alias.
719+ * @param language - Registered lextide language id or alias.
738720 * @param props - Optional theme override.
739721 * @returns A `VStack` containing one highlighted line per child.
740722 */
@@ -750,7 +732,7 @@ export function SyntaxHighlight(
750732 }
751733
752734 const key = stateKey(language, theme);
753- const state = findState(key, content) ?? createState(theme);
735+ const state = findState(key, content) ?? createState(language, theme);
754736 updateState(state, language, content);
755737 touchState(key, state);
756738 return renderHighlightedState(state);