xonecas.io
70160
Ce

cel-tui / files

cel-tui / packages / clew / README.md

3.1 KB
1# @cel-tui/clew
2
3`clew` is a stream-first syntax tokenization library.
4
5Parsing input is like solving a maze: you only know the right path once you have seen enough of it. That is the core idea behind `clew`.
6
7Instead of pretending tokenization is always final on first sight, `clew` is designed to:
8
9- emit tokens early
10- correct earlier output when new input changes the parse
11- stay renderer-agnostic
12- work well for TUIs, editors, markdown renderers, and streaming output
13
14## Goals
15
16- fast
17- correct
18- stream-aware
19- renderer-agnostic
20- Bun-friendly
21
22## API direction
23
24`clew` keeps a single entrypoint:
25
26```ts
27clew(content, options) => ClewStream | ClewOutput
28```
29
30### Defaults
31
32- `stream` defaults to `true`
33- `stability` defaults to `"eager"`
34
35That means `clew` is streaming-first by default.
36
37## Streaming mode
38
39```ts
40import { clew } from "@cel-tui/clew";
41
42const stream = clew("", { lang: "ts" });
43
44stream.onChunk((chunk) => {
45  console.log("chunk", chunk);
46});
47
48stream.onCorrection((correction) => {
49  console.log("correction", correction);
50});
51
52stream.write("c");
53stream.write("o");
54stream.write("n");
55stream.write("st x = 1");
56
57const final = stream.end();
58console.log(final.tokens);
59```
60
61### Why corrections exist
62
63Some tokenization decisions are only valid once more input arrives.
64
65For example, a partial token stream may need to be revised once a string closes, a comment continues, or a multi-line construct changes the meaning of earlier text.
66
67Because of that, `clew` uses **range-based corrections** instead of only append-only token events.
68
69A correction uses UTF-16 code-unit offsets, matching JavaScript string
70indexing, and says in effect:
71
72> replace the tokenization for source range `from..to` with these tokens
73
74That keeps the API general enough for:
75
76- per-character streaming
77- line streaming
78- editor integrations
79- TUI renderers
80- HTML renderers
81- diff tools
82
83## Sync mode
84
85If you already have the full content and do not want streaming behavior, turn it off explicitly:
86
87```ts
88import { clew } from "@cel-tui/clew";
89
90const output = clew("const x = 1", {
91  lang: "ts",
92  stream: false,
93});
94
95console.log(output.tokens);
96```
97
98## Public surface
99
100```ts
101const stream = clew(content, {
102  lang: "ts",
103  stream: true,
104  stability: "eager",
105});
106
107stream.onChunk((chunk) => {});
108stream.onCorrection((correction) => {});
109stream.write("more content");
110stream.snapshot();
111stream.end();
112```
113
114```ts
115const output = clew(content, {
116  lang: "ts",
117  stream: false,
118});
119```
120
121## Current implementation
122
123The current implementation ships a small internal language registry with:
124
125- **TypeScript / JavaScript ids**
126  - `typescript`, `ts`, `mts`, `cts`
127  - `javascript`, `js`, `mjs`, `cjs`
128- **Python**
129  - `python`, `py`
130- **Bash**
131  - `bash`
132- **JSON**
133  - `json`
134- **Markdown**
135  - `markdown`
136- **Diff / patch**
137  - `diff`, `patch`
138
139`clew` emits normalized tokens with canonical scopes like `keyword`, `string`,
140`property`, `link`, `markup.heading`, `meta.substitution.command`, and
141`string.heredoc`.
142
143## Notes
144
145The stream layer is shared across languages. Each language module only owns its
146own tokenization and stable-boundary rules, which is the structure new language
147support should follow.