70160
Ce
cel-tui / files
cel-tui / packages / clew / README.md
3.1 KB
1# @cel-tui/clew 2 3`clew` is a stream-first syntax tokenization library. 4 5Parsing input is like solving a maze: you only know the right path once you have seen enough of it. That is the core idea behind `clew`. 6 7Instead of pretending tokenization is always final on first sight, `clew` is designed to: 8 9- emit tokens early 10- correct earlier output when new input changes the parse 11- stay renderer-agnostic 12- work well for TUIs, editors, markdown renderers, and streaming output 13 14## Goals 15 16- fast 17- correct 18- stream-aware 19- renderer-agnostic 20- Bun-friendly 21 22## API direction 23 24`clew` keeps a single entrypoint: 25 26```ts 27clew(content, options) => ClewStream | ClewOutput 28``` 29 30### Defaults 31 32- `stream` defaults to `true` 33- `stability` defaults to `"eager"` 34 35That means `clew` is streaming-first by default. 36 37## Streaming mode 38 39```ts 40import { clew } from "@cel-tui/clew"; 41 42const stream = clew("", { lang: "ts" }); 43 44stream.onChunk((chunk) => { 45 console.log("chunk", chunk); 46}); 47 48stream.onCorrection((correction) => { 49 console.log("correction", correction); 50}); 51 52stream.write("c"); 53stream.write("o"); 54stream.write("n"); 55stream.write("st x = 1"); 56 57const final = stream.end(); 58console.log(final.tokens); 59``` 60 61### Why corrections exist 62 63Some tokenization decisions are only valid once more input arrives. 64 65For example, a partial token stream may need to be revised once a string closes, a comment continues, or a multi-line construct changes the meaning of earlier text. 66 67Because of that, `clew` uses **range-based corrections** instead of only append-only token events. 68 69A correction uses UTF-16 code-unit offsets, matching JavaScript string 70indexing, and says in effect: 71 72> replace the tokenization for source range `from..to` with these tokens 73 74That keeps the API general enough for: 75 76- per-character streaming 77- line streaming 78- editor integrations 79- TUI renderers 80- HTML renderers 81- diff tools 82 83## Sync mode 84 85If you already have the full content and do not want streaming behavior, turn it off explicitly: 86 87```ts 88import { clew } from "@cel-tui/clew"; 89 90const output = clew("const x = 1", { 91 lang: "ts", 92 stream: false, 93}); 94 95console.log(output.tokens); 96``` 97 98## Public surface 99 100```ts 101const stream = clew(content, { 102 lang: "ts", 103 stream: true, 104 stability: "eager", 105}); 106 107stream.onChunk((chunk) => {}); 108stream.onCorrection((correction) => {}); 109stream.write("more content"); 110stream.snapshot(); 111stream.end(); 112``` 113 114```ts 115const output = clew(content, { 116 lang: "ts", 117 stream: false, 118}); 119``` 120 121## Current implementation 122 123The current implementation ships a small internal language registry with: 124 125- **TypeScript / JavaScript ids** 126 - `typescript`, `ts`, `mts`, `cts` 127 - `javascript`, `js`, `mjs`, `cjs` 128- **Python** 129 - `python`, `py` 130- **Bash** 131 - `bash` 132- **JSON** 133 - `json` 134- **Markdown** 135 - `markdown` 136- **Diff / patch** 137 - `diff`, `patch` 138 139`clew` emits normalized tokens with canonical scopes like `keyword`, `string`, 140`property`, `link`, `markup.heading`, `meta.substitution.command`, and 141`string.heredoc`. 142 143## Notes 144 145The stream layer is shared across languages. Each language module only owns its 146own tokenization and stable-boundary rules, which is the structure new language 147support should follow.