@@ -0,0 +1,363 @@
1+# `llama.vim` implementation reference
2+
3+This document records the design of [`ggml-org/llama.vim`](https://github.com/ggml-org/llama.vim) for future roadmap work on `vim-ai-complete`. It describes the competitor; it is not a proposed design for this plugin.
4+
5+Source reviewed: commit [`77db2afe488a7f700a2027527f17ab771988e358`](https://github.com/ggml-org/llama.vim/tree/77db2afe488a7f700a2027527f17ab771988e358), dated June 5, 2026.
6+
7+## Origin and scope
8+
9+`llama.vim` began as [`llama.cpp/examples/llama.vim`](https://github.com/ggml-org/llama.cpp/blob/master/examples/llama.vim). Its first standalone commit is named `llama.vim : init from llama.cpp`.
10+
11+It now provides two related features:
12+
13+1. Fill-in-the-middle (FIM) completion while typing.
14+2. Instruction-based replacement of a selected line range.
15+
16+The second feature is relevant to the possible conversational direction of `vim-ai-complete`.
17+
18+The plugin does not manage a model or server. The user runs one or more `llama-server` instances. `llama.vim` sends HTTP requests to configurable endpoints using asynchronous `curl` jobs. The defaults are:
19+
20+```text
21+FIM: http://127.0.0.1:8012/infill
22+Instruction: http://127.0.0.1:8012/v1/chat/completions
23+```
24+
25+The instruction endpoint uses the OpenAI-compatible chat-completions protocol. Models and a Bearer API key can be configured, so the transport is not inherently limited to a local server, although the plugin is designed and documented around local inference.
26+
27+## Repository structure
28+
29+The implementation is small in file count but concentrated in one large module:
30+
31+```text
32+plugin/llama.vim Runtime entrypoint; calls llama#init()
33+autoload/llama.vim Configuration, FIM, instruction edits, commands, and state
34+autoload/llama_debug.vim In-memory debug log and scratch-buffer viewer
35+doc/llama.txt Vim help
36+README.md Setup and usage
37+```
38+
39+At the reviewed commit, `autoload/llama.vim` is about 2,100 lines. There are no automated tests or CI workflows.
40+
41+## Startup and configuration
42+
43+Loading `plugin/llama.vim` calls `llama#init()`. Initialization:
44+
45+1. Verifies that `curl` is executable.
46+2. Merges `g:llama_config` with defaults.
47+3. Registers commands.
48+4. Initializes FIM caches, context queues, timers, and instruction-request state.
49+5. Detects Neovim extmarks or classic Vim text properties.
50+6. Enables the plugin unless `enable_at_startup` is false.
51+
52+Commands include:
53+
54+```text
55+:LlamaEnable
56+:LlamaDisable
57+:LlamaToggle
58+:LlamaToggleAutoFim
59+:LlamaStatus
60+:LlamaInstruct
61+:LlamaDebugToggle
62+:LlamaDebugClear
63+```
64+
65+The plugin defines configurable mappings for FIM and instruction operations. Instruction editing defaults to:
66+
67+```text
68+Visual selection + <leader>lli Start an instructed edit
69+<leader>llr Rerun the current instruction
70+<leader>llc Continue/refine the current edit
71+Tab Accept a ready result
72+Escape Cancel the edit
73+```
74+
75+## Overall data flow
76+
77+```text
78+ ┌───────────────────────┐
79+Editor events ──────────▶│ FIM context and cache │──────┐
80+ └───────────────────────┘ │
81+ ▼
82+Visual selection ───────▶ instruction request ───▶ curl job ───▶ llama-server
83+ │ ▲ │
84+ │ │ ▼
85+ └── extmark target └── message history ◀── streamed response
86+ │ │
87+ └──────── virtual status/preview ◀─────┘
88+ │
89+ accept / cancel / rerun /
90+ continue
91+```
92+
93+FIM and instruction editing share configuration and extra context, but maintain separate request state and endpoints.
94+
95+## FIM completion internals
96+
97+### Local context
98+
99+For a completion, the plugin constructs:
100+
101+- `input_prefix`: lines before the cursor.
102+- `prompt`: text on the current line before the cursor.
103+- `input_suffix`: text after the cursor and following lines.
104+- `input_extra`: chunks accumulated by the ring-context system.
105+- Indentation and sampling parameters.
106+- Prompt and generation time limits.
107+
108+The default local context is 256 lines before and 64 lines after the cursor.
109+
110+### Ring context
111+
112+The plugin gradually collects additional chunks from:
113+
114+- Yanked text.
115+- The current area on buffer entry and exit.
116+- The current area after writing a file.
117+- Areas near completion requests after substantial cursor movement.
118+
119+Chunks enter a queue and are later moved into a fixed-size ring. Similar chunks are rejected or evicted using token-overlap similarity. The queue is processed while Neovim is in Normal mode or the cursor has been idle long enough.
120+
121+When a chunk enters the ring, the plugin sends a zero-generation request containing the current extra context. `cache_prompt` lets `llama-server` prepare that context before a real completion is needed. This is a latency optimization built around llama.cpp's server-side prompt cache.
122+
123+### Completion cache
124+
125+Responses are stored in an LRU cache keyed by a SHA-256 hash of the local prefix, cursor-line prefix, and suffix. Each key can retain multiple completions.
126+
127+When the cursor moves, the plugin first looks for an exact cached completion. It then searches nearby cache keys to determine whether newly typed text matches the beginning of a previous completion. If so, it displays the untyped remainder immediately.
128+
129+This separates generation from rendering:
130+
131+```text
132+request finishes → response enters cache
133+cursor moves → cache lookup → render matching remainder
134+```
135+
136+A first FIM request uses a short generation budget. Once its result is displayed, a speculative follow-up request treats the first result as already inserted and asks for a continuation. This makes longer completion chains available without delaying the first visible result.
137+
138+### FIM presentation
139+
140+Neovim uses extmarks with `virt_text` and `virt_lines`; classic Vim uses text properties. The plugin can show:
141+
142+- Inline completion text.
143+- Additional generated lines.
144+- Cache and ring-buffer sizes.
145+- Prompt and generation timings.
146+- The selected completion number when alternatives exist.
147+
148+The user can accept the full result, one line, or one word, or cycle through cached alternatives.
149+
150+## Instruction-edit lifecycle
151+
152+Instruction editing is an in-memory state machine associated with a selected line range.
153+
154+```text
155+prompting → processing → generating → ready
156+ ▲ │
157+ └──────── rerun ──────────┤
158+ ├── continue → generating
159+ ├── accept
160+ └── cancel
161+```
162+
163+### 1. Selection and warm-up
164+
165+`:LlamaInstruct` receives a line range, normally from Visual mode. Before asking for the instruction, the plugin builds the model context and sends a zero-token warm-up request.
166+
167+It then reads the instruction with:
168+
169+```vim
170+input('Instruction: ')
171+```
172+
173+The warm-up request allows a local server to process and cache the selected code while the user types the instruction.
174+
175+### 2. Prompt construction
176+
177+For a new edit, the message history starts with a system message containing:
178+
179+- A text-editing instruction that requires replacement text only.
180+- Ring-buffer context.
181+- Lines before the selection.
182+- The selected lines.
183+- Lines after the selection.
184+
185+The user's instruction is appended as a `user` message.
186+
187+Conceptually:
188+
189+```text
190+system:
191+ editing rules
192+ CONTEXT
193+ PREFIX
194+ SELECTION
195+ SUFFIX
196+
197+user:
198+ requested change
199+```
200+
201+This is similar to the current `vim-ai-complete` prompt, but the structured messages remain available for later turns.
202+
203+### 3. Per-edit request state
204+
205+Every edit receives a numeric request ID and an entry in `s:inst_reqs`. Each entry contains approximately:
206+
207+```text
208+id Request/session identifier
209+bufnr Target buffer
210+range Current target line range
211+status proc, gen, or ready
212+inst Current instruction
213+inst_prev Conversation message history
214+result Accumulated replacement text
215+job Active curl job
216+n_gen Streaming progress counter
217+extmark Highlight anchor for the source range
218+extmark_virt Virtual preview/status anchor
219+```
220+
221+Because requests are stored by ID, multiple edits can exist at the same time. Commands such as accept, cancel, rerun, and continue choose an edit by finding a request whose range contains the cursor.
222+
223+### 4. Target highlighting
224+
225+The selected range is highlighted immediately. In Neovim, an extmark anchors the range as surrounding lines move. A second extmark displays status and preview lines below the target.
226+
227+The target buffer itself remains unchanged during generation.
228+
229+### 5. Streaming generation
230+
231+The request is sent to `/v1/chat/completions` with streaming enabled. `curl` output is parsed as server-sent JSON lines. Assistant deltas are appended to the request's `result`.
232+
233+The display progresses through three states:
234+
235+- `proc`: endpoint, model, instruction, and “Processing”.
236+- `gen`: a generation counter and the latest generated line.
237+- `ready`: the complete proposed replacement as virtual lines.
238+
239+The status is co-located with the selected code rather than shown in a global modal or notification.
240+
241+### 6. Accept and cancel
242+
243+Accepting a ready request:
244+
245+1. Locates the active request under the cursor.
246+2. Removes its highlight and preview extmarks.
247+3. Deletes the currently tracked line range.
248+4. Inserts the generated result.
249+5. Removes the request from memory.
250+
251+Canceling stops the job, removes its UI, and drops the request without changing the buffer.
252+
253+### 7. Rerun
254+
255+After a result is ready, rerun:
256+
257+1. Clears the current result.
258+2. Removes the previous assistant response from message history.
259+3. Sends the same context and instruction again.
260+
261+This produces an alternative answer without adding another conversational turn.
262+
263+### 8. Continue/refine
264+
265+Continue prompts with:
266+
267+```vim
268+input('Next instruction: ')
269+```
270+
271+It retains the earlier messages, appends the new user instruction, and generates another assistant response. After each completed request, the assistant result is added to history.
272+
273+The conversation therefore has this form:
274+
275+```text
276+system Original selection and surrounding context
277+user Initial instruction
278+assistant First proposed replacement
279+user Refinement
280+assistant Refined replacement
281+...
282+```
283+
284+Only the latest assistant response is offered as the replacement. The conversation exists only while the request remains active and is not persisted across Neovim sessions.
285+
286+## Range tracking and concurrent edits
287+
288+The implementation uses an extmark to follow the start of the selected range when text moves. It does not perform conflict reconciliation.
289+
290+Notably, it does not:
291+
292+- Store and validate the original selected text before applying.
293+- Reject application based on `changedtick`.
294+- Detect edits made inside the highlighted target.
295+- Rebase the generated result onto newer text.
296+- Merge concurrent user and model changes.
297+
298+The tracked end is updated by preserving the original range's line count relative to the extmark's current start. If the user changes the target's shape while generation is active, accepting can replace newer text with a response generated from older text.
299+
300+The practical interaction rule is that the highlighted range is reserved until the user accepts or cancels. This is visible but not enforced.
301+
302+## Debugging and status
303+
304+`LlamaStatus` queries `/v1/models` and reports whether the configured FIM and instruction models are available.
305+
306+The debug module keeps an in-memory bounded log and renders it in a scratch split. It records request and completion information without introducing a file or logging dependency.
307+
308+## Strengths relevant to future roadmap work
309+
310+- A conversation is scoped to one requested edit.
311+- The edit has explicit state independent of the target buffer's contents.
312+- The original text remains unchanged until acceptance.
313+- Selection highlighting makes the active target visible.
314+- Progress and the generated result are displayed next to the target.
315+- Retry and refinement have distinct semantics.
316+- Refinement reuses structured message history.
317+- Multiple edit requests can be represented independently.
318+- Transport and model endpoints are configurable.
319+- The model server and plugin remain separate processes.
320+
321+## Limitations and open questions
322+
323+- It is a conversational text transformer, not a tool-using agent.
324+- Prompt input has the same long-command-line limitation currently tracked for `vim-ai-complete`.
325+- Conversation history is not displayed to the user.
326+- Sessions are neither named nor persisted.
327+- There is no way to return to an accepted or canceled conversation.
328+- Concurrent edits to the target are not made safe.
329+- Request and UI state are embedded in a large Vimscript module.
330+- Streaming parsing assumes convenient `curl` callback boundaries.
331+- Default global and buffer mappings can conflict with existing configuration.
332+- The source acknowledges problems removing mappings during disable.
333+- There are no automated tests for the state machine or editing behavior.
334+
335+## Comparison with current `vim-ai-complete`
336+
337+| Concern | `vim-ai-complete` now | `llama.vim` instruction edits |
338+|---|---|---|
339+| Model process | Configurable CLI, `pi` by default | Configurable HTTP endpoint, usually `llama-server` |
340+| Execution | Blocking, one process per request | Asynchronous `curl` job |
341+| Target | Previous Visual selection | Line range anchored by extmark |
342+| Result | Applied immediately on success | Previewed as virtual lines |
343+| Undo | Normal Neovim undo | Accept is required before mutation |
344+| Iteration | None | Rerun and continue |
345+| Conversation | None | Message list attached to edit request |
346+| Progress | Global notifications | Virtual status next to target |
347+| Concurrent target edits | Prevented by blocking | Allowed but not reconciled safely |
348+| Context | Prompt plus CLI access to files | Prefix, selection, suffix, and ring chunks |
349+
350+## Questions to revisit when defining a roadmap
351+
352+These are questions exposed by the reference implementation, not decisions:
353+
354+- What constitutes a session: one selection, one buffer, or a broader conversation?
355+- Should the configured CLI own session persistence, or should the plugin own message history?
356+- How is a session identified and resumed?
357+- Should a response be applied immediately, previewed, or configurable?
358+- How should the UI show earlier prompts and responses?
359+- What should happen if the target changes while a request is active?
360+- Can multiple sessions run concurrently, and how does the user select one?
361+- What cancellation and retry primitives can a configurable hosted-model CLI expose?
362+- Which context belongs in the plugin prompt, and which context should the external agent discover itself?
363+- Does preparing a hosted session while the user enters a prompt provide enough latency benefit to justify the lifecycle complexity?