xonecas.io
6119
My

mysis / files

mysis / documentation / architecture / LLM_LOOP.md

9.2 KB
1# LLM Processing Loop
2
3Single-turn conversation processing with multi-round tool call support.
4
5## Overview
6
7The LLM loop handles one conversation turn, which may involve multiple rounds of LLM calls and tool executions. A **turn** begins when the user sends a message and ends when the LLM provides a final text response without tool calls.
8
9**Key principle:** The loop accumulates history during the turn so the LLM sees its own tool calls and results across rounds.
10
11## Architecture
12
13```
14User Input → ProcessTurn → [Round 0, Round 1, ..., Round N] → Final Response
15                              ↓
16                          Compress History → LLM Call → Tool Execution
17                              ↑_______________|
18```
19
20## Flow Diagram
21
22```
23processTurn (cli.go:266-282)
24  │
25  ├─ Snapshot app.history → historyCopy
26  │
27  └─ ProcessTurn (llm/loop.go:32-116)
28      │
29      └─ Loop: round = 0 to MaxToolRounds (20)
30          │
31          ├─ Compress historyCopy → compressedHistory
32          │
33          ├─ LLM Call with compressedHistory + tools
34          │    └─ Returns: content, tool_calls, reasoning
35          │
36          ├─ If no tool_calls:
37          │    ├─ Display content
38          │    ├─ OnMessage(assistant_msg)
39          │    ├─ Append to historyCopy
40          │    └─ RETURN (turn complete)
41          │
42          └─ If tool_calls present:
43               ├─ OnMessage(assistant_msg_with_tools)
44               ├─ Append to historyCopy
45               │
46               ├─ executeToolCalls:
47               │    └─ For each tool:
48               │         ├─ Execute via MCP proxy
49               │         ├─ OnMessage(tool_result)
50               │         └─ Collect tool_result
51               │
52               ├─ Append all tool_results to historyCopy
53               └─ Continue to next round
54```
55
56## Implementation Details
57
58### Entry Point: `ProcessTurn`
59
60**Location:** `internal/llm/loop.go:32-116`
61
62**Signature:**
63
64```go
65func ProcessTurn(ctx context.Context, opts ProcessTurnOptions) error
66```
67
68**Options:**
69
70- `Provider`: LLM provider interface (Ollama, OpenCode Zen, etc.)
71- `Proxy`: MCP proxy for tool execution
72- `Tools`: Available tools for this turn
73- `History`: Initial conversation history snapshot
74- `OnMessage`: Callback to save messages to database and `app.history`
75- `MaxToolRounds`: Maximum iterations (default: 20)
76- `HistoryKeepLast`: How many recent turns to keep uncompressed (default: 10)
77
78### Round Loop
79
80Each round performs:
81
821. **History Compression** (line 43)
83   - Compresses old state query results
84   - Keeps recent turns full
85   - See `internal/store/compression.go` for details
86
872. **LLM Call** (line 69)
88   - Sends compressed history + available tools
89   - Receives response with optional tool calls
90
913. **Response Handling**
92   - **No tool calls** (lines 80-93): Display content, save message, update history, exit
93   - **Tool calls present** (lines 96-110): Save assistant message, execute tools, update history, continue
94
954. **History Update** (Critical Fix - Bug #1)
96   - After `OnMessage()` callback, message is also appended to `opts.History`
97   - This ensures next round sees messages from previous rounds
98   - Without this, LLM would repeat tool calls indefinitely
99
100### Tool Execution: `executeToolCalls`
101
102**Location:** `internal/llm/loop.go:130-195`
103
104**Signature:**
105
106```go
107func executeToolCalls(ctx context.Context, proxy *mcp.Proxy, toolCalls []provider.ToolCall, onMessage MessageCallback) ([]provider.Message, error)
108```
109
110**Flow:**
111
1121. For each tool call:
113   - Display tool name and arguments (truncated)
114   - Execute via MCP proxy
115   - Display result (success ✓ or error ✗)
116   - Create tool result message
117   - Call `onMessage()` to save to database
118   - Collect message for return
119
1202. Return all tool result messages
121   - Caller appends to `opts.History`
122   - Ensures next round sees tool results
123
124### Message Callback: `OnMessage`
125
126**Implementation:** `internal/cli/cli.go:285-293`
127
128```go
129func (app *App) addMessage(msg provider.Message) {
130    app.mu.Lock()
131    app.history = append(app.history, msg)
132    app.mu.Unlock()
133
134    if err := app.sessionMgr.SaveMessage(app.sessionID, msg); err != nil {
135        log.Warn().Err(err).Msg("Failed to save message to database")
136    }
137}
138```
139
140**Purpose:**
141
142- Updates canonical `app.history` with mutex protection
143- Saves message to SQLite database for session persistence
144- Called for every message: user, assistant, tool results
145
146**Note:** The callback updates `app.history` but NOT `opts.History`. The caller (`ProcessTurn`) must also update `opts.History` after calling the callback.
147
148## History Management
149
150### Snapshot vs Reference
151
152**Design Decision:** Pass history snapshot, not reference.
153
154**Rationale:**
155
156- Prevents concurrent modification during processing
157- Clean separation between "input to turn" and "output from turn"
158- Next turn gets fresh snapshot with all previous messages
159
160**Implementation:**
161
162```go
163// cli.go:268-271
164app.mu.Lock()
165historyCopy := make([]provider.Message, len(app.history))
166copy(historyCopy, app.history)
167app.mu.Unlock()
168```
169
170### History Accumulation During Turn
171
172**Critical Fix (2026-02-10):** After each message is saved via `OnMessage()`, it's also appended to `opts.History`.
173
174**Before Fix:**
175
176```go
177opts.OnMessage(assistantMsg)
178// opts.History unchanged - next round uses stale history
179```
180
181**After Fix:**
182
183```go
184opts.OnMessage(assistantMsg)
185opts.History = append(opts.History, assistantMsg)
186// Next round sees this message
187```
188
189**Why This Matters:**
190
191Round 0:
192
193- `opts.History = [user: "check status"]`
194- LLM calls `get_status`
195- Tool result saved via `OnMessage` to `app.history`
196- **Before fix:** `opts.History` still `[user: "check status"]`
197- **After fix:** `opts.History = [user: "check status", assistant: tool_call, tool: result]`
198
199Round 1:
200
201- **Before fix:** LLM sees compressed `[user: "check status"]` → calls `get_status` again (doesn't see result)
202- **After fix:** LLM sees compressed `[user: "check status", assistant: tool_call, tool: result]` → processes result
203
204## Compression During Turn
205
206**Key Insight:** Compression operates on `opts.History`, which now includes messages from previous rounds within the same turn.
207
208**Process:**
209
2101. `CompressHistory(opts.History, keepFullTurns=10)` compresses old state queries
2112. Recent turns (last 10) kept full
2123. Old action tool results truncated to 200 chars
2134. Authentication tools never compressed
214
215**See:** `internal/store/compression.go` for compression logic
216
217## Loop Termination
218
219**Success Exit:**
220
221- LLM returns response without tool calls (line 91)
222- Turn completes, user regains control
223
224**Error Exit:**
225
226- LLM call fails (line 70-72)
227- Tool execution fails (line 107-109)
228- Max rounds exceeded (line 115)
229
230**Max Rounds:** 20 iterations prevents infinite loops if LLM repeatedly calls same tool.
231
232## Message Roles
233
234### User Messages
235
236- **Origin:** User input or autoplay
237- **Added:** Before `processTurn()` call
238- **Location:** `cli.go:246` (manual), `autoplay.go:190` (autoplay)
239
240### Assistant Messages
241
242- **Origin:** LLM response
243- **Added:** During `ProcessTurn` round
244- **Variants:**
245  - Text only: `Role: "assistant", Content: "..."`
246  - With tool calls: `Role: "assistant", Content: "...", ToolCalls: [...]`
247
248### Tool Messages
249
250- **Origin:** Tool execution results
251- **Added:** During `executeToolCalls`
252- **Format:** `Role: "tool", Content: "result", ToolCallID: "..."`
253
254## Thread Safety
255
256**Mutex Protection:**
257
258- `app.mu` protects `app.history` (cli.go:34)
259- Lock acquired for snapshot creation (cli.go:268-271)
260- Lock acquired for message append via `addMessage()` (cli.go:286-288)
261- `opts.History` is local to `ProcessTurn`, no lock needed
262
263**Concurrency Scenarios:**
264
265- Manual input and autoplay both call `processTurn()` serially
266- Each turn operates on its own snapshot
267- Callbacks serialize access to `app.history` via mutex
268
269## Error Handling
270
271**LLM Call Failure:**
272
273- Returns error immediately
274- Caller displays error to user
275- Turn aborted
276
277**Tool Call Failure:**
278
279- Creates error tool result: `Content: "Error: ..."`
280- Continues to next tool in list
281- LLM sees error in next round, can recover
282
283**Max Rounds Exceeded:**
284
285- Returns error: `"too many tool call rounds (limit: 20)"`
286- Indicates LLM stuck in loop (bug or model issue)
287- User sees error, can try different prompt
288
289## Performance Characteristics
290
291**History Compression:**
292
293- Reduces token usage for long conversations
294- Trade-off: Old details lost vs context window limits
295- Compression runs every round (negligible overhead)
296
297**Round Complexity:**
298
299- Each round: 1 LLM call + N tool executions
300- Typical turn: 1-5 rounds
301- Worst case: 20 rounds (max limit)
302
303**Memory:**
304
305- `opts.History` grows during turn (O(messages))
306- Snapshot copied at turn start (O(messages))
307- Released at turn end
308
309## Configuration
310
311**MaxToolRounds:** 20
312
313- Prevents infinite loops
314- Increase if legitimate use cases need more rounds
315- Decrease to fail fast during debugging
316
317**HistoryKeepLast:** 10
318
319- Number of recent turns to keep uncompressed
320- Increase for more context (higher token usage)
321- Decrease for faster compression (less context)
322
323## Future Improvements
324
325**Streaming:** Support streaming LLM responses for faster perceived performance
326
327**History Window:** Smarter compression based on token count, not just turn count