6119
My
mysis / files
mysis / documentation / architecture / LLM_LOOP.md
9.2 KB
1# LLM Processing Loop 2 3Single-turn conversation processing with multi-round tool call support. 4 5## Overview 6 7The LLM loop handles one conversation turn, which may involve multiple rounds of LLM calls and tool executions. A **turn** begins when the user sends a message and ends when the LLM provides a final text response without tool calls. 8 9**Key principle:** The loop accumulates history during the turn so the LLM sees its own tool calls and results across rounds. 10 11## Architecture 12 13``` 14User Input → ProcessTurn → [Round 0, Round 1, ..., Round N] → Final Response 15 ↓ 16 Compress History → LLM Call → Tool Execution 17 ↑_______________| 18``` 19 20## Flow Diagram 21 22``` 23processTurn (cli.go:266-282) 24 │ 25 ├─ Snapshot app.history → historyCopy 26 │ 27 └─ ProcessTurn (llm/loop.go:32-116) 28 │ 29 └─ Loop: round = 0 to MaxToolRounds (20) 30 │ 31 ├─ Compress historyCopy → compressedHistory 32 │ 33 ├─ LLM Call with compressedHistory + tools 34 │ └─ Returns: content, tool_calls, reasoning 35 │ 36 ├─ If no tool_calls: 37 │ ├─ Display content 38 │ ├─ OnMessage(assistant_msg) 39 │ ├─ Append to historyCopy 40 │ └─ RETURN (turn complete) 41 │ 42 └─ If tool_calls present: 43 ├─ OnMessage(assistant_msg_with_tools) 44 ├─ Append to historyCopy 45 │ 46 ├─ executeToolCalls: 47 │ └─ For each tool: 48 │ ├─ Execute via MCP proxy 49 │ ├─ OnMessage(tool_result) 50 │ └─ Collect tool_result 51 │ 52 ├─ Append all tool_results to historyCopy 53 └─ Continue to next round 54``` 55 56## Implementation Details 57 58### Entry Point: `ProcessTurn` 59 60**Location:** `internal/llm/loop.go:32-116` 61 62**Signature:** 63 64```go 65func ProcessTurn(ctx context.Context, opts ProcessTurnOptions) error 66``` 67 68**Options:** 69 70- `Provider`: LLM provider interface (Ollama, OpenCode Zen, etc.) 71- `Proxy`: MCP proxy for tool execution 72- `Tools`: Available tools for this turn 73- `History`: Initial conversation history snapshot 74- `OnMessage`: Callback to save messages to database and `app.history` 75- `MaxToolRounds`: Maximum iterations (default: 20) 76- `HistoryKeepLast`: How many recent turns to keep uncompressed (default: 10) 77 78### Round Loop 79 80Each round performs: 81 821. **History Compression** (line 43) 83 - Compresses old state query results 84 - Keeps recent turns full 85 - See `internal/store/compression.go` for details 86 872. **LLM Call** (line 69) 88 - Sends compressed history + available tools 89 - Receives response with optional tool calls 90 913. **Response Handling** 92 - **No tool calls** (lines 80-93): Display content, save message, update history, exit 93 - **Tool calls present** (lines 96-110): Save assistant message, execute tools, update history, continue 94 954. **History Update** (Critical Fix - Bug #1) 96 - After `OnMessage()` callback, message is also appended to `opts.History` 97 - This ensures next round sees messages from previous rounds 98 - Without this, LLM would repeat tool calls indefinitely 99 100### Tool Execution: `executeToolCalls` 101 102**Location:** `internal/llm/loop.go:130-195` 103 104**Signature:** 105 106```go 107func executeToolCalls(ctx context.Context, proxy *mcp.Proxy, toolCalls []provider.ToolCall, onMessage MessageCallback) ([]provider.Message, error) 108``` 109 110**Flow:** 111 1121. For each tool call: 113 - Display tool name and arguments (truncated) 114 - Execute via MCP proxy 115 - Display result (success ✓ or error ✗) 116 - Create tool result message 117 - Call `onMessage()` to save to database 118 - Collect message for return 119 1202. Return all tool result messages 121 - Caller appends to `opts.History` 122 - Ensures next round sees tool results 123 124### Message Callback: `OnMessage` 125 126**Implementation:** `internal/cli/cli.go:285-293` 127 128```go 129func (app *App) addMessage(msg provider.Message) { 130 app.mu.Lock() 131 app.history = append(app.history, msg) 132 app.mu.Unlock() 133 134 if err := app.sessionMgr.SaveMessage(app.sessionID, msg); err != nil { 135 log.Warn().Err(err).Msg("Failed to save message to database") 136 } 137} 138``` 139 140**Purpose:** 141 142- Updates canonical `app.history` with mutex protection 143- Saves message to SQLite database for session persistence 144- Called for every message: user, assistant, tool results 145 146**Note:** The callback updates `app.history` but NOT `opts.History`. The caller (`ProcessTurn`) must also update `opts.History` after calling the callback. 147 148## History Management 149 150### Snapshot vs Reference 151 152**Design Decision:** Pass history snapshot, not reference. 153 154**Rationale:** 155 156- Prevents concurrent modification during processing 157- Clean separation between "input to turn" and "output from turn" 158- Next turn gets fresh snapshot with all previous messages 159 160**Implementation:** 161 162```go 163// cli.go:268-271 164app.mu.Lock() 165historyCopy := make([]provider.Message, len(app.history)) 166copy(historyCopy, app.history) 167app.mu.Unlock() 168``` 169 170### History Accumulation During Turn 171 172**Critical Fix (2026-02-10):** After each message is saved via `OnMessage()`, it's also appended to `opts.History`. 173 174**Before Fix:** 175 176```go 177opts.OnMessage(assistantMsg) 178// opts.History unchanged - next round uses stale history 179``` 180 181**After Fix:** 182 183```go 184opts.OnMessage(assistantMsg) 185opts.History = append(opts.History, assistantMsg) 186// Next round sees this message 187``` 188 189**Why This Matters:** 190 191Round 0: 192 193- `opts.History = [user: "check status"]` 194- LLM calls `get_status` 195- Tool result saved via `OnMessage` to `app.history` 196- **Before fix:** `opts.History` still `[user: "check status"]` 197- **After fix:** `opts.History = [user: "check status", assistant: tool_call, tool: result]` 198 199Round 1: 200 201- **Before fix:** LLM sees compressed `[user: "check status"]` → calls `get_status` again (doesn't see result) 202- **After fix:** LLM sees compressed `[user: "check status", assistant: tool_call, tool: result]` → processes result 203 204## Compression During Turn 205 206**Key Insight:** Compression operates on `opts.History`, which now includes messages from previous rounds within the same turn. 207 208**Process:** 209 2101. `CompressHistory(opts.History, keepFullTurns=10)` compresses old state queries 2112. Recent turns (last 10) kept full 2123. Old action tool results truncated to 200 chars 2134. Authentication tools never compressed 214 215**See:** `internal/store/compression.go` for compression logic 216 217## Loop Termination 218 219**Success Exit:** 220 221- LLM returns response without tool calls (line 91) 222- Turn completes, user regains control 223 224**Error Exit:** 225 226- LLM call fails (line 70-72) 227- Tool execution fails (line 107-109) 228- Max rounds exceeded (line 115) 229 230**Max Rounds:** 20 iterations prevents infinite loops if LLM repeatedly calls same tool. 231 232## Message Roles 233 234### User Messages 235 236- **Origin:** User input or autoplay 237- **Added:** Before `processTurn()` call 238- **Location:** `cli.go:246` (manual), `autoplay.go:190` (autoplay) 239 240### Assistant Messages 241 242- **Origin:** LLM response 243- **Added:** During `ProcessTurn` round 244- **Variants:** 245 - Text only: `Role: "assistant", Content: "..."` 246 - With tool calls: `Role: "assistant", Content: "...", ToolCalls: [...]` 247 248### Tool Messages 249 250- **Origin:** Tool execution results 251- **Added:** During `executeToolCalls` 252- **Format:** `Role: "tool", Content: "result", ToolCallID: "..."` 253 254## Thread Safety 255 256**Mutex Protection:** 257 258- `app.mu` protects `app.history` (cli.go:34) 259- Lock acquired for snapshot creation (cli.go:268-271) 260- Lock acquired for message append via `addMessage()` (cli.go:286-288) 261- `opts.History` is local to `ProcessTurn`, no lock needed 262 263**Concurrency Scenarios:** 264 265- Manual input and autoplay both call `processTurn()` serially 266- Each turn operates on its own snapshot 267- Callbacks serialize access to `app.history` via mutex 268 269## Error Handling 270 271**LLM Call Failure:** 272 273- Returns error immediately 274- Caller displays error to user 275- Turn aborted 276 277**Tool Call Failure:** 278 279- Creates error tool result: `Content: "Error: ..."` 280- Continues to next tool in list 281- LLM sees error in next round, can recover 282 283**Max Rounds Exceeded:** 284 285- Returns error: `"too many tool call rounds (limit: 20)"` 286- Indicates LLM stuck in loop (bug or model issue) 287- User sees error, can try different prompt 288 289## Performance Characteristics 290 291**History Compression:** 292 293- Reduces token usage for long conversations 294- Trade-off: Old details lost vs context window limits 295- Compression runs every round (negligible overhead) 296 297**Round Complexity:** 298 299- Each round: 1 LLM call + N tool executions 300- Typical turn: 1-5 rounds 301- Worst case: 20 rounds (max limit) 302 303**Memory:** 304 305- `opts.History` grows during turn (O(messages)) 306- Snapshot copied at turn start (O(messages)) 307- Released at turn end 308 309## Configuration 310 311**MaxToolRounds:** 20 312 313- Prevents infinite loops 314- Increase if legitimate use cases need more rounds 315- Decrease to fail fast during debugging 316 317**HistoryKeepLast:** 10 318 319- Number of recent turns to keep uncompressed 320- Increase for more context (higher token usage) 321- Decrease for faster compression (less context) 322 323## Future Improvements 324 325**Streaming:** Support streaming LLM responses for faster perceived performance 326 327**History Window:** Smarter compression based on token count, not just turn count