60255
Zn
Zoea-Nova / files
Zoea-Nova / documentation / architecture / CONTEXT_COMPRESSION.md
14.2 KB
1# Context Compression 2 3Zoea Nova uses a loop-based context composition strategy to keep LLM context small and stable. Instead of sending large conversation windows, each LLM request includes only three components: system prompt, a single prompt source, and the most recent tool loop. This guarantees bounded context size (typically 2-10 messages) while the full conversation history remains searchable. 4 5## Design 6 7### The Problem 8 9Without compression, Mysis context grows unbounded: 10- Each tool call adds multiple messages (request + result) 11- SpaceMolt gameplay involves frequent tool use (mining, trading, navigation) 12- Large contexts slow inference and increase costs 13- Eventually hits provider token limits 14- Traditional sliding windows create orphaned tool results and broken message sequences 15 16### The Solution: Loop Slices 17 18``` 19┌─────────────────────────────────────────────────────┐ 20│ LLM Context │ 21│ (Loop-Based Composition) │ 22├─────────────────────────────────────────────────────┤ 23│ [1] System Prompt (always first) │ 24├─────────────────────────────────────────────────────┤ 25│ [2] Current Turn Start │ 26│ Selected by recency (most recent wins): │ 27│ • Commander direct message │ 28│ • Last commander broadcast │ 29│ • Last swarm broadcast │ 30│ • Synthetic nudge (if no prompts found) │ 31├─────────────────────────────────────────────────────┤ 32│ [3] Latest Tool Loop (if any) │ 33│ • Tool call message (assistant with tools) │ 34│ • All subsequent tool results │ 35└─────────────────────────────────────────────────────┘ 36 37┌─────────────────────────────────────────────────────┐ 38│ Full Memory (SQLite) │ 39├─────────────────────────────────────────────────────┤ 40│ All messages since Mysis creation │ 41│ Searchable via zoea_search_messages tool │ 42│ Reasoning searchable via zoea_search_reasoning │ 43└─────────────────────────────────────────────────────┘ 44``` 45 46## Architecture 47 48### Loop Slice Model 49 50A **loop slice** is the atomic unit of tool interaction: 511. **Tool call message** (assistant role, contains `[TOOL_CALLS]` prefix) 522. **Tool results** (tool role, consecutive messages following the call) 53 54The most recent loop slice is always included in context to maintain tool-calling coherence. 55 56### Prompt Source Priority 57 58Prompt sources are user messages that drive Mysis behavior and mark the start of the current turn. Selection order (most recent message wins): 59 601. **Commander direct message** (`source="direct"`) - Direct command from user 612. **Last commander broadcast** (`source="broadcast"`, `sender_id=""`) - Commander identified by empty sender_id 623. **Last swarm broadcast** (`source="broadcast"`, `sender_id!=""`) - Most recent broadcast from another Mysis 634. **Synthetic nudge** (generated, not stored) - Fallback when no prompt sources exist 64 65See [Terminology](#terminology) for definitions of turn boundaries and context composition. 66 67### Context Composition Algorithm 68 69```go 70// internal/core/mysis.go:1374-1421 71 72func (m *Mysis) getContextMemories() ([]*store.Memory, error) { 73 // Fetch recent memories from DB 74 allMemories := GetRecentMemories(m.id, MaxContextMessages) 75 76 result := [] 77 78 // Step 1: Add system prompt (if available) 79 system := GetSystemMemory(m.id) 80 if system != nil { 81 result.append(system) 82 } 83 84 // Step 2: Find current turn boundary (most recent user message) 85 turnStart := selectPromptSource(allMemories) 86 if turnStart == nil { 87 // Generate synthetic nudge (not stored in DB) 88 nudge := createEphemeralNudge() 89 result.append(nudge) 90 } else { 91 result.append(turnStart) 92 } 93 94 // Step 3: Extract latest tool loop 95 toolLoop := extractLatestToolLoop(allMemories) 96 result.append(toolLoop...) 97 98 return result 99} 100``` 101 102### Loop Extraction 103 104```go 105// internal/core/mysis.go:1326-1372 106 107func extractLatestToolLoop(memories []*store.Memory) []*store.Memory { 108 // Scan backwards (newest first) for most recent tool call 109 toolCallIdx := findLastToolCallMessage(memories) 110 if toolCallIdx == -1 { 111 return nil 112 } 113 114 // Collect tool call + all consecutive tool results 115 loop := [memories[toolCallIdx]] 116 for i := toolCallIdx + 1; i < len(memories); i++ { 117 if memories[i].Role == "tool" { 118 loop.append(memories[i]) 119 } else { 120 break // Stop at first non-tool message 121 } 122 } 123 124 return loop 125} 126``` 127 128Tool call messages are identified by: 129- Role: `assistant` 130- Content prefix: `[TOOL_CALLS]` 131 132## Bounded Size Guarantees 133 134The loop slice model provides strict size bounds: 135 136| Component | Typical Size | Max Size | 137|-----------|--------------|----------| 138| System prompt | 1 message | 1 message | 139| Prompt source | 1 message | 1 message | 140| Tool loop | 1-8 messages | ~15 messages (1 call + max results) | 141| **Total** | **3-10 messages** | **~17 messages** | 142 143This is dramatically smaller than traditional sliding windows (20-50 messages) and eliminates orphaned tool sequencing issues. 144 145## Synthetic Nudge Behavior 146 147When no prompt source is found in recent memory, a synthetic nudge is generated: 148 149```go 150// internal/core/mysis.go:1398-1409 151 152nudgeContent := "Continue your mission. Check notifications and coordinate with the swarm." 153nudgeMemory := &store.Memory{ 154 Role: "user", 155 Source: "system", 156 Content: nudgeContent, 157 SenderID: "", 158 CreatedAt: time.Now(), 159} 160``` 161 162**Key properties:** 163- **NOT stored in database** - Ephemeral automation scaffolding 164- **Counted for circuit breaker** - Nudge counter increments in ticker loop (see `internal/core/mysis.go:1624-1641`) 165- **Escalating prompts** - Content changes based on `encouragementCount` to increase urgency: 166 - Attempt 1: "Continue your mission. Check notifications and coordinate with the swarm." (gentle) 167 - Attempt 2: "You need to respond. Check your notifications and status." (firm) 168 - Attempt 3: "URGENT: Respond immediately. Check your system status." (urgent) 169 - After 3 failures: Mysis transitions to idle state with error "Failed to respond after 3 encouragements" 170- **Escalation intervals** - Nudges sent every 30 seconds for idle myses, 2 minutes for wait states (see `internal/constants/constants.go:69-73`) 171 172### Nudge Circuit Breaker 173 174```go 175// internal/core/mysis.go:1624-1641 176 177if shouldNudge(time.Now()) { 178 encouragementCount++ 179 180 if encouragementCount >= 3 { 181 setIdle("Failed to respond after 3 encouragements") 182 return 183 } 184 185 nudgeCh <- struct{}{} // Trigger nudge processing 186} 187``` 188 189**Counter reset:** 190- Reset to 0 on successful LLM response (line 616) 191- Prevents idle Myses from looping indefinitely 192- Forces transition to idle state after 3 failed nudges 193 194For state machine transitions related to nudge failures, see [MYSIS_STATE_MACHINE.md](MYSIS_STATE_MACHINE.md#state-transitions). 195 196### Nudge Intervals 197 198```go 199// internal/constants/constants.go:69-73 200 201IdleNudgeInterval = 30 * time.Second // Default idle nudging 202WaitStateNudgeInterval = 2 * time.Minute // Nudging during wait states 203``` 204 205## Search Tools 206 207Myses can retrieve older memories using MCP tools exposed through the orchestrator: 208 209### zoea_search_messages 210 211Search a Mysis's past messages by text content. 212 213```json 214{ 215 "mysis_id": "abc123", 216 "query": "iron ore", 217 "limit": 20 218} 219``` 220 221Returns matching messages with role, source, content, and timestamp. 222 223### zoea_search_reasoning 224 225Search a Mysis's past reasoning content by text. 226 227```json 228{ 229 "mysis_id": "abc123", 230 "query": "travel cooldown", 231 "limit": 20 232} 233``` 234 235Returns matching reasoning entries with role, source, content, reasoning, and timestamp. 236 237## System Prompt Guidance 238 239The system prompt instructs Myses to use their captain's log for persistent memory and search tools for older context: 240 241``` 242## Critical Rules 243Context is limited - use search tools for older information. 244``` 245 246The continue prompt also reminds Myses about search tools when generated by nudging logic. 247 248This encourages Myses to: 2491. Externalize important information to the game's built-in note system 2502. Use search tools to retrieve older context when needed 2513. Understand that their context window is minimal and focused 252 253## Benefits Over Sliding Windows 254 255| Aspect | Loop Slices | Traditional Sliding Window | 256|--------|-------------|---------------------------| 257| Size | 2-10 messages | 20-50 messages | 258| Stability | Always same structure | Variable, depends on history | 259| Tool coherence | Complete loops guaranteed | Often broken mid-sequence | 260| Orphaned messages | Never | Common (tool results without calls) | 261| Context bloat | None | Redundant state snapshots accumulate | 262| Inference speed | Fast (minimal tokens) | Slower (large context) | 263 264## Implementation Details 265 266### Constants 267 268```go 269// internal/constants/constants.go:62-64 270 271MaxContextMessages = 20 // Window for scanning recent memories 272``` 273 274Note: `MaxContextMessages` is the scanning window, NOT the final context size. The loop slice extraction typically produces 2-10 messages regardless of this value. 275 276### Store Methods 277 278```go 279// internal/store/memories.go 280 281// GetSystemMemory retrieves the initial system prompt for a Mysis. 282func (s *Store) GetSystemMemory(mysisID string) (*Memory, error) 283 284// GetRecentMemories retrieves the most recent N memories for a Mysis. 285func (s *Store) GetRecentMemories(mysisID string, limit int) ([]*Memory, error) 286 287// SearchMemories searches memories by content text (case-sensitive). 288func (s *Store) SearchMemories(mysisID, query string, limit int) ([]*Memory, error) 289``` 290 291## Tradeoffs 292 293| Aspect | Benefit | Cost | 294|--------|---------|------| 295| Inference speed | Very fast with minimal context | May lose relevant older context | 296| Token costs | Minimal per-request costs | Search queries add overhead | 297| Memory access | Full history searchable | Requires explicit search | 298| Coherence | Recent tool loops always complete | Long-term plans may be forgotten | 299| Predictability | Same structure every time | Less conversational continuity | 300 301## Tuning 302 303The loop slice model has minimal tuning surface: 304 305- **`MaxContextMessages`**: Scanning window for finding turn boundaries and tool loops. Larger values increase DB query cost but provide more history to search. Default: 20 messages (~2 server ticks). 306- **Turn boundary selection**: Most recent user message wins (commander direct > commander broadcast > swarm broadcast > nudge). 307- **Nudge intervals**: Control how often idle Myses are prompted. Faster intervals increase responsiveness but may interrupt LLM processing. 308 309## Migration Notes 310 311This architecture replaced the previous snapshot compaction model (v0.4.x) which used a 20-message sliding window with snapshot deduplication. The old model: 312- Sent 20 recent messages per LLM request 313- Deduplicated redundant `get_*` tool results 314- Often broke tool call sequences mid-loop 315- Created orphaned tool results 316 317The loop slice model eliminates these issues by composing context from atomic units (system + prompt + loop) rather than arbitrary time windows. 318 319See `documentation/archive/` for historical compaction implementation details. 320 321## Turn-Aware Context Composition (v0.5.0+) 322 323As of v0.5.0, context composition distinguishes between historical turns and the current turn. 324 325### Architecture 326 327**Turn Boundary:** Detected by finding the most recent user-initiated prompt (direct message, broadcast, or system nudge). 328 329**Historical Turns:** All messages before the turn boundary 330- Compressed using `extractLatestToolLoop()` 331- Only the most recent complete tool loop is included 332- Saves context space for older conversations 333 334**Current Turn:** All messages from turn boundary to present 335- Included WITHOUT compression 336- Preserves complete multi-step tool sequences 337- Essential for multi-step reasoning (e.g., login → get_status → get_notifications) 338 339### Example 340 341``` 342Historical: 343 [user: "old question"] 344 [assistant: tool_call_1] 345 [tool: result_1] 346 [assistant: tool_call_2] ← Only latest loop included 347 [tool: result_2] ← Only latest loop included 348 349Current Turn: 350 [user: "new question"] ← Turn boundary 351 [assistant: login_call] ← All included 352 [tool: session_id: abc123] ← All included 353 [assistant: status_call] ← All included 354 [tool: status_result] ← All included 355``` 356 357### Benefits 358 3591. **Multi-step reasoning:** LLM can reference earlier tool results within the same turn 3602. **Session persistence:** Session IDs from login remain visible throughout the turn 3613. **Context efficiency:** Historical turns compressed, current turn preserved 3624. **Orphan prevention:** Complete tool loops stay together 363 364### Implementation 365 366**Function:** `getContextMemories()` in `internal/core/mysis.go` 367 368**Turn Detection:** `findLastUserPromptIndex()` scans backwards to find the most recent user prompt 369 370**Composition:** 3711. System prompt (always first) 3722. Historical context (compressed via `extractLatestToolLoop()`) 3733. Current turn (uncompressed, from turn boundary to present) 374 375### Migration from v0.4.x 376 377**Before (Loop-based):** 378- Context: `[system] + [prompt source] + [latest tool loop only]` 379- Issue: Multi-step tool sequences truncated (login result lost) 380 381**After (Turn-aware):** 382- Context: `[system] + [historical compressed] + [current turn complete]` 383- Fixed: Complete current turn preserved, enabling multi-step reasoning