xonecas.io
60255
Zn

Zoea-Nova / files

Zoea-Nova / documentation / architecture / CONTEXT_COMPRESSION.md

14.2 KB
1# Context Compression
2
3Zoea Nova uses a loop-based context composition strategy to keep LLM context small and stable. Instead of sending large conversation windows, each LLM request includes only three components: system prompt, a single prompt source, and the most recent tool loop. This guarantees bounded context size (typically 2-10 messages) while the full conversation history remains searchable.
4
5## Design
6
7### The Problem
8
9Without compression, Mysis context grows unbounded:
10- Each tool call adds multiple messages (request + result)
11- SpaceMolt gameplay involves frequent tool use (mining, trading, navigation)
12- Large contexts slow inference and increase costs
13- Eventually hits provider token limits
14- Traditional sliding windows create orphaned tool results and broken message sequences
15
16### The Solution: Loop Slices
17
18```
19┌─────────────────────────────────────────────────────┐
20│                   LLM Context                       │
21│              (Loop-Based Composition)               │
22├─────────────────────────────────────────────────────┤
23│  [1] System Prompt (always first)                   │
24├─────────────────────────────────────────────────────┤
25│  [2] Current Turn Start                              │
26│      Selected by recency (most recent wins):        │
27│      • Commander direct message                     │
28│      • Last commander broadcast                     │
29│      • Last swarm broadcast                         │
30│      • Synthetic nudge (if no prompts found)        │
31├─────────────────────────────────────────────────────┤
32│  [3] Latest Tool Loop (if any)                      │
33│      • Tool call message (assistant with tools)     │
34│      • All subsequent tool results                  │
35└─────────────────────────────────────────────────────┘
36
37┌─────────────────────────────────────────────────────┐
38│              Full Memory (SQLite)                   │
39├─────────────────────────────────────────────────────┤
40│  All messages since Mysis creation                  │
41│  Searchable via zoea_search_messages tool           │
42│  Reasoning searchable via zoea_search_reasoning     │
43└─────────────────────────────────────────────────────┘
44```
45
46## Architecture
47
48### Loop Slice Model
49
50A **loop slice** is the atomic unit of tool interaction:
511. **Tool call message** (assistant role, contains `[TOOL_CALLS]` prefix)
522. **Tool results** (tool role, consecutive messages following the call)
53
54The most recent loop slice is always included in context to maintain tool-calling coherence.
55
56### Prompt Source Priority
57
58Prompt sources are user messages that drive Mysis behavior and mark the start of the current turn. Selection order (most recent message wins):
59
601. **Commander direct message** (`source="direct"`) - Direct command from user
612. **Last commander broadcast** (`source="broadcast"`, `sender_id=""`) - Commander identified by empty sender_id
623. **Last swarm broadcast** (`source="broadcast"`, `sender_id!=""`) - Most recent broadcast from another Mysis
634. **Synthetic nudge** (generated, not stored) - Fallback when no prompt sources exist
64
65See [Terminology](#terminology) for definitions of turn boundaries and context composition.
66
67### Context Composition Algorithm
68
69```go
70// internal/core/mysis.go:1374-1421
71
72func (m *Mysis) getContextMemories() ([]*store.Memory, error) {
73    // Fetch recent memories from DB
74    allMemories := GetRecentMemories(m.id, MaxContextMessages)
75    
76    result := []
77    
78    // Step 1: Add system prompt (if available)
79    system := GetSystemMemory(m.id)
80    if system != nil {
81        result.append(system)
82    }
83    
84    // Step 2: Find current turn boundary (most recent user message)
85    turnStart := selectPromptSource(allMemories)
86    if turnStart == nil {
87        // Generate synthetic nudge (not stored in DB)
88        nudge := createEphemeralNudge()
89        result.append(nudge)
90    } else {
91        result.append(turnStart)
92    }
93    
94    // Step 3: Extract latest tool loop
95    toolLoop := extractLatestToolLoop(allMemories)
96    result.append(toolLoop...)
97    
98    return result
99}
100```
101
102### Loop Extraction
103
104```go
105// internal/core/mysis.go:1326-1372
106
107func extractLatestToolLoop(memories []*store.Memory) []*store.Memory {
108    // Scan backwards (newest first) for most recent tool call
109    toolCallIdx := findLastToolCallMessage(memories)
110    if toolCallIdx == -1 {
111        return nil
112    }
113    
114    // Collect tool call + all consecutive tool results
115    loop := [memories[toolCallIdx]]
116    for i := toolCallIdx + 1; i < len(memories); i++ {
117        if memories[i].Role == "tool" {
118            loop.append(memories[i])
119        } else {
120            break  // Stop at first non-tool message
121        }
122    }
123    
124    return loop
125}
126```
127
128Tool call messages are identified by:
129- Role: `assistant`
130- Content prefix: `[TOOL_CALLS]`
131
132## Bounded Size Guarantees
133
134The loop slice model provides strict size bounds:
135
136| Component | Typical Size | Max Size |
137|-----------|--------------|----------|
138| System prompt | 1 message | 1 message |
139| Prompt source | 1 message | 1 message |
140| Tool loop | 1-8 messages | ~15 messages (1 call + max results) |
141| **Total** | **3-10 messages** | **~17 messages** |
142
143This is dramatically smaller than traditional sliding windows (20-50 messages) and eliminates orphaned tool sequencing issues.
144
145## Synthetic Nudge Behavior
146
147When no prompt source is found in recent memory, a synthetic nudge is generated:
148
149```go
150// internal/core/mysis.go:1398-1409
151
152nudgeContent := "Continue your mission. Check notifications and coordinate with the swarm."
153nudgeMemory := &store.Memory{
154    Role:      "user",
155    Source:    "system",
156    Content:   nudgeContent,
157    SenderID:  "",
158    CreatedAt: time.Now(),
159}
160```
161
162**Key properties:**
163- **NOT stored in database** - Ephemeral automation scaffolding
164- **Counted for circuit breaker** - Nudge counter increments in ticker loop (see `internal/core/mysis.go:1624-1641`)
165- **Escalating prompts** - Content changes based on `encouragementCount` to increase urgency:
166  - Attempt 1: "Continue your mission. Check notifications and coordinate with the swarm." (gentle)
167  - Attempt 2: "You need to respond. Check your notifications and status." (firm)
168  - Attempt 3: "URGENT: Respond immediately. Check your system status." (urgent)
169  - After 3 failures: Mysis transitions to idle state with error "Failed to respond after 3 encouragements"
170- **Escalation intervals** - Nudges sent every 30 seconds for idle myses, 2 minutes for wait states (see `internal/constants/constants.go:69-73`)
171
172### Nudge Circuit Breaker
173
174```go
175// internal/core/mysis.go:1624-1641
176
177if shouldNudge(time.Now()) {
178    encouragementCount++
179    
180    if encouragementCount >= 3 {
181        setIdle("Failed to respond after 3 encouragements")
182        return
183    }
184    
185    nudgeCh <- struct{}{}  // Trigger nudge processing
186}
187```
188
189**Counter reset:**
190- Reset to 0 on successful LLM response (line 616)
191- Prevents idle Myses from looping indefinitely
192- Forces transition to idle state after 3 failed nudges
193
194For state machine transitions related to nudge failures, see [MYSIS_STATE_MACHINE.md](MYSIS_STATE_MACHINE.md#state-transitions).
195
196### Nudge Intervals
197
198```go
199// internal/constants/constants.go:69-73
200
201IdleNudgeInterval = 30 * time.Second      // Default idle nudging
202WaitStateNudgeInterval = 2 * time.Minute  // Nudging during wait states
203```
204
205## Search Tools
206
207Myses can retrieve older memories using MCP tools exposed through the orchestrator:
208
209### zoea_search_messages
210
211Search a Mysis's past messages by text content.
212
213```json
214{
215  "mysis_id": "abc123",
216  "query": "iron ore",
217  "limit": 20
218}
219```
220
221Returns matching messages with role, source, content, and timestamp.
222
223### zoea_search_reasoning
224
225Search a Mysis's past reasoning content by text.
226
227```json
228{
229  "mysis_id": "abc123",
230  "query": "travel cooldown",
231  "limit": 20
232}
233```
234
235Returns matching reasoning entries with role, source, content, reasoning, and timestamp.
236
237## System Prompt Guidance
238
239The system prompt instructs Myses to use their captain's log for persistent memory and search tools for older context:
240
241```
242## Critical Rules
243Context is limited - use search tools for older information.
244```
245
246The continue prompt also reminds Myses about search tools when generated by nudging logic.
247
248This encourages Myses to:
2491. Externalize important information to the game's built-in note system
2502. Use search tools to retrieve older context when needed
2513. Understand that their context window is minimal and focused
252
253## Benefits Over Sliding Windows
254
255| Aspect | Loop Slices | Traditional Sliding Window |
256|--------|-------------|---------------------------|
257| Size | 2-10 messages | 20-50 messages |
258| Stability | Always same structure | Variable, depends on history |
259| Tool coherence | Complete loops guaranteed | Often broken mid-sequence |
260| Orphaned messages | Never | Common (tool results without calls) |
261| Context bloat | None | Redundant state snapshots accumulate |
262| Inference speed | Fast (minimal tokens) | Slower (large context) |
263
264## Implementation Details
265
266### Constants
267
268```go
269// internal/constants/constants.go:62-64
270
271MaxContextMessages = 20  // Window for scanning recent memories
272```
273
274Note: `MaxContextMessages` is the scanning window, NOT the final context size. The loop slice extraction typically produces 2-10 messages regardless of this value.
275
276### Store Methods
277
278```go
279// internal/store/memories.go
280
281// GetSystemMemory retrieves the initial system prompt for a Mysis.
282func (s *Store) GetSystemMemory(mysisID string) (*Memory, error)
283
284// GetRecentMemories retrieves the most recent N memories for a Mysis.
285func (s *Store) GetRecentMemories(mysisID string, limit int) ([]*Memory, error)
286
287// SearchMemories searches memories by content text (case-sensitive).
288func (s *Store) SearchMemories(mysisID, query string, limit int) ([]*Memory, error)
289```
290
291## Tradeoffs
292
293| Aspect | Benefit | Cost |
294|--------|---------|------|
295| Inference speed | Very fast with minimal context | May lose relevant older context |
296| Token costs | Minimal per-request costs | Search queries add overhead |
297| Memory access | Full history searchable | Requires explicit search |
298| Coherence | Recent tool loops always complete | Long-term plans may be forgotten |
299| Predictability | Same structure every time | Less conversational continuity |
300
301## Tuning
302
303The loop slice model has minimal tuning surface:
304
305- **`MaxContextMessages`**: Scanning window for finding turn boundaries and tool loops. Larger values increase DB query cost but provide more history to search. Default: 20 messages (~2 server ticks).
306- **Turn boundary selection**: Most recent user message wins (commander direct > commander broadcast > swarm broadcast > nudge).
307- **Nudge intervals**: Control how often idle Myses are prompted. Faster intervals increase responsiveness but may interrupt LLM processing.
308
309## Migration Notes
310
311This architecture replaced the previous snapshot compaction model (v0.4.x) which used a 20-message sliding window with snapshot deduplication. The old model:
312- Sent 20 recent messages per LLM request
313- Deduplicated redundant `get_*` tool results
314- Often broke tool call sequences mid-loop
315- Created orphaned tool results
316
317The loop slice model eliminates these issues by composing context from atomic units (system + prompt + loop) rather than arbitrary time windows.
318
319See `documentation/archive/` for historical compaction implementation details.
320
321## Turn-Aware Context Composition (v0.5.0+)
322
323As of v0.5.0, context composition distinguishes between historical turns and the current turn.
324
325### Architecture
326
327**Turn Boundary:** Detected by finding the most recent user-initiated prompt (direct message, broadcast, or system nudge).
328
329**Historical Turns:** All messages before the turn boundary
330- Compressed using `extractLatestToolLoop()`
331- Only the most recent complete tool loop is included
332- Saves context space for older conversations
333
334**Current Turn:** All messages from turn boundary to present
335- Included WITHOUT compression
336- Preserves complete multi-step tool sequences
337- Essential for multi-step reasoning (e.g., login → get_status → get_notifications)
338
339### Example
340
341```
342Historical:
343  [user: "old question"]
344  [assistant: tool_call_1]
345  [tool: result_1]
346  [assistant: tool_call_2]  ← Only latest loop included
347  [tool: result_2]           ← Only latest loop included
348
349Current Turn:
350  [user: "new question"]      ← Turn boundary
351  [assistant: login_call]     ← All included
352  [tool: session_id: abc123]  ← All included
353  [assistant: status_call]    ← All included
354  [tool: status_result]       ← All included
355```
356
357### Benefits
358
3591. **Multi-step reasoning:** LLM can reference earlier tool results within the same turn
3602. **Session persistence:** Session IDs from login remain visible throughout the turn
3613. **Context efficiency:** Historical turns compressed, current turn preserved
3624. **Orphan prevention:** Complete tool loops stay together
363
364### Implementation
365
366**Function:** `getContextMemories()` in `internal/core/mysis.go`
367
368**Turn Detection:** `findLastUserPromptIndex()` scans backwards to find the most recent user prompt
369
370**Composition:**
3711. System prompt (always first)
3722. Historical context (compressed via `extractLatestToolLoop()`)
3733. Current turn (uncompressed, from turn boundary to present)
374
375### Migration from v0.4.x
376
377**Before (Loop-based):**
378- Context: `[system] + [prompt source] + [latest tool loop only]`
379- Issue: Multi-step tool sequences truncated (login result lost)
380
381**After (Turn-aware):**
382- Context: `[system] + [historical compressed] + [current turn complete]`
383- Fixed: Complete current turn preserved, enabling multi-step reasoning