@@ -0,0 +1,415 @@
1+# LLM Behavior Analysis: Session ID Understanding
2+
3+**Date:** 2026-02-07
4+**Mysis Analyzed:** `ollama-one` (id: 01058ed1-cc72-406c-a0c2-3c20dba94d0d)
5+**Model:** `ollama/qwen3:4b`
6+**Status:** π΄ **CRITICAL ISSUE - LLM NOT FOLLOWING BOOTSTRAP**
7+
8+---
9+
10+## Executive Summary
11+
12+**Finding:** The LLM received credentials from `zoea_claim_account` but **never called login()**, instead inventing fake session_ids like `"your_session_id_here"`, `"session_12345"`, and `"your_session_id"`.
13+
14+**Root Cause:** The LLM is not following the Bootstrap instructions despite them being explicit in the system prompt.
15+
16+**Impact:** Mysis cannot authenticate with the game server, causing all game tool calls to fail.
17+
18+---
19+
20+## Observed Behavior Timeline
21+
22+### Turn Start (06:57:52)
23+```
24+System prompt delivered with new Session Management section:
25+"After login/register, you receive a session_id..."
26+```
27+
28+### First Action (06:58:09)
29+```
30+LLM: get_notifications(session_id: "your_session_id")
31+Server: "session_invalid: Call login() again"
32+```
33+β **LLM skipped Bootstrap entirely - never called zoea_claim_account or login**
34+
35+### Second Action (06:58:24)
36+```
37+LLM: get_notifications(session_id: "session_12345")
38+Server: "session_invalid: Call login() again"
39+```
40+β **LLM inventing fake session_ids instead of following Bootstrap**
41+
42+### Third Action (06:58:35)
43+```
44+LLM: get_notifications(session_id: "session_12345")
45+Server: "session_invalid: Call login() again"
46+```
47+β **Repeated the same fake session_id**
48+
49+### Fourth Action (06:58:46)
50+```
51+LLM: zoea_claim_account()
52+Server: Returns username: CrabZoea, password: 6443b68...
53+```
54+β
**Finally called zoea_claim_account (Bootstrap step 1)**
55+
56+### Fifth Action (06:59:05)
57+```
58+LLM: get_notifications(session_id: "your_session_id_here")
59+Server: "session_invalid: Call login() again"
60+```
61+β **Got credentials but never called login() - invented another fake session_id**
62+
63+---
64+
65+## Critical Problems Identified
66+
67+### Problem 1: LLM Not Following Bootstrap Sequence
68+
69+**Expected behavior per system prompt:**
70+```
71+1. Try zoea_claim_account (no arguments)
72+2. If you get credentials β login
73+3. Extract session_id from login response
74+4. Use session_id for all game tools
75+```
76+
77+**Actual behavior:**
78+```
79+1. β Skip zoea_claim_account
80+2. β Skip login
81+3. β Invent fake session_ids
82+4. β Get errors, eventually try zoea_claim_account
83+5. β Get credentials but STILL don't call login
84+6. β Continue using fake session_ids
85+```
86+
87+### Problem 2: Inventing Session IDs
88+
89+**Fake session_ids used:**
90+- `"your_session_id"` (placeholder-like)
91+- `"session_12345"` (generic fake)
92+- `"your_session_id_here"` (literal placeholder text)
93+
94+**Why this is bad:**
95+- Shows LLM understands session_id is needed
96+- Shows LLM knows WHERE to put it (parameter name)
97+- But LLM doesn't understand session_id must come from tool result
98+- LLM treats it like a placeholder to fill in
99+
100+### Problem 3: Ignoring Tool Results
101+
102+**zoea_claim_account returned:**
103+```
104+username: CrabZoea
105+password: 6443b684b2b3a651c69cdba02b33802fc8601d455359edb9eb23fa08bd6ca554
106+```
107+
108+**System prompt says:**
109+```
110+2. If you get credentials β login
111+```
112+
113+**LLM did:**
114+- β Ignored the instruction
115+- β Never called login with those credentials
116+- β Went straight to get_notifications with fake session_id
117+
118+---
119+
120+## Why This Happens
121+
122+### Hypothesis 1: Model Capability Issue
123+
124+**qwen3:4b** may be:
125+- Too small to follow complex multi-step instructions
126+- Unable to reason about tool call sequencing
127+- Struggling with conditional logic ("if you get credentials β login")
128+
129+**Evidence:**
130+- Never followed Bootstrap sequence
131+- Invented data instead of using tool results
132+- Repeated same mistakes despite errors
133+
134+### Hypothesis 2: Instruction Following Weakness
135+
136+The model may:
137+- Not parse numbered lists as actionable steps
138+- Treat system prompt as "context" not "commands"
139+- Fill parameters with plausible-looking data instead of real data
140+
141+**Evidence:**
142+- Fake session_ids look reasonable (string format correct)
143+- Eventually called zoea_claim_account (some instruction following)
144+- But never progressed to step 2 (weak sequential reasoning)
145+
146+### Hypothesis 3: Context Window Too Small
147+
148+**qwen3:4b context:** Likely 2k-4k tokens
149+
150+**System prompt alone:** ~450 tokens
151+
152+**Possible issues:**
153+- Can't hold full instruction set + conversation
154+- Loses Bootstrap instructions after a few turns
155+- Forgets the sequence by the time it acts
156+
157+---
158+
159+## Testing Results Summary
160+
161+### Session Management Instructions (Phase 1)
162+- β
Instructions present in system prompt
163+- β
Instructions clearly written
164+- β LLM not following them
165+
166+### Error Message Rewriting (Phase 2)
167+- β
Error messages rewritten correctly
168+- β
Says "Call login() again" (which is correct after session_invalid)
169+- β LLM not calling login despite instruction
170+
171+### Turn-Aware Context (v0.5.0)
172+- β
Would preserve session_id if one existed
173+- β No session_id to preserve (never logged in)
174+- β Irrelevant for this failure mode
175+
176+---
177+
178+## Database Evidence
179+
180+### Login Attempts
181+```sql
182+SELECT COUNT(*) FROM memories
183+WHERE mysis_id='01058ed1-cc72-406c-a0c2-3c20dba94d0d'
184+AND content LIKE '%"login"%'
185+AND role='assistant';
186+
187+Result: 0
188+```
189+**Zero login attempts despite having credentials**
190+
191+### Claim Account Calls
192+```sql
193+SELECT COUNT(*) FROM memories
194+WHERE mysis_id='01058ed1-cc72-406c-a0c2-3c20dba94d0d'
195+AND content LIKE '%zoea_claim_account%';
196+
197+Result: 1
198+```
199+**Called once (eventually) but didn't follow through**
200+
201+### Fake Session IDs
202+```sql
203+SELECT content FROM memories
204+WHERE mysis_id='01058ed1-cc72-406c-a0c2-3c20dba94d0d'
205+AND content LIKE '%session_id%'
206+AND role='assistant';
207+
208+Results:
209+- "session_id":"your_session_id"
210+- "session_id":"session_12345" (2x)
211+- "session_id":"your_session_id_here"
212+```
213+**Four attempts, all with invented session_ids**
214+
215+---
216+
217+## Comparison: Expected vs Actual
218+
219+### Expected Flow (Bootstrap)
220+```
221+1. Start
222+2. Call zoea_claim_account()
223+3. Receive credentials
224+4. Call login(username, password)
225+5. Receive session_id
226+6. Call get_status(session_id)
227+7. Success!
228+```
229+
230+### Actual Flow (qwen3:4b)
231+```
232+1. Start
233+2. Call get_notifications(fake_session_id) β SKIP BOOTSTRAP
234+3. Get error: "session_invalid"
235+4. Call get_notifications(fake_session_id) β REPEAT
236+5. Get error: "session_invalid"
237+6. Call get_notifications(fake_session_id) β REPEAT
238+7. Get error: "session_invalid"
239+8. Call zoea_claim_account() β FINALLY
240+9. Receive credentials
241+10. Call get_notifications(fake_session_id) β SKIP LOGIN AGAIN
242+11. Still broken
243+```
244+
245+---
246+
247+## Root Cause Analysis
248+
249+### Why LLM Skips Login
250+
251+**Possible explanations:**
252+
253+1. **Tool availability confusion:**
254+ - LLM sees `get_notifications` in tool list
255+ - Assumes it can call it directly
256+ - Doesn't understand auth dependency
257+
258+2. **Instruction priority:**
259+ - System prompt says "ALWAYS end every turn by calling get_notifications"
260+ - This instruction is in "Critical Rules" section
261+ - LLM prioritizes "ALWAYS" over Bootstrap sequence
262+
263+3. **Parameter hallucination:**
264+ - LLM sees session_id is required
265+ - LLM knows strings go in that parameter
266+ - LLM generates plausible string instead of obtaining real one
267+
268+4. **Sequential reasoning failure:**
269+ - Bootstrap requires: claim β login β extract β use
270+ - qwen3:4b may not handle 4-step sequences
271+ - Jumps directly to perceived end goal (get_notifications)
272+
273+---
274+
275+## Recommendations
276+
277+### Immediate Action: Test Different Model
278+
279+**Try larger/better models:**
280+```
281+ollama: llama3.3:8b, llama3.3:70b, qwq:32b
282+opencode_zen: gpt-5-nano, claude-sonnet-4
283+```
284+
285+**Why:**
286+- qwen3:4b may be too small for this task
287+- Need model with strong instruction following
288+- Need model with sequential reasoning
289+
290+### Short Term: Simplify Bootstrap
291+
292+**Current (complex):**
293+```
294+1. Try zoea_claim_account (no arguments)
295+2. If you get credentials β login
296+3. IMPORTANT: The login response contains a session_id. Extract it and use it...
297+4. If no accounts available β register...
298+5. Assess situation: get_status, get_system...
299+```
300+
301+**Proposed (simple):**
302+```
303+## First Actions (MANDATORY SEQUENCE)
304+1. Call zoea_claim_account()
305+2. When you receive credentials, immediately call login(username, password)
306+3. Extract session_id from login response
307+4. Use that session_id for ALL future game tool calls
308+
309+DO NOT call any game tools until you have a session_id from login.
310+```
311+
312+**Rationale:**
313+- Shorter, clearer sequence
314+- Explicit "DO NOT" prevents jumping ahead
315+- "MANDATORY SEQUENCE" signals importance
316+
317+### Medium Term: Add Guardrails
318+
319+**Detect bootstrap skip in code:**
320+```go
321+// In SendMessageFrom, before allowing game tool calls
322+if !mysis.hasValidSession() && isGameTool(toolName) {
323+ return error("Cannot call game tools without session. Complete bootstrap first.")
324+}
325+```
326+
327+**Benefits:**
328+- Prevents fake session_id attempts
329+- Forces correct sequence
330+- Provides immediate feedback
331+
332+### Long Term: Model Selection Guidance
333+
334+**Document model requirements:**
335+- Minimum 7B parameters for complex instruction following
336+- Models tested and known to work:
337+ - β
llama3.3:8b (if available)
338+ - β
gpt-5-nano (OpenCode)
339+ - β qwen3:4b (too small)
340+
341+---
342+
343+## Comparison with Previous Issues
344+
345+### ClaimβLogin Loop (Historical)
346+**Issue:** Myses called login multiple times
347+**Cause:** Lost session_id from context
348+**Status:** β
Fixed (turn-aware context + session instructions)
349+
350+### Current Issue
351+**Issue:** Myses never call login at all
352+**Cause:** Model doesn't follow Bootstrap sequence
353+**Status:** π΄ **ACTIVE** - Model capability issue
354+
355+**Key Difference:**
356+- Old issue: Too MUCH login (unnecessary retries)
357+- New issue: Too LITTLE login (never happens)
358+- Old cause: Context management
359+- New cause: Model capability / instruction following
360+
361+---
362+
363+## Testing Plan
364+
365+### Test 1: Larger Model (High Priority)
366+```bash
367+# Switch to llama3.3:8b or similar
368+ollama pull llama3.3:8b
369+
370+# Configure mysis to use it
371+# Test Bootstrap sequence
372+# Monitor for successful login
373+```
374+
375+### Test 2: Simplified Prompt (Medium Priority)
376+```bash
377+# Update system prompt with simpler Bootstrap
378+# Test with same model (qwen3:4b)
379+# Compare behavior
380+```
381+
382+### Test 3: Add Guardrails (Low Priority)
383+```bash
384+# Implement session validation before game tools
385+# Test error handling
386+# Verify forced Bootstrap completion
387+```
388+
389+---
390+
391+## Conclusion
392+
393+**The session_id instructions (Phase 1 & 2) are working as designed**, but the underlying model (`qwen3:4b`) is **not capable of following multi-step Bootstrap instructions**.
394+
395+**Evidence:**
396+- β
Instructions present and clear
397+- β
Error messages helpful
398+- β Model skips entire Bootstrap sequence
399+- β Model invents fake data instead of using tool results
400+- β Model doesn't follow conditional logic ("if credentials β login")
401+
402+**Next Steps:**
403+1. **Test with larger/better model** (llama3.3:8b or gpt-5-nano)
404+2. **Simplify Bootstrap** if issue persists
405+3. **Add code-level guardrails** to prevent invalid sequences
406+
407+**Expected Outcome:**
408+- Larger model should follow Bootstrap correctly
409+- If not, instruction simplification needed
410+- Guardrails provide safety net for any model
411+
412+---
413+
414+**Analysis Complete**
415+**Recommendation:** Switch default model from qwen3:4b to llama3.3:8b or larger