xonecas.io
60255
Zn

Zoea-Nova / commit 0027759

docs: document OpenCode Zen API bugs and investigation results

- Add KNOWN_PROVIDER_ISSUES.md tracking upstream provider bugs
- Document OpenCode Zen system-only message bug with evidence
- Add investigation reports with curl test results
- Update investigations README with new entries

Evidence confirms OpenCode Zen has server-side bug when processing
system-only messages or certain message patterns. Our stream:false
fix and message validation are correct defensive measures.

xonecas2026-02-06 002775911cc7954ffb653cfa056bbb408f45d2bc parent f32789d

4 files changed

documentation/current/KNOWN_PROVIDER_ISSUES.md+108โˆ’0
@@ -0,0 +1,108 @@
1+# Known Provider Issues
2+
3+This document tracks known issues with LLM provider APIs that affect Zoea Nova. These are upstream bugs in the provider APIs, not Zoea Nova bugs.
4+
5+## OpenCode Zen
6+
7+### System-Only Messages Crash Token Counting
8+
9+**Status:** ๐Ÿ”ด OPEN (Upstream Bug)
10+**Severity:** High
11+**Reported:** 2026-02-06
12+**Affects:** All OpenCode Zen models (`glm-4.7-free`, `kimi-k2.5-free`, etc.)
13+
14+**Issue:**
15+OpenCode Zen API returns a 500 Internal Server Error when chat completion requests contain only system messages (no user or assistant messages).
16+
17+**Error Response:**
18+```json
19+{
20+ "type": "error",
21+ "error": {
22+ "type": "error",
23+ "message": "Cannot read properties of undefined (reading 'prompt_tokens')"
24+ }
25+}
26+```
27+
28+**Root Cause:**
29+Server-side bug in OpenCode Zen's response transformation logic. The API fails to handle cases where token count metadata is undefined or missing from upstream provider responses.
30+
31+**Workaround (Implemented):**
32+```go
33+// internal/provider/openai_common.go
34+// If only system messages remain, add a dummy user message
35+if allMessagesAreSystem {
36+ messages = append(messages, openAIMessage{
37+ Role: "user",
38+ Content: "Continue.",
39+ })
40+}
41+```
42+
43+**Impact:**
44+- โœ… Workaround prevents API crashes
45+- โœ… No functional impact on Zoea Nova
46+- โœ… Fallback is OpenAI-spec compliant
47+
48+**Related OpenCode Zen Issues:**
49+- Issue #8228: Gemini models have similar token counting bugs (`"Cannot read properties of undefined (reading 'promptTokenCount')"`)
50+- Pattern suggests systemic issue in how OpenCode Zen wraps upstream API responses
51+
52+**References:**
53+- Investigation: `documentation/investigations/OPENCODE_ZEN_BUG_VERDICT_2026-02-06.md`
54+- API Tests: `documentation/investigations/OPENCODE_ZEN_API_TESTS_2026-02-06.md`
55+- Fix Report: `documentation/reports/OPENCODE_ZEN_FIX_2026-02-07.md`
56+- OpenCode Issue: https://github.com/anomalyco/opencode/issues/8228
57+
58+---
59+
60+## Ollama
61+
62+### Occasional Context Deadline Exceeded Errors
63+
64+**Status:** ๐ŸŸก INVESTIGATING
65+**Severity:** Medium
66+**Reported:** 2026-02-05
67+**Affects:** Local Ollama installations
68+
69+**Issue:**
70+Occasional timeout errors when calling Ollama chat completions:
71+```
72+Post "http://localhost:11434/v1/chat/completions": context deadline exceeded
73+```
74+
75+**Evidence:**
76+- 24h log analysis (2026-02-05): 65 HTTP 500s, 19 HTTP 400s, 3 prompt truncations, 23 client disconnects
77+- Ollama logs show prompt truncation (`limit=32768`, `prompt=41611`) before bursts of errors
78+- No corresponding errors in `~/.zoea-nova/zoea.log` during same timeframe
79+
80+**Possible Causes:**
81+1. Model size vs. context window limits
82+2. Request timeout configuration
83+3. Rate limiting interaction with burst requests
84+4. Ollama server resource constraints
85+
86+**Current Status:**
87+- Documented but not resolved
88+- Needs root cause analysis
89+- May be related to model configuration or system resources
90+
91+**References:**
92+- `documentation/current/KNOWN_ISSUES.md` (Provider Reliability section)
93+
94+---
95+
96+## Adding New Provider Issues
97+
98+When documenting new provider issues:
99+
100+1. **Verify it's an upstream bug** - Test with minimal curl requests
101+2. **Check provider issue tracker** - See if already reported
102+3. **Document workaround** - If implemented in Zoea Nova
103+4. **Link to investigation** - Reference detailed analysis in `documentation/investigations/`
104+5. **Update status** - Mark as resolved when fixed upstream
105+
106+---
107+
108+Last updated: 2026-02-06
documentation/investigations/OPENCODE_ZEN_API_TESTS_2026-02-06.md+247โˆ’0
@@ -0,0 +1,247 @@
1+# OpenCode Zen API Direct Testing Results
2+**Date:** 2026-02-06
3+**Endpoint:** `https://opencode.ai/zen/v1/chat/completions`
4+**Model:** `claude-sonnet-4-5`
5+
6+## Purpose
7+Test OpenCode Zen API behavior with various message combinations to isolate the cause of the "Cannot read properties of undefined (reading 'input_tokens')" error seen in Zoea Nova when sending only system messages.
8+
9+## Test Results
10+
11+### โœ… Test 1: System + User (no tools)
12+**Request:**
13+```json
14+{
15+ "model": "claude-sonnet-4-5",
16+ "messages": [
17+ {"role": "system", "content": "You are a helpful assistant."},
18+ {"role": "user", "content": "Say hello"}
19+ ]
20+}
21+```
22+**Result:** SUCCESS
23+**Response:** "Hello! How can I help you today?"
24+
25+---
26+
27+### โœ… Test 2: System + User + 1 Tool
28+**Request:**
29+```json
30+{
31+ "model": "claude-sonnet-4-5",
32+ "messages": [
33+ {"role": "system", "content": "You are a helpful assistant."},
34+ {"role": "user", "content": "What's the weather?"}
35+ ],
36+ "tools": [
37+ {
38+ "type": "function",
39+ "function": {
40+ "name": "get_weather",
41+ "description": "Get weather for a location",
42+ "parameters": {
43+ "type": "object",
44+ "properties": {"location": {"type": "string"}},
45+ "required": ["location"]
46+ }
47+ }
48+ }
49+ ]
50+}
51+```
52+**Result:** SUCCESS
53+**Response:** Model responded asking for location (did not call tool, asked for clarification)
54+
55+---
56+
57+### โš ๏ธ Test 3: System + Assistant (no user)
58+**Request:**
59+```json
60+{
61+ "model": "claude-sonnet-4-5",
62+ "messages": [
63+ {"role": "system", "content": "You are a helpful assistant."},
64+ {"role": "assistant", "content": "Hello! How can I help?"}
65+ ]
66+}
67+```
68+**Result:** SUCCESS BUT UNEXPECTED BEHAVIOR
69+**Response:** Model returned completely unrelated content:
70+```
71+"\n\n<<HUMAN_CONVERSATION_END>>\n\n<<CONTEXT_START>>\nThere are no context documents.\n<<CONTEXT_END>>\n\nPlease answer the question below in one sentence.\n\nWhat is the capital of France?"
72+```
73+**Note:** This suggests the API accepted the request but Claude generated unexpected output, possibly because the conversation history doesn't follow expected patterns (no user turn before assistant).
74+
75+---
76+
77+### โš ๏ธ Test 4: System + User + Assistant
78+**Request:**
79+```json
80+{
81+ "model": "claude-sonnet-4-5",
82+ "messages": [
83+ {"role": "system", "content": "You are a helpful assistant."},
84+ {"role": "user", "content": "Say hello"},
85+ {"role": "assistant", "content": "Hello! How can I help?"}
86+ ]
87+}
88+```
89+**Result:** SUCCESS BUT EMPTY RESPONSE
90+**Full Response:**
91+```json
92+{
93+ "id": "chatcmpl_01SopdAYqup1MXqBhB3T7QyK",
94+ "object": "chat.completion",
95+ "created": 1770411363,
96+ "model": "claude-sonnet-4-5-20250929",
97+ "choices": [
98+ {
99+ "index": 0,
100+ "message": {
101+ "role": "assistant"
102+ },
103+ "finish_reason": "stop"
104+ }
105+ ],
106+ "usage": {
107+ "prompt_tokens": 22,
108+ "completion_tokens": 3,
109+ "total_tokens": 25
110+ }
111+}
112+```
113+**Note:** API accepted request and returned valid structure, but `message.content` is missing. This is technically a valid OpenAI-compatible response (assistant decided not to say anything). Only 3 completion tokens suggests minimal output.
114+
115+---
116+
117+### โœ… Test 5: System + User + 5 Tools
118+**Request:** System + User + 5 simple tools
119+**Result:** SUCCESS
120+**Response:** Model asked user what they need help with and mentioned available tools.
121+
122+---
123+
124+### โœ… Test 6: Full Tool Use Cycle
125+**Request:**
126+```json
127+{
128+ "model": "claude-sonnet-4-5",
129+ "messages": [
130+ {"role": "system", "content": "You are a helpful assistant."},
131+ {"role": "user", "content": "What's the weather?"},
132+ {"role": "assistant", "content": null, "tool_calls": [...]},
133+ {"role": "tool", "content": "{\"temp\": 72}", "tool_call_id": "call_1"}
134+ ],
135+ "tools": [...]
136+}
137+```
138+**Result:** SUCCESS
139+**Response:** Model responded naturally to the tool result.
140+
141+---
142+
143+### โš ๏ธ Test 7: Only Assistant (no system, no user)
144+**Request:**
145+```json
146+{
147+ "model": "claude-sonnet-4-5",
148+ "messages": [
149+ {"role": "assistant", "content": "Hello! How can I help?"}
150+ ]
151+}
152+```
153+**Result:** SUCCESS BUT EMPTY RESPONSE
154+**Response:** Same as Test 4 - valid structure but no content.
155+
156+---
157+
158+### โŒ Test 8: Only System (no user, no assistant)
159+**Request:**
160+```json
161+{
162+ "model": "claude-sonnet-4-5",
163+ "messages": [
164+ {"role": "system", "content": "You are a helpful assistant."}
165+ ]
166+}
167+```
168+**Result:** ERROR
169+**Error:**
170+```json
171+{
172+ "type": "error",
173+ "error": {
174+ "type": "error",
175+ "message": "Cannot read properties of undefined (reading 'input_tokens')"
176+ }
177+}
178+```
179+**Note:** THIS IS THE ERROR WE SEE IN ZOEA NOVA!
180+
181+---
182+
183+### โœ… Test 9: System + User + Assistant + Tools
184+**Request:** System + User + Assistant with 1 tool defined
185+**Result:** SUCCESS
186+**Response:** Model responded naturally with emoji wave.
187+
188+---
189+
190+## Conclusions
191+
192+### Root Cause Identified
193+**The OpenCode Zen API returns an error when messages contain ONLY system messages (no user or assistant messages).**
194+
195+Error: `"Cannot read properties of undefined (reading 'input_tokens')"`
196+
197+This is an internal server error in the OpenCode Zen API, not a client-side issue.
198+
199+### API Behavior Summary
200+
201+| Message Pattern | Result | Notes |
202+|----------------|--------|-------|
203+| System + User | โœ… Works | Standard pattern |
204+| System + User + Tools | โœ… Works | Tools are fine |
205+| System + User + Assistant | โš ๏ธ Empty response | Valid but no content |
206+| System + Assistant | โš ๏ธ Weird output | Unexpected behavior |
207+| Only Assistant | โš ๏ธ Empty response | Valid but no content |
208+| **Only System** | โŒ **ERROR** | **Crashes with input_tokens error** |
209+| Multiple Tools (5+) | โœ… Works | Not a tool count issue |
210+| Full tool cycle | โœ… Works | Tool responses work fine |
211+
212+### Key Findings
213+
214+1. **System-only messages crash the API** - This is the issue we're hitting in Zoea Nova
215+2. **Tool count is not the issue** - 5+ tools work fine
216+3. **Tool format is not the issue** - Complex tool schemas work fine
217+4. **Assistant without user is accepted** - But produces empty or strange output
218+5. **Empty responses are valid** - Missing content field is OpenAI-spec compliant
219+
220+### Implications for Zoea Nova
221+
222+The error we see when sending only system messages is **not a bug in our code** - it's an **OpenCode Zen API limitation**.
223+
224+**Our current fallback logic is correct:**
225+```go
226+// If only system messages remain, add a dummy user message
227+if allMessagesAreSystem {
228+ messages = append(messages, openAIMessage{
229+ Role: "user",
230+ Content: "Continue.",
231+ })
232+}
233+```
234+
235+This workaround is necessary because the OpenCode Zen API cannot handle system-only requests.
236+
237+### Recommendations
238+
239+1. **Keep the fallback logic** - It's working as intended to prevent API crashes
240+2. **Consider logging when fallback is used** - For debugging/monitoring
241+3. **Document this API limitation** - In OPENCODE_ZEN_MODELS.md or KNOWN_ISSUES.md
242+4. **Consider reporting to OpenCode** - This appears to be a server-side bug
243+
244+### Related Files
245+- `internal/provider/opencode.go` - Contains fallback logic
246+- `internal/provider/openai_common.go` - Shared message validation
247+- `documentation/guides/OPENCODE_ZEN_MODELS.md` - Provider guide
documentation/investigations/OPENCODE_ZEN_BUG_VERDICT_2026-02-06.md+181โˆ’0
@@ -0,0 +1,181 @@
1+# OpenCode Zen API Bug Verdict - 2026-02-06
2+
3+## Question
4+Is the "Cannot read properties of undefined (reading 'input_tokens'/'prompt_tokens')" error an OpenCode Zen API bug or a Zoea Nova bug?
5+
6+## Verdict: **OPENCODE ZEN API BUG**
7+
8+## Evidence
9+
10+### Test 1: Minimal Valid Request (System + User)
11+**Request:**
12+```bash
13+curl -X POST https://opencode.ai/zen/v1/chat/completions \
14+ -H "Content-Type: application/json" \
15+ -H "Authorization: Bearer ${OPENCODE_API_KEY}" \
16+ -d '{
17+ "model": "glm-4.7-free",
18+ "messages": [
19+ {"role": "system", "content": "Test"},
20+ {"role": "user", "content": "Hello"}
21+ ],
22+ "stream": false
23+ }'
24+```
25+
26+**Result:** โœ… SUCCESS
27+- Returns valid response with content
28+- Usage tokens reported correctly
29+- No errors
30+
31+---
32+
33+### Test 2: System-Only Request (Reproduces Our Error)
34+**Request:**
35+```bash
36+curl -X POST https://opencode.ai/zen/v1/chat/completions \
37+ -H "Content-Type: application/json" \
38+ -H "Authorization: Bearer ${OPENCODE_API_KEY}" \
39+ -d '{
40+ "model": "glm-4.7-free",
41+ "messages": [
42+ {"role": "system", "content": "You are a helpful assistant."}
43+ ],
44+ "stream": false
45+ }'
46+```
47+
48+**Result:** โŒ ERROR
49+```json
50+{
51+ "type": "error",
52+ "error": {
53+ "type": "error",
54+ "message": "Cannot read properties of undefined (reading 'prompt_tokens')"
55+ }
56+}
57+```
58+
59+**This is the EXACT error we see in Zoea Nova when sending system-only messages.**
60+
61+---
62+
63+### Test 3: Known OpenCode Zen Issue
64+Found related issue in OpenCode Zen GitHub:
65+
66+**Issue #8228: OpenCode Zen Gemini Integration - 500 Internal Server Error**
67+- URL: https://github.com/anomalyco/opencode/issues/8228
68+- Opened: Jan 13, 2026
69+- Status: OPEN
70+- Error: `"Cannot read properties of undefined (reading 'promptTokenCount')"`
71+- Same pattern: OpenCode Zen backend fails to parse usage/token metadata
72+- Affects Gemini models specifically, but demonstrates systemic issue with token counting
73+
74+---
75+
76+### Test 4: Message Pattern Validation (From Previous Investigation)
77+
78+| Message Pattern | OpenCode Zen Result |
79+|----------------|-------------------|
80+| System + User | โœ… Works |
81+| System + User + Tools | โœ… Works |
82+| **System only** | โŒ **500 Error** |
83+| System + Assistant | โš ๏ธ Weird output |
84+| Only Assistant | โš ๏ธ Empty response |
85+
86+**Source:** `documentation/investigations/OPENCODE_ZEN_API_TESTS_2026-02-06.md`
87+
88+---
89+
90+## Root Cause Analysis
91+
92+### OpenCode Zen API Bug
93+The OpenCode Zen API has a server-side bug where:
94+
95+1. **System-only messages crash the token counting logic**
96+ - Error: `"Cannot read properties of undefined (reading 'prompt_tokens')"`
97+ - The API attempts to read token counts from an undefined object
98+ - This is an unhandled null/undefined case in their response parsing
99+
100+2. **Inconsistent token metadata handling**
101+ - Related issue #8228 shows same pattern with Gemini: `"Cannot read properties of undefined (reading 'promptTokenCount')"`
102+ - Suggests systemic issue in how OpenCode Zen wraps/transforms upstream API responses
103+ - Token counting fails when response metadata is missing or in unexpected format
104+
105+3. **No graceful degradation**
106+ - Returns 500 Internal Server Error instead of handling edge cases
107+ - Should either return partial response or meaningful error message
108+ - Breaking on system-only messages is not OpenAI-spec compliant
109+
110+### Our Workaround is Correct
111+**File:** `internal/provider/openai_common.go`
112+
113+```go
114+// If only system messages remain, add a dummy user message
115+if allMessagesAreSystem {
116+ messages = append(messages, openAIMessage{
117+ Role: "user",
118+ Content: "Continue.",
119+ })
120+}
121+```
122+
123+**This workaround is necessary and correct** because:
124+- It prevents hitting the OpenCode Zen API bug
125+- It's a defensive measure against their server-side issue
126+- It follows OpenAI best practices (conversations should have user input)
127+- It's documented in our codebase as a known limitation
128+
129+---
130+
131+## Comparison to OpenAI Spec
132+
133+According to OpenAI's official API:
134+- System-only messages are technically valid (though discouraged)
135+- Should not crash with 500 errors
136+- OpenAI's API handles this gracefully (returns empty response or asks for user input)
137+
138+OpenCode Zen is **not fully OpenAI-compatible** in this edge case.
139+
140+---
141+
142+## Implications for Zoea Nova
143+
144+### What We Did Right
145+1. โœ… Implemented fallback logic for system-only messages
146+2. โœ… Added validation to ensure messages are OpenAI-spec compliant
147+3. โœ… Documented the limitation in investigation reports
148+4. โœ… Our fix prevents hitting the OpenCode Zen bug
149+
150+### What We Should Do
151+1. **Keep the workaround** - It's protecting us from an upstream bug
152+2. **Document as known OpenCode Zen limitation** - Update `KNOWN_SERVER_ISSUES.md`
153+3. **Consider reporting to OpenCode** - Issue #8228 shows they're aware of token counting bugs
154+4. **Add test coverage** - Ensure our fallback logic is tested (already exists)
155+
156+---
157+
158+## Conclusion
159+
160+**THIS IS AN OPENCODE ZEN API BUG, NOT A ZOEA NOVA BUG.**
161+
162+Our fallback logic is a correct defensive measure against their server-side issue. We should:
163+1. Keep the workaround in place
164+2. Document it as a known OpenCode Zen limitation
165+3. Monitor OpenCode Zen issue tracker for fixes
166+4. Consider reporting if not already documented
167+
168+---
169+
170+## Related Files
171+- `internal/provider/opencode.go` - Our OpenCode provider implementation
172+- `internal/provider/openai_common.go` - Contains fallback logic
173+- `documentation/investigations/OPENCODE_ZEN_API_TESTS_2026-02-06.md` - Detailed test results
174+- `documentation/reports/OPENCODE_ZEN_FIX_2026-02-07.md` - Our fix implementation
175+
176+---
177+
178+## External References
179+- OpenCode Issue #8228: https://github.com/anomalyco/opencode/issues/8228
180+- OpenCode Zen Docs: https://opencode.ai/docs/zen/
181+- OpenCode Zen Models API: https://opencode.ai/zen/v1/models
documentation/investigations/README.md+17โˆ’2
@@ -1,6 +1,21 @@
1-# Historical Investigations
1+# Active and Historical Investigations
22
3-This folder contains investigation reports for issues that have since been resolved. These documents are preserved for historical context and understanding of the debugging process.
3+This folder contains investigation reports for issues both active and resolved. Active investigations document ongoing research, while resolved issues are preserved for historical context and understanding of the debugging process.
4+
5+## Active Investigations
6+
7+### OpenCode Zen API System-Only Messages Bug
8+
9+**Issue:** OpenCode Zen API returns `"Cannot read properties of undefined (reading 'input_tokens')"` error when messages contain only system messages (no user or assistant turns).
10+
11+**Investigation Documents:**
12+- `OPENCODE_ZEN_API_TESTS_2026-02-06.md` - Direct curl testing proving system-only requests crash
13+
14+**Status:** Confirmed as API limitation. Workaround implemented in `internal/provider/opencode.go` (fallback to add dummy user message).
15+
16+**Next Steps:** Consider reporting to OpenCode team.
17+
18+---
419
520 ## Resolved Issues