@@ -0,0 +1,500 @@
1+# ClaimAccount() Race Condition Analysis
2+
3+**Date:** 2026-02-07
4+**Context:** Analysis of edge cases and race conditions in the ClaimAccount() fix
5+**Key Question:** Is the simple fix (no locking) safe, or do we need a claim-lock mechanism?
6+
7+---
8+
9+## Executive Summary
10+
11+**Recommendation: The simple fix is SAFE. No claim-lock mechanism needed.**
12+
13+The current ClaimAccount() implementation using optimistic locking via `UPDATE ... WHERE in_use = 0` with retry logic is sufficient. The database-level concurrency controls (MaxOpenConns=1, WAL mode, busy_timeout=5000ms) combined with the check-and-set pattern provide adequate safety for all identified edge cases.
14+
15+---
16+
17+## Current Implementation
18+
19+### Account State Transitions
20+
21+```
22+[Available: in_use=0]
23+ ↓ ClaimAccount() - optimistic lock via UPDATE WHERE in_use=0
24+[Claimed: in_use=1]
25+ ↓ MCP login tool call
26+[Logged In: in_use=1] (MarkAccountInUse - idempotent)
27+ ↓ MCP logout tool call OR Mysis.Stop() OR Mysis.setError()
28+[Released: in_use=0] (ReleaseAccount)
29+```
30+
31+### Key Code Paths
32+
33+**ClaimAccount()** (`internal/store/accounts.go:87-132`):
34+```go
35+for attempts := 0; attempts < 5; attempts++ {
36+ // 1. SELECT first available account (WHERE in_use = 0)
37+ var username, password string
38+ err := s.db.QueryRow(`SELECT ... WHERE in_use = 0 LIMIT 1`)
39+
40+ // 2. Optimistic lock: UPDATE only if still available
41+ result := s.db.Exec(`UPDATE accounts SET in_use = 1 WHERE username = ? AND in_use = 0`)
42+
43+ // 3. Check if UPDATE succeeded
44+ rows, _ := result.RowsAffected()
45+ if rows == 0 {
46+ continue // Another thread won the race, retry
47+ }
48+
49+ return account
50+}
51+```
52+
53+**MarkAccountInUse()** (`internal/store/accounts.go:134-147`):
54+- Called by MCP proxy on successful login
55+- Idempotent: sets `in_use=1` without conditions
56+- Safe even if account already claimed
57+
58+**ReleaseAccount()** (`internal/store/accounts.go:149-160`):
59+- Called by MCP proxy on logout
60+- Called by Mysis.releaseCurrentAccount() on Stop/setError
61+- Sets `in_use=0` without conditions
62+
63+### Database Concurrency Controls
64+
65+**From** `internal/store/store.go:38-56`:
66+```go
67+dsn += "?_busy_timeout=5000" // 5 second busy timeout
68+db.SetMaxOpenConns(1) // Serialize all DB access
69+PRAGMA journal_mode=WAL // Write-Ahead Log for read concurrency
70+```
71+
72+**Critical insight:** `SetMaxOpenConns(1)` means **all database operations are serialized through a single connection**. This eliminates true race conditions at the SQL execution level.
73+
74+---
75+
76+## Edge Case Analysis
77+
78+### Scenario 1: Multiple Myses Call ClaimAccount() Simultaneously
79+
80+**Flow:**
81+```
82+Time Mysis A Mysis B Database
83+---- ------- ------- --------
84+T1 ClaimAccount() [waiting for DB conn] in_use=0
85+T2 SELECT crab_01 [waiting for DB conn] in_use=0
86+T3 UPDATE ... in_use=1 [waiting for DB conn] in_use=1
87+T4 return crab_01 ClaimAccount() in_use=1
88+T5 SELECT crab_01 in_use=1
89+T6 UPDATE (rows=0) in_use=1
90+T7 continue retry in_use=1
91+T8 SELECT crab_02 in_use=0
92+T9 UPDATE ... in_use=1 in_use=1
93+T10 return crab_02 in_use=1
94+```
95+
96+**Result:** ✅ SAFE
97+- MaxOpenConns=1 serializes all DB operations
98+- Mysis B's UPDATE fails (rows=0) because Mysis A already claimed crab_01
99+- Retry logic succeeds on next available account
100+- No double-claim possible
101+
102+**Test Coverage:** `internal/store/store_test.go:320-350` (TestReleaseAllAccounts with sequential claims)
103+
104+---
105+
106+### Scenario 2: Mysis Claims Account But Never Logs In
107+
108+**Flow:**
109+```
110+Time Mysis A MCP Tool Database
111+---- ------- -------- --------
112+T1 ClaimAccount() - in_use=0
113+T2 UPDATE ... in_use=1 - in_use=1
114+T3 return crab_01 - in_use=1
115+T4 [LLM ignores credentials] - in_use=1
116+T5 [never calls login tool] - in_use=1
117+... (account stuck in_use=1 indefinitely)
118+```
119+
120+**Result:** ⚠️ KNOWN LIMITATION (acceptable)
121+- Account remains locked until Mysis is stopped/errored
122+- This is **by design**: claimed accounts are reserved for the claiming Mysis
123+- Mitigation: Mysis lifecycle management (Stop/Error) releases account
124+- Alternative fix: Add timeout-based auto-release (NOT RECOMMENDED - adds complexity)
125+
126+**Design Decision:** Accept this limitation. Account claiming is a resource allocation, not a transaction. The Mysis "owns" the account until released.
127+
128+**User Impact:** If a Mysis claims but never uses an account, operator must restart/delete the Mysis to free the account. This is acceptable operational overhead.
129+
130+---
131+
132+### Scenario 3: Mysis Crashes After Claim But Before Login
133+
134+**Flow:**
135+```
136+Time Mysis A Process Database
137+---- ------- ------- --------
138+T1 ClaimAccount() running in_use=0
139+T2 UPDATE ... in_use=1 running in_use=1
140+T3 return crab_01 running in_use=1
141+T4 [process killed] SIGKILL in_use=1
142+T5 [no cleanup runs] dead in_use=1
143+... (account stuck in_use=1 until manual intervention)
144+```
145+
146+**Result:** ⚠️ KNOWN LIMITATION (acceptable)
147+- Same as Scenario 2: account remains locked
148+- Zoea Nova has no crash recovery mechanism for account locks
149+- Mitigation: Application restart calls `ReleaseAllAccounts()` on startup (NOT CURRENTLY IMPLEMENTED)
150+
151+**Recommendation:** Add `ReleaseAllAccounts()` call to main.go initialization:
152+```go
153+func main() {
154+ store, _ := store.New()
155+ defer store.Close()
156+
157+ // Release any accounts locked by previous run
158+ if err := store.ReleaseAllAccounts(); err != nil {
159+ log.Warn().Err(err).Msg("failed to release accounts on startup")
160+ }
161+
162+ // ... continue normal startup
163+}
164+```
165+
166+**Impact:** Low priority. Workaround is `make db-reset-accounts` or manual SQL.
167+
168+---
169+
170+### Scenario 4: Race Between ClaimAccount() and MarkAccountInUse()
171+
172+**Flow:**
173+```
174+Time Mysis A (goroutine 1) MCP Proxy (goroutine 2) Database
175+---- ---------------------- ----------------------- --------
176+T1 ClaimAccount() - in_use=0
177+T2 UPDATE ... in_use=1 - in_use=1
178+T3 return crab_01 - in_use=1
179+T4 [calls login tool] CallTool("login", ...) in_use=1
180+T5 - MarkAccountInUse("crab_01") in_use=1
181+T6 - UPDATE ... in_use=1 in_use=1 (no-op)
182+```
183+
184+**Result:** ✅ SAFE
185+- MarkAccountInUse() is **idempotent** - setting `in_use=1` when already `in_use=1` is harmless
186+- No race condition because both operations set the same value
187+- MaxOpenConns=1 serializes operations anyway
188+
189+**Edge case:** What if another Mysis claims the account between T3 and T5?
190+
191+**Flow (contended):**
192+```
193+Time Mysis A Mysis B Database
194+---- ------- ------- --------
195+T1 ClaimAccount() - in_use=0
196+T2 UPDATE ... in_use=1 - in_use=1
197+T3 return crab_01 - in_use=1
198+T4 [paused by scheduler] ClaimAccount() in_use=1
199+T5 [paused] SELECT (finds no in_use=0) in_use=1
200+T6 [paused] return error in_use=1
201+T7 [calls login tool] - in_use=1
202+T8 MarkAccountInUse("crab_01") - in_use=1
203+```
204+
205+**Result:** ✅ SAFE
206+- Mysis B cannot claim crab_01 because it's already `in_use=1` at T5
207+- Mysis A's login succeeds without interference
208+- No double-claim possible
209+
210+---
211+
212+## Concurrency Model Analysis
213+
214+### SQLite Configuration
215+
216+| Setting | Value | Impact on ClaimAccount() |
217+|---------|-------|-------------------------|
218+| `MaxOpenConns` | 1 | All DB operations serialized - eliminates true races |
219+| `_busy_timeout` | 5000ms | Goroutines wait up to 5s for DB lock - prevents lock errors |
220+| `journal_mode` | WAL | Allows concurrent reads, but writes still serialized by MaxOpenConns |
221+
222+### Critical Insight: MaxOpenConns=1 Is The Safety Mechanism
223+
224+The `SetMaxOpenConns(1)` setting means:
225+1. All SQL statements execute serially (no true parallelism)
226+2. `ClaimAccount()` calls are queued by the database/sql connection pool
227+3. The `UPDATE ... WHERE in_use = 0` pattern becomes a serialized atomic operation
228+
229+**Without MaxOpenConns=1:**
230+- SELECT and UPDATE could interleave (classic TOCTOU race)
231+- Multiple threads could SELECT the same account
232+- Multiple threads could UPDATE the same account to in_use=1
233+
234+**With MaxOpenConns=1:**
235+- SELECT and UPDATE execute atomically (no other thread can execute SQL between them)
236+- Retry loop handles "lost updates" gracefully (rows=0 detection)
237+
238+### Why Optimistic Locking Works Here
239+
240+**Optimistic locking pattern:**
241+```sql
242+-- Thread 1
243+SELECT username FROM accounts WHERE in_use = 0 LIMIT 1; -- Returns crab_01
244+UPDATE accounts SET in_use = 1 WHERE username = 'crab_01' AND in_use = 0; -- Succeeds
245+
246+-- Thread 2 (after Thread 1's UPDATE)
247+SELECT username FROM accounts WHERE in_use = 0 LIMIT 1; -- Returns crab_02 (or crab_01 if re-released)
248+UPDATE accounts SET in_use = 1 WHERE username = 'crab_01' AND in_use = 0; -- Fails (rows=0)
249+```
250+
251+The `AND in_use = 0` condition is the optimistic lock. If another thread claimed the account between SELECT and UPDATE, the UPDATE fails without side effects.
252+
253+**Retry logic** (5 attempts) handles transient contention:
254+- If 10 Myses start simultaneously and only 1 account exists, 9 will retry
255+- Each retry re-runs SELECT to find next available account
256+- Worst case: all 5 retries fail → return "no accounts available"
257+
258+---
259+
260+## ListAvailableAccounts() Safety
261+
262+**Function:** `internal/store/accounts.go:56-85`
263+
264+```go
265+func (s *Store) ListAvailableAccounts() ([]*Account, error) {
266+ rows := s.db.Query(`SELECT ... WHERE in_use = 0 ORDER BY created_at ASC`)
267+ // ... scan and return
268+}
269+```
270+
271+**Race condition analysis:**
272+
273+| Scenario | Safety |
274+|----------|--------|
275+| List accounts while another Mysis claims one | ✅ SAFE - snapshot read (WAL mode) |
276+| List accounts immediately after ClaimAccount() | ✅ SAFE - MaxOpenConns=1 serializes operations |
277+| List accounts during login tool call | ✅ SAFE - reads don't block/interfere with writes |
278+
279+**Result:** No changes needed to ListAvailableAccounts().
280+
281+---
282+
283+## ReleaseAccount() Safety
284+
285+**Function:** `internal/store/accounts.go:149-160`
286+
287+```go
288+func (s *Store) ReleaseAccount(username string) error {
289+ _, err := s.db.Exec(`UPDATE accounts SET in_use = 0 WHERE username = ?`)
290+}
291+```
292+
293+**Race condition analysis:**
294+
295+| Scenario | Safety |
296+|----------|--------|
297+| Release account while another Mysis claims it | ✅ SAFE - MaxOpenConns=1 serializes operations |
298+| Release account twice (double-release) | ✅ SAFE - idempotent (setting in_use=0 when already 0 is harmless) |
299+| Release account during ClaimAccount() retry loop | ✅ SAFE - account becomes available for next retry |
300+
301+**Edge case:** What if a Mysis releases an account it doesn't own?
302+
303+**Flow:**
304+```
305+Time Mysis A Mysis B Database
306+---- ------- ------- --------
307+T1 ClaimAccount() ClaimAccount() in_use=0
308+T2 return crab_01 return crab_02 in_use=1 (both)
309+T3 ReleaseAccount("crab_02") - in_use=0 (crab_02)
310+T4 - [still using crab_02] in_use=0 (crab_02)
311+T5 ClaimAccount() - in_use=0 (crab_02)
312+T6 return crab_02 - in_use=1 (crab_02)
313+T7 [double-claim!] [still using crab_02] in_use=1 (crab_02)
314+```
315+
316+**Result:** ⚠️ POTENTIAL BUG - but **NOT POSSIBLE** in current implementation
317+
318+**Why it's not possible:**
319+- Mysis tracks `currentAccountUsername` (internal/core/mysis.go:1084-1090)
320+- ReleaseAccount() only called with Mysis's own account (mysis.go:1093-1103)
321+- No API exists for one Mysis to release another's account
322+
323+**Mitigation (defense in depth):** Add ownership check to ReleaseAccount()
324+```go
325+func (m *Mysis) releaseCurrentAccount() {
326+ m.mu.Lock()
327+ username := m.currentAccountUsername
328+ m.currentAccountUsername = ""
329+ m.mu.Unlock()
330+
331+ if username != "" {
332+ // Only release account we actually own
333+ _ = m.store.ReleaseAccount(username)
334+ }
335+}
336+```
337+
338+**Current implementation already has this pattern** - no changes needed.
339+
340+---
341+
342+## Do We Need a "Claimed But Not Logged In" State?
343+
344+**Proposed 3-state model:**
345+```
346+[Available: in_use=0]
347+ ↓ ClaimAccount()
348+[Claimed: in_use=1, logged_in=0] ← NEW STATE
349+ ↓ MarkAccountInUse()
350+[Logged In: in_use=1, logged_in=1]
351+ ↓ ReleaseAccount()
352+[Released: in_use=0, logged_in=0]
353+```
354+
355+**Analysis:**
356+
357+| Benefit | Cost | Verdict |
358+|---------|------|---------|
359+| Distinguish claim vs login | Schema change (add logged_in column) | ❌ Not worth it |
360+| Timeout-based auto-release | Complex timeout tracking logic | ❌ Not worth it |
361+| Better observability | Minimal - current logs already show state | ❌ Not worth it |
362+
363+**Recommendation:** **NO** - Current 2-state model is sufficient.
364+
365+**Rationale:**
366+1. The claim→login window is typically <1 second (LLM tool call latency)
367+2. Failure modes (scenario 2 & 3) are rare and acceptable
368+3. Adding a third state increases complexity without meaningful benefit
369+4. Operator can always Stop/Delete Mysis to release stuck accounts
370+
371+---
372+
373+## Recommendations
374+
375+### 1. Keep Current Implementation ✅
376+
377+The simple fix (no locking in ClaimAccount, use optimistic locking) is **SAFE and CORRECT**.
378+
379+**No changes needed to:**
380+- ClaimAccount() - retry logic handles contention
381+- MarkAccountInUse() - idempotent, no race conditions
382+- ReleaseAccount() - idempotent, protected by Mysis ownership
383+- ListAvailableAccounts() - snapshot reads are safe
384+
385+### 2. Add Startup Account Release (Optional, Low Priority)
386+
387+**Add to** `cmd/zoea/main.go`:
388+```go
389+func main() {
390+ store, err := store.New()
391+ if err != nil {
392+ log.Fatal().Err(err).Msg("failed to open store")
393+ }
394+ defer store.Close()
395+
396+ // Release accounts locked by previous run (crash recovery)
397+ if err := store.ReleaseAllAccounts(); err != nil {
398+ log.Warn().Err(err).Msg("failed to release accounts on startup")
399+ }
400+
401+ // ... continue normal startup
402+}
403+```
404+
405+**Benefit:** Handles scenario 3 (crash recovery) gracefully
406+**Cost:** Minimal (1 SQL UPDATE at startup)
407+**Priority:** Low - crashes are rare, workaround exists
408+
409+### 3. Document Edge Cases (This Report)
410+
411+**Add to** `documentation/architecture/ACCOUNT_LIFECYCLE.md`:
412+- Document 2-state model (available → claimed)
413+- Document acceptable limitations (scenarios 2 & 3)
414+- Document operator recovery procedures
415+
416+### 4. No Claim-Lock Mechanism Needed ❌
417+
418+**Rejected alternatives:**
419+- Distributed lock (Redis, etc.) - Overkill for single-process app
420+- Mutex in Go code - Doesn't survive process restarts
421+- SQL transactions - Already have optimistic locking via WHERE clause
422+- Timeout-based auto-release - Complex, error-prone, unnecessary
423+
424+---
425+
426+## Test Coverage Assessment
427+
428+### Existing Tests
429+
430+✅ **Basic claim/release cycle:** `internal/store/store_test.go:274-318`
431+✅ **Multiple claims:** `internal/store/store_test.go:320-350`
432+✅ **Account release on error:** `internal/core/mysis_account_release_test.go:10-85`
433+
434+### Missing Tests (Recommended)
435+
436+❌ **Concurrent ClaimAccount() calls** (10 goroutines, 1 account)
437+❌ **ClaimAccount() during ReleaseAccount()** (interleaved ops)
438+❌ **MarkAccountInUse() idempotency** (call twice on same account)
439+
440+**Test plan:**
441+```go
442+func TestClaimAccount_Concurrent(t *testing.T) {
443+ s := setupStore(t)
444+ s.CreateAccount("crab_01", "pass")
445+
446+ var wg sync.WaitGroup
447+ claims := make(chan *Account, 10)
448+
449+ // 10 goroutines try to claim 1 account
450+ for i := 0; i < 10; i++ {
451+ wg.Add(1)
452+ go func() {
453+ defer wg.Done()
454+ acc, err := s.ClaimAccount()
455+ if err == nil {
456+ claims <- acc
457+ }
458+ }()
459+ }
460+
461+ wg.Wait()
462+ close(claims)
463+
464+ // Only 1 should succeed
465+ count := 0
466+ for range claims {
467+ count++
468+ }
469+
470+ if count != 1 {
471+ t.Errorf("expected 1 successful claim, got %d", count)
472+ }
473+}
474+```
475+
476+**Priority:** Medium - current implementation is safe, but test would validate assumptions.
477+
478+---
479+
480+## Conclusion
481+
482+**The simple fix is SAFE. No claim-lock mechanism needed.**
483+
484+**Key safety mechanisms:**
485+1. **MaxOpenConns=1** - Serializes all DB operations, eliminates true race conditions
486+2. **Optimistic locking** - `UPDATE ... WHERE in_use = 0` provides atomic check-and-set
487+3. **Retry logic** - 5 attempts handle transient contention gracefully
488+4. **Idempotent operations** - MarkAccountInUse and ReleaseAccount are safe to call multiple times
489+
490+**Acceptable limitations:**
491+1. Claimed but unused accounts remain locked until Mysis stopped (Scenario 2)
492+2. Crashed Mysis leaves accounts locked until manual intervention (Scenario 3)
493+
494+**Recommended follow-ups:**
495+1. ✅ Keep current implementation (no changes)
496+2. 🟡 Add ReleaseAllAccounts() to main.go startup (low priority)
497+3. 🟡 Add concurrent ClaimAccount() test (medium priority)
498+4. ✅ Document edge cases (this report)
499+
500+**Final verdict:** Ship it. The implementation is production-ready.