@@ -0,0 +1,635 @@
1+# Goroutine Cleanup Analysis - Exit Hang Investigation
2+
3+**Date:** 2026-02-06
4+**Issue:** Application hangs when trying to exit, terminal becomes unresponsive
5+**Root Cause:** Multiple goroutines not receiving shutdown signals properly
6+
7+---
8+
9+## Executive Summary
10+
11+The application has **5 distinct categories of goroutines** that are not being properly cleaned up on exit:
12+
13+1. **Mysis processing loops** - Each running mysis has a goroutine in `run()` that waits on channels
14+2. **Event bus subscriber** - TUI listens on event channel forever
15+3. **Signal handler goroutine** - Shutdown handler in main.go
16+4. **Provider streaming goroutines** - Ollama/OpenCode streaming responses
17+5. **Async message senders** - Short-lived goroutines spawned by Commander
18+
19+**Critical Finding:** The TUI's event listening goroutine (`listenForEvents()`) creates an **infinite loop** that never terminates because the event channel is never closed before `tea.Quit()` is called.
20+
21+---
22+
23+## Detailed Analysis
24+
25+### 1. Mysis Processing Loop (`internal/core/mysis.go:913-944`)
26+
27+**Location:** `Mysis.run(ctx context.Context)`
28+
29+```go
30+func (m *Mysis) run(ctx context.Context) {
31+ ticker := time.NewTicker(constants.IdleNudgeInterval)
32+ defer ticker.Stop()
33+
34+ for {
35+ select {
36+ case <-ctx.Done():
37+ return // ✅ CORRECT: Returns on context cancellation
38+ case <-ticker.C:
39+ // Nudge logic
40+ case <-a.nudgeCh:
41+ // Process nudge
42+ }
43+ }
44+}
45+```
46+
47+**Status:** ✅ **CORRECT** - Properly exits when context is canceled
48+
49+**How it's cleaned up:**
50+1. `Mysis.Stop()` (line 225) calls `a.cancel()` which cancels the context
51+2. `run()` receives signal on `<-ctx.Done()` and returns
52+3. `commander.StopAll()` (main.go:166) calls `Stop()` on all myses
53+
54+**Verification:**
55+- Context is created in `Mysis.Start()` (line 191): `ctx, cancel := context.WithCancel(context.Background())`
56+- Goroutine is launched (line 216): `go a.run(ctx)`
57+- Cleanup waits for turn to complete (line 239-240): `a.turnMu.Lock()` ensures current operation finishes
58+
59+**Potential Issue:** If a mysis is in the middle of a long-running `SendMessage()` call (which can take time for LLM processing), the `Stop()` will block waiting for `turnMu.Lock()`. This could cause **delays but not hangs** since LLM calls have timeouts.
60+
61+---
62+
63+### 2. Event Bus Subscriber Loop (`internal/tui/app.go:862-870`)
64+
65+**Location:** `Model.listenForEvents()`
66+
67+```go
68+func (m Model) listenForEvents() tea.Cmd {
69+ return func() tea.Msg {
70+ event, ok := <-m.eventCh
71+ if !ok {
72+ return nil // Channel closed
73+ }
74+ return EventMsg{Event: event}
75+ }
76+}
77+```
78+
79+**Status:** ⚠️ **POTENTIAL HANG** - Depends on event channel being closed
80+
81+**How it's supposed to work:**
82+1. `bus.Subscribe()` (main.go:146) returns a buffered channel
83+2. TUI reads from channel in `listenForEvents()`
84+3. On each event, it calls `listenForEvents()` again (line 191)
85+4. This creates a **recursive event loop**
86+
87+**The Problem:**
88+- `bus.Close()` (main.go:81) is deferred and **runs after `program.Run()` returns**
89+- `program.Run()` (main.go:161) **blocks until TUI exits**
90+- TUI is waiting for events from `eventCh`
91+- **Deadlock potential:** TUI waits for event → program.Run() blocks → bus.Close() never called
92+
93+**Shutdown sequence (main.go:141-173):**
94+```go
95+sigCh := make(chan os.Signal, 1)
96+signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
97+
98+eventCh := bus.Subscribe()
99+model := tui.New(commander, s, eventCh)
100+program := tea.NewProgram(model, tea.WithAltScreen())
101+
102+go func() {
103+ <-sigCh
104+ log.Info().Msg("Received shutdown signal")
105+ commander.StopAll() // ← This stops myses
106+ program.Quit() // ← This should exit TUI
107+}()
108+
109+if _, err := program.Run(); err != nil { // ← BLOCKS HERE
110+ log.Fatal().Err(err).Msg("TUI error")
111+}
112+
113+commander.StopAll() // Redundant
114+```
115+
116+**The Race Condition:**
117+1. User presses `q` in TUI → `keys.Quit` matched (app.go:171) → `return m, tea.Quit`
118+2. Bubble Tea starts shutdown
119+3. TUI's `Update()` loop exits
120+4. **BUT** `listenForEvents()` command is still waiting on `<-m.eventCh`
121+5. Event channel won't close until `bus.Close()` is called
122+6. `bus.Close()` is deferred until after `program.Run()` returns
123+7. `program.Run()` won't return until all commands complete
124+8. **HANG:** Waiting for event that will never come
125+
126+---
127+
128+### 3. Signal Handler Goroutine (`main.go:152-158`)
129+
130+**Location:** `go func()` in main
131+
132+```go
133+go func() {
134+ <-sigCh
135+ log.Info().Msg("Received shutdown signal")
136+ commander.StopAll()
137+ program.Quit()
138+}()
139+```
140+
141+**Status:** ✅ **CORRECT** - Will exit when sigCh receives signal
142+
143+**How it works:**
144+1. Goroutine blocks on `<-sigCh`
145+2. When SIGINT/SIGTERM received, calls `program.Quit()`
146+3. Goroutine exits after calling `Quit()`
147+
148+**Verification:** This is a standard pattern and should work correctly.
149+
150+---
151+
152+### 4. Provider Streaming Goroutines
153+
154+**Ollama Location:** `internal/provider/ollama.go:240`
155+**OpenCode Location:** `internal/provider/opencode.go:203`
156+
157+```go
158+// Ollama
159+go func() {
160+ defer close(streamCh)
161+ for {
162+ response, err := stream.Recv()
163+ if err == io.EOF {
164+ return
165+ }
166+ if err != nil {
167+ streamCh <- streamResult{err: err}
168+ return
169+ }
170+ streamCh <- streamResult{chunk: response}
171+ }
172+}()
173+```
174+
175+**Status:** ✅ **CORRECT** - Returns on EOF or error
176+
177+**How cleanup works:**
178+1. These goroutines are spawned during LLM streaming calls
179+2. They exit naturally when stream ends (EOF) or errors
180+3. Context timeout in `SendMessage()` (mysis.go:312) ensures they don't block forever
181+4. Short-lived - only exist during active LLM calls
182+
183+**Verification:** These follow standard streaming patterns and have proper cleanup.
184+
185+---
186+
187+### 5. Async Message Sender Goroutines
188+
189+**Location:** `internal/core/commander.go:266-271`
190+
191+```go
192+func (c *Commander) SendMessageAsync(id, content string) error {
193+ // ...validation...
194+ go func() {
195+ if err := mysis.SendMessage(content, store.MemorySourceDirect); err != nil {
196+ // Error is published to bus by mysis.SendMessage
197+ }
198+ }()
199+ return nil
200+}
201+```
202+
203+**Status:** ✅ **CORRECT** - Fire-and-forget pattern
204+
205+**How cleanup works:**
206+1. These are spawned for async operations
207+2. They complete and exit naturally
208+3. No cleanup needed - they're not long-lived
209+
210+**Verification:** Standard async pattern, no issues.
211+
212+---
213+
214+## Root Cause Analysis
215+
216+### Primary Issue: Event Bus Cleanup Order
217+
218+**The Bug:**
219+```go
220+// main.go
221+defer bus.Close() // Line 81 - Deferred
222+// ...
223+program := tea.NewProgram(model, tea.WithAltScreen())
224+// ...
225+if _, err := program.Run(); err != nil { // Blocks until TUI exits
226+ log.Fatal().Err(err).Msg("TUI error")
227+}
228+// defer bus.Close() executes here (AFTER program.Run returns)
229+```
230+
231+**The TUI Event Loop:**
232+```go
233+// app.go:862
234+func (m Model) listenForEvents() tea.Cmd {
235+ return func() tea.Msg {
236+ event, ok := <-m.eventCh // ← BLOCKS FOREVER if channel not closed
237+ if !ok {
238+ return nil
239+ }
240+ return EventMsg{Event: event}
241+ }
242+}
243+
244+// app.go:191
245+case EventMsg:
246+ m.handleEvent(msg.Event)
247+ return m, m.listenForEvents() // ← Re-registers listener
248+```
249+
250+**Why it hangs:**
251+
252+1. TUI calls `listenForEvents()` which returns a command
253+2. Bubble Tea runtime executes command in background
254+3. Command blocks on `<-m.eventCh`
255+4. User presses `q` → TUI returns `tea.Quit`
256+5. Bubble Tea tries to shut down
257+6. **BUT** the `listenForEvents` command is still blocked waiting for an event
258+7. `program.Run()` won't return until all pending commands complete
259+8. Event channel won't be closed until `bus.Close()` runs
260+9. `bus.Close()` won't run until `program.Run()` returns
261+10. **DEADLOCK**
262+
263+---
264+
265+## Secondary Issues
266+
267+### Issue 2A: Redundant StopAll Call
268+
269+```go
270+// main.go:152-158
271+go func() {
272+ <-sigCh
273+ log.Info().Msg("Received shutdown signal")
274+ commander.StopAll() // Called here
275+ program.Quit()
276+}()
277+
278+// main.go:166 (after program.Run returns)
279+commander.StopAll() // Called again (redundant)
280+```
281+
282+**Impact:** Minor - `StopAll()` is idempotent (checks state before stopping), but wasteful.
283+
284+### Issue 2B: No Wait for Goroutines
285+
286+The shutdown sequence doesn't wait for goroutines to complete:
287+
288+```go
289+commander.StopAll() // Initiates stop
290+// No wait here - proceeds immediately
291+if err := s.ReleaseAllAccounts(); err != nil { // Might race with myses still stopping
292+ log.Warn().Err(err).Msg("Failed to release accounts on shutdown")
293+}
294+```
295+
296+**Impact:** Myses might still be finishing their turns when accounts are released.
297+
298+---
299+
300+## Solutions
301+
302+### Solution 1: Close Event Bus Before Waiting for TUI (Recommended)
303+
304+**Change shutdown sequence to close bus first:**
305+
306+```go
307+// main.go (current)
308+defer bus.Close() // BAD: Runs after program.Run() returns
309+
310+// main.go (fixed)
311+defer func() {
312+ bus.Close() // Close before waiting for program
313+ s.Close() // Then close store
314+}()
315+```
316+
317+**Better approach - explicit cleanup:**
318+
319+```go
320+// main.go:152-158 (in signal handler)
321+go func() {
322+ <-sigCh
323+ log.Info().Msg("Received shutdown signal")
324+ commander.StopAll()
325+ bus.Close() // ← Close bus to unblock TUI event listener
326+ program.Quit() // ← Then quit TUI
327+}()
328+
329+// main.go:171-173 (after program.Run)
330+commander.StopAll() // Ensure all myses stopped
331+bus.Close() // Ensure bus closed (idempotent if already closed)
332+if err := s.ReleaseAllAccounts(); err != nil {
333+ log.Warn().Err(err).Msg("Failed to release accounts on shutdown")
334+}
335+s.Close()
336+```
337+
338+**Why this works:**
339+1. Signal handler closes bus **before** calling `program.Quit()`
340+2. Closing bus makes `<-m.eventCh` return immediately with `ok = false`
341+3. `listenForEvents()` returns `nil` → command completes
342+4. Bubble Tea can now exit cleanly
343+5. `program.Run()` returns
344+6. Cleanup proceeds normally
345+
346+---
347+
348+### Solution 2: Add Context to TUI Event Listener (More Robust)
349+
350+**Modify TUI to accept a context:**
351+
352+```go
353+// tui/app.go
354+type Model struct {
355+ // ...existing fields...
356+ ctx context.Context // Add shutdown context
357+}
358+
359+func New(commander *core.Commander, s *store.Store, eventCh <-chan core.Event, ctx context.Context) Model {
360+ return Model{
361+ // ...existing initialization...
362+ ctx: ctx,
363+ }
364+}
365+
366+func (m Model) listenForEvents() tea.Cmd {
367+ return func() tea.Msg {
368+ select {
369+ case event, ok := <-m.eventCh:
370+ if !ok {
371+ return nil
372+ }
373+ return EventMsg{Event: event}
374+ case <-m.ctx.Done(): // ← Exit on context cancellation
375+ return nil
376+ }
377+ }
378+}
379+```
380+
381+**Main.go changes:**
382+
383+```go
384+// main.go
385+ctx, cancel := context.WithCancel(context.Background())
386+defer cancel()
387+
388+eventCh := bus.Subscribe()
389+model := tui.New(commander, s, eventCh, ctx) // Pass context
390+program := tea.NewProgram(model, tea.WithAltScreen())
391+
392+go func() {
393+ <-sigCh
394+ log.Info().Msg("Received shutdown signal")
395+ commander.StopAll()
396+ cancel() // Cancel context to unblock event listener
397+ program.Quit()
398+}()
399+```
400+
401+**Why this is more robust:**
402+- Event listener can exit even if bus isn't closed
403+- Provides explicit shutdown signal
404+- Follows Go idioms for goroutine cleanup
405+
406+---
407+
408+### Solution 3: Make Bus.Close() Idempotent and Call Earlier
409+
410+**Modify bus.Close() to allow multiple calls:**
411+
412+```go
413+// internal/core/bus.go:179
414+func (b *EventBus) Close() {
415+ b.mu.Lock()
416+ defer b.mu.Unlock()
417+
418+ if b.subscribers == nil { // Already closed
419+ return
420+ }
421+
422+ for _, ch := range b.subscribers {
423+ ch.close()
424+ }
425+ b.subscribers = nil
426+}
427+```
428+
429+**Status:** Already appears to be idempotent (line 185 sets subscribers to nil).
430+
431+**Update main.go:**
432+
433+```go
434+// Call Close() explicitly before waiting for TUI
435+go func() {
436+ <-sigCh
437+ log.Info().Msg("Received shutdown signal")
438+ commander.StopAll()
439+ bus.Close() // Close here
440+ program.Quit()
441+}()
442+
443+// Also close on normal quit
444+case key.Matches(msg, keys.Quit):
445+ bus.Close() // Close bus before quit
446+ return m, tea.Quit
447+```
448+
449+**Issue:** This requires accessing `bus` from inside TUI, which breaks encapsulation.
450+
451+---
452+
453+## Recommended Fix
454+
455+**Implement Solution 1 + Solution 2 hybrid:**
456+
457+### Step 1: Close bus in signal handler
458+
459+```go
460+// main.go:152-158
461+go func() {
462+ <-sigCh
463+ log.Info().Msg("Received shutdown signal")
464+ commander.StopAll()
465+ bus.Close() // Close bus to unblock TUI
466+ program.Quit()
467+}()
468+```
469+
470+### Step 2: Close bus on normal quit
471+
472+**Add a cleanup callback to TUI:**
473+
474+```go
475+// tui/app.go
476+type Model struct {
477+ // ...existing fields...
478+ onQuit func() // Cleanup callback
479+}
480+
481+func New(commander *core.Commander, s *store.Store, eventCh <-chan core.Event) Model {
482+ return Model{
483+ // ...existing...
484+ onQuit: nil, // Set by main
485+ }
486+}
487+
488+func (m Model) SetQuitCallback(cb func()) {
489+ m.onQuit = cb
490+}
491+
492+// In Update() when handling quit
493+case key.Matches(msg, keys.Quit):
494+ if m.onQuit != nil {
495+ m.onQuit() // Call cleanup before quit
496+ }
497+ return m, tea.Quit
498+```
499+
500+**Main.go:**
501+
502+```go
503+model := tui.New(commander, s, eventCh)
504+model.SetQuitCallback(func() {
505+ commander.StopAll()
506+ bus.Close()
507+})
508+program := tea.NewProgram(model, tea.WithAltScreen())
509+```
510+
511+---
512+
513+## Testing Strategy
514+
515+### Test 1: Normal Exit (Press 'q')
516+
517+**Expected behavior:**
518+1. TUI receives quit key
519+2. Calls `commander.StopAll()`
520+3. Calls `bus.Close()` to unblock event listener
521+4. Returns `tea.Quit`
522+5. Bubble Tea exits cleanly
523+6. `program.Run()` returns
524+7. Cleanup code runs
525+8. Application exits
526+
527+**How to test:**
528+```bash
529+./bin/zoea --offline
530+# Press 'q'
531+# Should exit immediately (< 1 second)
532+```
533+
534+### Test 2: Signal Exit (Ctrl+C)
535+
536+**Expected behavior:**
537+1. Signal handler receives SIGINT
538+2. Calls `commander.StopAll()`
539+3. Calls `bus.Close()`
540+4. Calls `program.Quit()`
541+5. Same as Test 1 from step 5 onward
542+
543+**How to test:**
544+```bash
545+./bin/zoea --offline
546+# Press Ctrl+C
547+# Should exit immediately
548+```
549+
550+### Test 3: Exit with Running Myses
551+
552+**Expected behavior:**
553+1. Create 3 myses, start them
554+2. Wait for them to be actively processing
555+3. Press 'q'
556+4. Each mysis receives `Stop()` call
557+5. Each mysis finishes current turn (respects `turnMu`)
558+6. All myses exit cleanly
559+7. Application exits
560+
561+**How to test:**
562+```bash
563+./bin/zoea --offline
564+# Press 'n', create 3 myses
565+# Wait a few seconds for them to start processing
566+# Press 'q'
567+# Should exit within 2-3 seconds (waiting for LLM timeouts)
568+```
569+
570+### Test 4: Exit During LLM Call
571+
572+**Expected behavior:**
573+1. Start mysis, send it a message
574+2. While LLM is processing (loading indicator visible)
575+3. Press 'q'
576+4. Mysis respects context timeout (constants.LLMRequestTimeout)
577+5. Application exits after timeout expires
578+
579+**How to test:**
580+```bash
581+./bin/zoea
582+# Create mysis, start it
583+# Send message with 'm'
584+# Immediately press 'q' while "thinking"
585+# Should exit within LLMRequestTimeout seconds
586+```
587+
588+---
589+
590+## Additional Observations
591+
592+### Good Patterns Found
593+
594+1. **Context-based cancellation in Mysis.run()** - Clean shutdown pattern
595+2. **Turnlock mutex in Stop()** - Ensures operations complete before stopping
596+3. **Deferred ticker.Stop() in run()** - Prevents timer leaks
597+4. **Buffered event bus** - Reduces blocking risk
598+
599+### Areas of Concern
600+
601+1. **No sync.WaitGroup for goroutine tracking** - Can't wait for all goroutines to finish
602+2. **No graceful shutdown timeout** - If myses hang, app hangs forever
603+3. **Event bus subscriber never unsubscribed** - TUI doesn't call `Unsubscribe()`
604+4. **No goroutine leak testing** - Runtime stats not monitored
605+
606+---
607+
608+## Implementation Checklist
609+
610+- [ ] Implement Solution 1: Close bus before quit in signal handler
611+- [ ] Add quit callback to TUI model
612+- [ ] Test normal exit (press 'q')
613+- [ ] Test signal exit (Ctrl+C)
614+- [ ] Test exit with running myses
615+- [ ] Test exit during LLM call
616+- [ ] Add graceful shutdown timeout (optional)
617+- [ ] Add sync.WaitGroup for goroutine tracking (optional)
618+- [ ] Add runtime/pprof goroutine profiling (debugging)
619+- [ ] Document cleanup sequence in AGENTS.md
620+
621+---
622+
623+## References
624+
625+- `internal/core/mysis.go:913-944` - Mysis.run() loop
626+- `internal/core/commander.go:354-368` - Commander.StopAll()
627+- `internal/tui/app.go:862-870` - TUI event listener
628+- `internal/core/bus.go:179-187` - EventBus.Close()
629+- `cmd/zoea/main.go:141-173` - Main shutdown sequence
630+- `internal/constants/constants.go` - Timeout constants
631+
632+---
633+
634+**Analysis Date:** 2026-02-06
635+**Analyst:** OpenCode Agent (Claude Sonnet 4.5)