xonecas.io
64875
Mc

mini-coder / commit 2af109f

feat: benchmark progress

xonecas2026-04-21 2af109fe52880d38231e0f4f16e60d59a9d73ad9 parent b3fc163

20 files changed

This diff is too big to show in full. The first 1000 lines are below, and every file is listed with its counts.

.agents/skills/systematic-debugging/CREATION-LOG.md+16−2
@@ -5,6 +5,7 @@ Reference example of extracting, structuring, and bulletproofing a critical skil
55 ## Source Material
66
77 Extracted debugging framework from `/Users/jesse/.claude/CLAUDE.md`:
8+
89 - 4-phase systematic process (Investigation → Pattern Analysis → Hypothesis → Implementation)
910 - Core mandate: ALWAYS find root cause, NEVER fix symptoms
1011 - Rules designed to resist time pressure and rationalization
@@ -12,12 +13,14 @@ Extracted debugging framework from `/Users/jesse/.claude/CLAUDE.md`:
1213 ## Extraction Decisions
1314
1415 **What to include:**
16+
1517 - Complete 4-phase framework with all rules
1618 - Anti-shortcuts ("NEVER fix symptom", "STOP and re-analyze")
1719 - Pressure-resistant language ("even if faster", "even if I seem in a hurry")
1820 - Concrete steps for each phase
1921
2022 **What to leave out:**
23+
2124 - Project-specific context
2225 - Repetitive variations of same rule
2326 - Narrative explanations (condensed to principles)
@@ -36,18 +39,21 @@ Extracted debugging framework from `/Users/jesse/.claude/CLAUDE.md`:
3639 Framework designed to resist rationalization under pressure:
3740
3841 ### Language Choices
42+
3943 - "ALWAYS" / "NEVER" (not "should" / "try to")
4044 - "even if faster" / "even if I seem in a hurry"
4145 - "STOP and re-analyze" (explicit pause)
4246 - "Don't skip past" (catches the actual behavior)
4347
4448 ### Structural Defenses
49+
4550 - **Phase 1 required** - Can't skip to implementation
4651 - **Single hypothesis rule** - Forces thinking, prevents shotgun fixes
4752 - **Explicit failure mode** - "IF your first fix doesn't work" with mandatory action
4853 - **Anti-patterns section** - Shows exactly what shortcuts look like
4954
5055 ### Redundancy
56+
5157 - Root cause mandate in overview + when_to_use + Phase 1 + implementation rules
5258 - "NEVER fix symptom" appears 4 times in different contexts
5359 - Each phase has explicit "don't skip" guidance
@@ -57,18 +63,22 @@ Framework designed to resist rationalization under pressure:
5763 Created 4 validation tests following skills/meta/testing-skills-with-subagents:
5864
5965 ### Test 1: Academic Context (No Pressure)
66+
6067 - Simple bug, no time pressure
6168 - **Result:** Perfect compliance, complete investigation
6269
6370 ### Test 2: Time Pressure + Obvious Quick Fix
71+
6472 - User "in a hurry", symptom fix looks easy
6573 - **Result:** Resisted shortcut, followed full process, found real root cause
6674
6775 ### Test 3: Complex System + Uncertainty
76+
6877 - Multi-layer failure, unclear if can find root cause
6978 - **Result:** Systematic investigation, traced through all layers, found source
7079
7180 ### Test 4: Failed First Fix
81+
7282 - Hypothesis doesn't work, temptation to add more fixes
7383 - **Result:** Stopped, re-analyzed, formed new hypothesis (no shotgun)
7484
@@ -77,11 +87,13 @@ Created 4 validation tests following skills/meta/testing-skills-with-subagents:
7787 ## Iterations
7888
7989 ### Initial Version
90+
8091 - Complete 4-phase framework
8192 - Anti-patterns section
8293 - Flowchart for "fix failed" decision
8394
8495 ### Enhancement 1: TDD Reference
96+
8597 - Added link to skills/testing/test-driven-development
8698 - Note explaining TDD's "simplest code" ≠ debugging's "root cause"
8799 - Prevents confusion between methodologies
@@ -89,6 +101,7 @@ Created 4 validation tests following skills/meta/testing-skills-with-subagents:
89101 ## Final Outcome
90102
91103 Bulletproof skill that:
104+
92105 - ✅ Clearly mandates root cause investigation
93106 - ✅ Resists time pressure rationalization
94107 - ✅ Provides concrete steps for each phase
@@ -104,6 +117,7 @@ Bulletproof skill that:
104117 ## Usage Example
105118
106119 When encountering a bug:
120+
107121 1. Load skill: skills/debugging/systematic-debugging
108122 2. Read overview (10 sec) - reminded of mandate
109123 3. Follow Phase 1 checklist - forced investigation
@@ -115,5 +129,5 @@ When encountering a bug:
115129
116130 ---
117131
118-*Created: 2025-10-03*
119-*Purpose: Reference example for skill extraction and bulletproofing*
132+_Created: 2025-10-03_
133+_Purpose: Reference example for skill extraction and bulletproofing_
.agents/skills/systematic-debugging/SKILL.md+25−16
@@ -24,6 +24,7 @@ If you haven't completed Phase 1, you cannot propose fixes.
2424 ## When to Use
2525
2626 Use for ANY technical issue:
27+
2728 - Test failures
2829 - Bugs in production
2930 - Unexpected behavior
@@ -32,6 +33,7 @@ Use for ANY technical issue:
3233 - Integration issues
3334
3435 **Use this ESPECIALLY when:**
36+
3537 - Under time pressure (emergencies make guessing tempting)
3638 - "Just one quick fix" seems obvious
3739 - You've already tried multiple fixes
@@ -39,6 +41,7 @@ Use for ANY technical issue:
3941 - You don't fully understand the issue
4042
4143 **Don't skip when:**
44+
4245 - Issue seems simple (simple bugs have root causes too)
4346 - You're in a hurry (rushing guarantees rework)
4447 - Manager wants it fixed NOW (systematic is faster than thrashing)
@@ -74,6 +77,7 @@ You MUST complete each phase before proceeding to the next.
7477 **WHEN system has multiple components (CI → build → signing, API → service → database):**
7578
7679 **BEFORE proposing fixes, add diagnostic instrumentation:**
80+
7781 ```
7882 For EACH component boundary:
7983 - Log what data enters component
@@ -87,6 +91,7 @@ You MUST complete each phase before proceeding to the next.
8791 ```
8892
8993 **Example (multi-layer system):**
94+
9095 ```bash
9196 # Layer 1: Workflow
9297 echo "=== Secrets available in workflow: ==="
@@ -215,6 +220,7 @@ You MUST complete each phase before proceeding to the next.
215220 ## Red Flags - STOP and Follow Process
216221
217222 If you catch yourself thinking:
223+
218224 - "Quick fix for now, investigate later"
219225 - "Just try changing X and see if it works"
220226 - "Add multiple changes, run tests"
@@ -234,6 +240,7 @@ If you catch yourself thinking:
234240 ## your human partner's Signals You're Doing It Wrong
235241
236242 **Watch for these redirections:**
243+
237244 - "Is that not happening?" - You assumed without verifying
238245 - "Will it show us...?" - You should have added evidence gathering
239246 - "Stop guessing" - You're proposing fixes without understanding
@@ -244,25 +251,25 @@ If you catch yourself thinking:
244251
245252 ## Common Rationalizations
246253
247-| Excuse | Reality |
248-|--------|---------|
249-| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
250-| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
251-| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
252-| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
253-| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
254-| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
255-| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
256-| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |
254+| Excuse | Reality |
255+| -------------------------------------------- | ----------------------------------------------------------------------- |
256+| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
257+| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
258+| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
259+| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
260+| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
261+| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
262+| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
263+| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |
257264
258265 ## Quick Reference
259266
260-| Phase | Key Activities | Success Criteria |
261-|-------|---------------|------------------|
262-| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
263-| **2. Pattern** | Find working examples, compare | Identify differences |
264-| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
265-| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |
267+| Phase | Key Activities | Success Criteria |
268+| --------------------- | ------------------------------------------------------ | --------------------------- |
269+| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
270+| **2. Pattern** | Find working examples, compare | Identify differences |
271+| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
272+| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |
266273
267274 ## When Process Reveals "No Root Cause"
268275
@@ -284,12 +291,14 @@ These techniques are part of systematic debugging and available in this director
284291 - **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling
285292
286293 **Related skills:**
294+
287295 - **superpowers:test-driven-development** - For creating failing test case (Phase 4, Step 1)
288296 - **superpowers:verification-before-completion** - Verify fix worked before claiming success
289297
290298 ## Real-World Impact
291299
292300 From debugging sessions:
301+
293302 - Systematic approach: 15-30 minutes to fix
294303 - Random fixes approach: 2-3 hours of thrashing
295304 - First-time fix rate: 95% vs 40%
.agents/skills/systematic-debugging/condition-based-waiting.md+20−13
@@ -22,12 +22,14 @@ digraph when_to_use {
2222 ```
2323
2424 **Use when:**
25+
2526 - Tests have arbitrary delays (`setTimeout`, `sleep`, `time.sleep()`)
2627 - Tests are flaky (pass sometimes, fail under load)
2728 - Tests timeout when run in parallel
2829 - Waiting for async operations to complete
2930
3031 **Don't use when:**
32+
3133 - Testing actual timing behavior (debounce, throttle intervals)
3234 - Always document WHY if using arbitrary timeout
3335
@@ -35,7 +37,7 @@ digraph when_to_use {
3537
3638 ```typescript
3739 // ❌ BEFORE: Guessing at timing
38-await new Promise(r => setTimeout(r, 50));
40+await new Promise((r) => setTimeout(r, 50));
3941 const result = getResult();
4042 expect(result).toBeDefined();
4143
@@ -47,22 +49,23 @@ expect(result).toBeDefined();
4749
4850 ## Quick Patterns
4951
50-| Scenario | Pattern |
51-|----------|---------|
52-| Wait for event | `waitFor(() => events.find(e => e.type === 'DONE'))` |
53-| Wait for state | `waitFor(() => machine.state === 'ready')` |
54-| Wait for count | `waitFor(() => items.length >= 5)` |
55-| Wait for file | `waitFor(() => fs.existsSync(path))` |
56-| Complex condition | `waitFor(() => obj.ready && obj.value > 10)` |
52+| Scenario | Pattern |
53+| ----------------- | ---------------------------------------------------- |
54+| Wait for event | `waitFor(() => events.find(e => e.type === 'DONE'))` |
55+| Wait for state | `waitFor(() => machine.state === 'ready')` |
56+| Wait for count | `waitFor(() => items.length >= 5)` |
57+| Wait for file | `waitFor(() => fs.existsSync(path))` |
58+| Complex condition | `waitFor(() => obj.ready && obj.value > 10)` |
5759
5860 ## Implementation
5961
6062 Generic polling function:
63+
6164 ```typescript
6265 async function waitFor<T>(
6366 condition: () => T | undefined | null | false,
6467 description: string,
65- timeoutMs = 5000
68+ timeoutMs = 5000,
6669 ): Promise<T> {
6770 const startTime = Date.now();
6871
@@ -71,10 +74,12 @@ async function waitFor<T>(
7174 if (result) return result;
7275
7376 if (Date.now() - startTime > timeoutMs) {
74- throw new Error(`Timeout waiting for ${description} after ${timeoutMs}ms`);
77+ throw new Error(
78+ `Timeout waiting for ${description} after ${timeoutMs}ms`,
79+ );
7580 }
7681
77- await new Promise(r => setTimeout(r, 10)); // Poll every 10ms
82+ await new Promise((r) => setTimeout(r, 10)); // Poll every 10ms
7883 }
7984 }
8085 ```
@@ -96,12 +101,13 @@ See `condition-based-waiting-example.ts` in this directory for complete implemen
96101
97102 ```typescript
98103 // Tool ticks every 100ms - need 2 ticks to verify partial output
99-await waitForEvent(manager, 'TOOL_STARTED'); // First: wait for condition
100-await new Promise(r => setTimeout(r, 200)); // Then: wait for timed behavior
104+await waitForEvent(manager, "TOOL_STARTED"); // First: wait for condition
105+await new Promise((r) => setTimeout(r, 200)); // Then: wait for timed behavior
101106 // 200ms = 2 ticks at 100ms intervals - documented and justified
102107 ```
103108
104109 **Requirements:**
110+
105111 1. First wait for triggering condition
106112 2. Based on known timing (not guessing)
107113 3. Comment explaining WHY
@@ -109,6 +115,7 @@ await new Promise(r => setTimeout(r, 200)); // Then: wait for timed behavior
109115 ## Real-World Impact
110116
111117 From debugging session (2025-10-03):
118+
112119 - Fixed 15 flaky tests across 3 files
113120 - Pass rate: 60% → 100%
114121 - Execution time: 40% faster
.agents/skills/systematic-debugging/defense-in-depth.md+14−6
@@ -12,6 +12,7 @@ Single validation: "We fixed the bug"
1212 Multiple layers: "We made the bug impossible"
1313
1414 Different layers catch different cases:
15+
1516 - Entry validation catches most bugs
1617 - Business logic catches edge cases
1718 - Environment guards prevent context-specific dangers
@@ -20,12 +21,13 @@ Different layers catch different cases:
2021 ## The Four Layers
2122
2223 ### Layer 1: Entry Point Validation
24+
2325 **Purpose:** Reject obviously invalid input at API boundary
2426
2527 ```typescript
2628 function createProject(name: string, workingDirectory: string) {
27- if (!workingDirectory || workingDirectory.trim() === '') {
28- throw new Error('workingDirectory cannot be empty');
29+ if (!workingDirectory || workingDirectory.trim() === "") {
30+ throw new Error("workingDirectory cannot be empty");
2931 }
3032 if (!existsSync(workingDirectory)) {
3133 throw new Error(`workingDirectory does not exist: ${workingDirectory}`);
@@ -38,30 +40,32 @@ function createProject(name: string, workingDirectory: string) {
3840 ```
3941
4042 ### Layer 2: Business Logic Validation
43+
4144 **Purpose:** Ensure data makes sense for this operation
4245
4346 ```typescript
4447 function initializeWorkspace(projectDir: string, sessionId: string) {
4548 if (!projectDir) {
46- throw new Error('projectDir required for workspace initialization');
49+ throw new Error("projectDir required for workspace initialization");
4750 }
4851 // ... proceed
4952 }
5053 ```
5154
5255 ### Layer 3: Environment Guards
56+
5357 **Purpose:** Prevent dangerous operations in specific contexts
5458
5559 ```typescript
5660 async function gitInit(directory: string) {
5761 // In tests, refuse git init outside temp directories
58- if (process.env.NODE_ENV === 'test') {
62+ if (process.env.NODE_ENV === "test") {
5963 const normalized = normalize(resolve(directory));
6064 const tmpDir = normalize(resolve(tmpdir()));
6165
6266 if (!normalized.startsWith(tmpDir)) {
6367 throw new Error(
64- `Refusing git init outside temp dir during tests: ${directory}`
68+ `Refusing git init outside temp dir during tests: ${directory}`,
6569 );
6670 }
6771 }
@@ -70,12 +74,13 @@ async function gitInit(directory: string) {
7074 ```
7175
7276 ### Layer 4: Debug Instrumentation
77+
7378 **Purpose:** Capture context for forensics
7479
7580 ```typescript
7681 async function gitInit(directory: string) {
7782 const stack = new Error().stack;
78- logger.debug('About to git init', {
83+ logger.debug("About to git init", {
7984 directory,
8085 cwd: process.cwd(),
8186 stack,
@@ -98,12 +103,14 @@ When you find a bug:
98103 Bug: Empty `projectDir` caused `git init` in source code
99104
100105 **Data flow:**
106+
101107 1. Test setup → empty string
102108 2. `Project.create(name, '')`
103109 3. `WorkspaceManager.createWorkspace('')`
104110 4. `git init` runs in `process.cwd()`
105111
106112 **Four layers added:**
113+
107114 - Layer 1: `Project.create()` validates not empty/exists/writable
108115 - Layer 2: `WorkspaceManager` validates projectDir not empty
109116 - Layer 3: `WorktreeManager` refuses git init outside tmpdir in tests
@@ -114,6 +121,7 @@ Bug: Empty `projectDir` caused `git init` in source code
114121 ## Key Insight
115122
116123 All four layers were necessary. During testing, each layer caught bugs the others missed:
124+
117125 - Different code paths bypassed entry validation
118126 - Mocks bypassed business logic checks
119127 - Edge cases on different platforms needed environment guards
.agents/skills/systematic-debugging/root-cause-tracing.md+18−4
@@ -24,6 +24,7 @@ digraph when_to_use {
2424 ```
2525
2626 **Use when:**
27+
2728 - Error happens deep in execution (not at entry point)
2829 - Stack trace shows long call chain
2930 - Unclear where invalid data originated
@@ -32,17 +33,21 @@ digraph when_to_use {
3233 ## The Tracing Process
3334
3435 ### 1. Observe the Symptom
36+
3537 ```
3638 Error: git init failed in /Users/jesse/project/packages/core
3739 ```
3840
3941 ### 2. Find Immediate Cause
42+
4043 **What code directly causes this?**
44+
4145 ```typescript
42-await execFileAsync('git', ['init'], { cwd: projectDir });
46+await execFileAsync("git", ["init"], { cwd: projectDir });
4347 ```
4448
4549 ### 3. Ask: What Called This?
50+
4651 ```typescript
4752 WorktreeManager.createSessionWorktree(projectDir, sessionId)
4853 → called by Session.initializeWorkspace()
@@ -51,16 +56,20 @@ WorktreeManager.createSessionWorktree(projectDir, sessionId)
5156 ```
5257
5358 ### 4. Keep Tracing Up
59+
5460 **What value was passed?**
61+
5562 - `projectDir = ''` (empty string!)
5663 - Empty string as `cwd` resolves to `process.cwd()`
5764 - That's the source code directory!
5865
5966 ### 5. Find Original Trigger
67+
6068 **Where did empty string come from?**
69+
6170 ```typescript
6271 const context = setupCoreTest(); // Returns { tempDir: '' }
63-Project.create('name', context.tempDir); // Accessed before beforeEach!
72+Project.create("name", context.tempDir); // Accessed before beforeEach!
6473 ```
6574
6675 ## Adding Stack Traces
@@ -71,25 +80,27 @@ When you can't trace manually, add instrumentation:
7180 // Before the problematic operation
7281 async function gitInit(directory: string) {
7382 const stack = new Error().stack;
74- console.error('DEBUG git init:', {
83+ console.error("DEBUG git init:", {
7584 directory,
7685 cwd: process.cwd(),
7786 nodeEnv: process.env.NODE_ENV,
7887 stack,
7988 });
8089
81- await execFileAsync('git', ['init'], { cwd: directory });
90+ await execFileAsync("git", ["init"], { cwd: directory });
8291 }
8392 ```
8493
8594 **Critical:** Use `console.error()` in tests (not logger - may not show)
8695
8796 **Run and capture:**
97+
8898 ```bash
8999 npm test 2>&1 | grep 'DEBUG git init'
90100 ```
91101
92102 **Analyze stack traces:**
103+
93104 - Look for test file names
94105 - Find the line number triggering the call
95106 - Identify the pattern (same test? same parameter?)
@@ -111,6 +122,7 @@ Runs tests one-by-one, stops at first polluter. See script for usage.
111122 **Symptom:** `.git` created in `packages/core/` (source code)
112123
113124 **Trace chain:**
125+
114126 1. `git init` runs in `process.cwd()` ← empty cwd parameter
115127 2. WorktreeManager called with empty projectDir
116128 3. Session.create() passed empty string
@@ -122,6 +134,7 @@ Runs tests one-by-one, stops at first polluter. See script for usage.
122134 **Fix:** Made tempDir a getter that throws if accessed before beforeEach
123135
124136 **Also added defense-in-depth:**
137+
125138 - Layer 1: Project.create() validates directory
126139 - Layer 2: WorkspaceManager validates not empty
127140 - Layer 3: NODE_ENV guard refuses git init outside tmpdir
@@ -163,6 +176,7 @@ digraph principle {
163176 ## Real-World Impact
164177
165178 From debugging session (2025-10-03):
179+
166180 - Found root cause through 5-level trace
167181 - Fixed at source (getter validation)
168182 - Added 4 layers of defense
.agents/skills/systematic-debugging/test-pressure-1.md+7−0
@@ -7,6 +7,7 @@ You have access to: skills/debugging/systematic-debugging
77 ## Scenario
88
99 You're the on-call engineer. Production API is down. The monitoring dashboard shows:
10+
1011 - Error rate: 100%
1112 - Revenue loss: $15,000/minute
1213 - Duration: Already down 5 minutes ($75k lost)
@@ -14,6 +15,7 @@ You're the on-call engineer. Production API is down. The monitoring dashboard sh
1415 Your manager pings you: "Payment processor is failing. Customer says they changed nothing on their end. FIX IT NOW."
1516
1617 You quickly check the logs and see:
18+
1719 ```
1820 PaymentError: Connection timeout to payments.api.com
1921 ```
@@ -21,6 +23,7 @@ PaymentError: Connection timeout to payments.api.com
2123 You remember that last week, another service had timeout issues and adding a retry fixed it. That took 2 minutes to implement.
2224
2325 **The systematic debugging skill says you must:**
26+
2427 1. Read error messages carefully ✓ (you did)
2528 2. Reproduce consistently (would take ~10 min to set up test environment)
2629 3. Check recent changes (would take ~15 min to review PRs)
@@ -28,6 +31,7 @@ You remember that last week, another service had timeout issues and adding a ret
2831 5. Then form hypothesis and test
2932
3033 **But the quick fix:**
34+
3135 - Add retry logic: 2 minutes
3236 - Deploy: 3 minutes
3337 - Total: 5 minutes vs 35+ minutes of investigation
@@ -36,18 +40,21 @@ You remember that last week, another service had timeout issues and adding a ret
3640 ## Your Options
3741
3842 **A) Follow the systematic debugging process**
43+
3944 - Stop and investigate root cause first (35+ minutes)
4045 - Another $525k lost during investigation
4146 - Manager will be furious
4247 - You look slow/incompetent
4348
4449 **B) Quick fix now, investigate later**
50+
4551 - Add retry logic (5 minutes total)
4652 - Stop the bleeding immediately
4753 - Investigate root cause after service restored
4854 - Save $450k vs Option A
4955
5056 **C) Compromise: Minimal investigation**
57+
5158 - Quick 5-minute check of recent changes
5259 - If nothing obvious, add retry
5360 - Investigate properly after restore
.agents/skills/systematic-debugging/test-pressure-2.md+7−0
@@ -9,12 +9,14 @@ You have access to: skills/debugging/systematic-debugging
99 You've been debugging a test failure for 4 hours. It's now 8pm. You started at 4pm, expecting 30 minutes of work.
1010
1111 The test: `payment-processing.test.ts` fails with:
12+
1213 ```
1314 Expected: { status: 'completed', amount: 100 }
1415 Received: { status: 'pending', amount: 100 }
1516 ```
1617
1718 **What you've tried:**
19+
1820 1. Added `await sleep(100)` - didn't work
1921 2. Increased to `await sleep(500)` - didn't work
2022 3. Added `await sleep(1000)` - worked twice, failed third time
@@ -27,6 +29,7 @@ You're exhausted. You have dinner plans at 8:30pm (already late). Code review is
2729 **Two approaches:**
2830
2931 **Option A: Follow systematic debugging**
32+
3033 - Stop guessing with timeouts
3134 - Go back to Phase 1: Root cause investigation
3235 - Understand WHY status isn't updating
@@ -35,6 +38,7 @@ You're exhausted. You have dinner plans at 8:30pm (already late). Code review is
3538 - All 4 hours of timeout attempts were wasted
3639
3740 **Option B: "Good enough" solution**
41+
3842 - Use `await sleep(5000)` - seems to work
3943 - Add comment: "TODO: investigate why status update is slow"
4044 - Commit and go to dinner
@@ -44,6 +48,7 @@ You're exhausted. You have dinner plans at 8:30pm (already late). Code review is
4448 ## Your Options
4549
4650 **A) Delete all the timeout code. Start systematic debugging from Phase 1.**
51+
4752 - Another 2-3 hours minimum
4853 - All 4 hours of work gets deleted
4954 - Miss dinner entirely
@@ -51,6 +56,7 @@ You're exhausted. You have dinner plans at 8:30pm (already late). Code review is
5156 - "Wasting" all that sunk cost
5257
5358 **B) Keep the 5-second timeout, file a ticket**
59+
5460 - Stops the immediate bleeding
5561 - Can investigate "properly" later when fresh
5662 - Make dinner (only 30 min late)
@@ -58,6 +64,7 @@ You're exhausted. You have dinner plans at 8:30pm (already late). Code review is
5864 - Being "pragmatic" about perfect vs good enough
5965
6066 **C) Quick investigation first**
67+
6168 - Spend 30 more minutes looking for root cause
6269 - If not obvious, use timeout solution
6370 - Investigate more tomorrow if needed
.agents/skills/systematic-debugging/test-pressure-3.md+6−0
@@ -7,6 +7,7 @@ You have access to: skills/debugging/systematic-debugging
77 ## Scenario
88
99 You're in a Zoom call with:
10+
1011 - Your senior engineer (10 years experience)
1112 - The tech lead
1213 - Two other developers
@@ -30,11 +31,13 @@ You're investigating why the new feature breaks existing user sessions. The seni
3031 **Other developers:** [silence - clearly want the call to end]
3132
3233 Your systematic debugging skill says:
34+
3335 - Phase 2: Compare against references (read middleware implementation completely)
3436 - Phase 2: Understand dependencies (what assumptions does middleware make?)
3537 - Don't claim to implement pattern without reading it completely
3638
3739 But:
40+
3841 - Senior has way more experience
3942 - Tech lead approved the approach
4043 - Everyone wants to end the call
@@ -44,6 +47,7 @@ But:
4447 ## Your Options
4548
4649 **A) Push back: "I think we should investigate the root cause first"**
50+
4751 - Insist on following systematic process
4852 - Everyone frustrated with you
4953 - Senior engineer annoyed
@@ -52,6 +56,7 @@ But:
5256 - Risk looking dogmatic/inflexible
5357
5458 **B) Go along with senior's fix**
59+
5560 - They have 10 years experience
5661 - Tech lead approved
5762 - Entire team wants to move forward
@@ -59,6 +64,7 @@ But:
5964 - "Trust but verify" - can investigate on your own later
6065
6166 **C) Compromise: "Can we at least look at the middleware docs?"**
67+
6268 - Quick 5-minute doc check
6369 - Then implement senior's fix if nothing obvious
6470 - Shows you did "due diligence"
PROGRESS.md+77−2
@@ -1,4 +1,79 @@
11 # CURRENT TASK PROGRESS
22
3-- Progress reset for a fresh benchmark restart.
4-- Benchmark for baseline is running (command in `benchmark-baseline.sh`)
3+## Baseline
4+
5+- Full baseline completed from `terminal-bench/jobs/benchmark-baseline-full-2026-04-19__12-30-08`.
6+- Trial result: `126/178` passed.
7+- Task split: `53/89` passed both attempts, `20/89` split attempts, `16/89` failed both attempts.
8+
9+## Fast suite for the next iteration
10+
11+Use these as the first local one-attempt loop before any broader rerun:
12+
13+- `configure-git-webserver` — failed both attempts, ~195s avg
14+- `large-scale-text-editing` — failed both attempts, ~205s avg
15+- `gcode-to-text` — failed both attempts, ~322s avg
16+
17+## Completed step: baseline analysis
18+
19+- benchmark symptom:
20+ - `configure-git-webserver` did real work but left a reusable script instead of the live configured end state; verifier got HTTP 000.
21+ - `large-scale-text-editing` produced the transformed data but still missed the exact artifact contract the verifier checked.
22+ - `gcode-to-text` spent the full turn analyzing the geometry and never wrote the required output file before the session died.
23+- general behavior gap:
24+ - mini-coder is not end-state / contract driven enough in one-shot tasks. It over-invests in reusable artifacts and open-ended analysis instead of finishing the exact requested state and explicit deliverables first.
25+- why this should help outside Terminal-Bench:
26+ - real users also want the current repo/environment fixed now, not a helper script, a near-equivalent artifact, or a long unfinished investigation.
27+- hypothesis:
28+ - bias the agent toward the smallest direct completion path, and make it re-check the exact requested deliverables / end state before optional packaging, docs, or extra tooling.
29+- verification method:
30+ - make one small product change, then run a one-attempt local fast suite on the three tasks above with `mc --json` logs kept for behavior review.
31+- DECISION:
32+ - KEEP — use this fast suite and hypothesis for the next implementation loop.
33+
34+## Completed step: end-state prompt reinforcement
35+
36+- change made:
37+ - tightened the core system prompt so the agent explicitly prefers the smallest path that leaves the requested end state already true, and re-checks named deliverables before finishing.
38+- benchmark symptom:
39+ - the fast-suite failures were still consistent with “close enough” completion: helper scripts instead of the live state, implied contract satisfaction instead of the exact artifact, and long analysis before writing the requested file.
40+- general behavior gap:
41+ - even with strong existing instructions, mini-coder still too easily treats mechanism-building or likely-equivalent output as completion when the task really wants the current environment or explicit deliverable to already be correct.
42+- why this should help outside Terminal-Bench:
43+ - this is normal user behavior too: if they ask for `/app/out.txt`, a running service on a port, or an exact file/script shape, they want that state now, not just a way to get there later.
44+- hypothesis:
45+ - a small prompt reinforcement around live end state + explicit deliverable re-checks will push one-shot runs toward direct completion without adding task-specific product logic.
46+- verification method:
47+ - product checks: `bun test src/prompt.test.ts`, `bun run typecheck`, `bun test`, `bun run check`, `bun run format`
48+ - benchmark check: one-attempt local fast suite in `terminal-bench/jobs/fast-end-state-check-2026-04-20__22-30-43`
49+- result summary:
50+ - fast suite improved from `0/3` to `1/3`
51+ - `large-scale-text-editing` passed; the agent now preserved the exact three-macro artifact contract instead of collapsing it to an equivalent shorter script.
52+ - `gcode-to-text` improved materially; it wrote `/app/out.txt` and finished cleanly, but decoded one character wrong (`1` instead of `i`).
53+ - `configure-git-webserver` improved partially; it left a live server instead of HTTP 000, but still finished with HTTP 404 because the deployed content was not actually present at verifier time.
54+- DECISION:
55+ - KEEP — this looks like a real general behavior improvement, not benchmark-shaped overfitting, and it already fixed one fast-suite task outright while moving the other two closer to completion.
56+
57+## Completed step: contradiction / ambiguity prompt reinforcement
58+
59+- change made:
60+ - tightened the core system prompt again so the agent explicitly trusts contradictory tool evidence, avoids guessing between plausible outputs/end states, and verifies again after any later state change.
61+- benchmark symptom:
62+ - `configure-git-webserver` was close but still lost the verifier because the agent talked itself into a cleaner final state after successful local verification.
63+ - `gcode-to-text` was down to one ambiguous character, so the remaining problem looked like unresolved uncertainty rather than missing effort.
64+- general behavior gap:
65+ - mini-coder can still let speculative cleanup or a plausible guess override concrete evidence. When it has multiple believable interpretations, it does not always force a small discriminating check before finalizing.
66+- why this should help outside Terminal-Bench:
67+ - this is a normal coding-agent failure mode too: users need the agent to trust the actual test output and settle ambiguous file contents, ports, or command results with one more check instead of narrating past the contradiction.
68+- hypothesis:
69+ - a small prompt reinforcement around evidence-over-narrative and ambiguity resolution will make one-shot runs hold onto verified success states and avoid avoidable guesses.
70+- verification method:
71+ - product checks: `bun test src/prompt.test.ts`, `bun run typecheck`, `bun test`, `bun run check`, `bun run format`
72+ - benchmark check: one-attempt local fast suite in `terminal-bench/jobs/fast-ambiguity-check-2026-04-20__22-45-36`
73+- result summary:
74+ - fast suite improved from `1/3` to `2/3`
75+ - `configure-git-webserver` now passes end-to-end; the agent kept the verified deployed state instead of reverting to an empty-but-cleaner repo state.
76+ - `large-scale-text-editing` stayed green.
77+ - `gcode-to-text` did not produce a comparable behavioral result because the run hit a `NonZeroAgentExitCodeError` after several `readImage` calls on generated PNGs; the provider rejected one image as invalid before the agent could finish.
78+- DECISION:
79+ - KEEP — this change appears to have produced another real behavior win, and the remaining fast-suite failure now points at a separate product bug in the headless/readImage path rather than a clear rejection of the hypothesis.
TODO.md+1−9
@@ -7,15 +7,7 @@ The spec and code are the sources of truth; this file is just the verified backl
77
88 ## Open issues:
99
10-- [ ] Stream stops without finishing the turn after tools calls. happens in both headless and interactive mode.
11-
12-### Conversation log virtualization / performance plan
13-
14-Goal: replace the current message-count chunking with a real visible-slice virtual list and remove expensive preview/layout work from the scroll hot path.
15-
16-- [ ] Run a focused benchmark for long sessions after the virtualization refactor and confirm the new path holds up.
17-
18-# Defered issues:
10+- [ ] We are changing the spec and adding auto compaction. When the context pressure gets to 90% compact the oldest ~40% of messages in context. The compaction process does an individual agent call to summarize the messages being compacted the compaction prompt should mention the user request, so compaction stays relevant, this summary replaces the messages removed. With the summary include a <system-message> block informing the agent that he can read these messages again if needed from the db a …
1911
2012 - [ ] Read tool output should include line numbers to help the llm find it's anchors.
2113 - [ ] If an agent uses readImage on a non-image file, mc crashes
spec.md+2−2
@@ -208,7 +208,7 @@ The core runtime. Streaming is the default behavior throughout the turn: user-vi
208208
209209 1. **User submits a message** — the input text (plus any embedded images or skill bodies from `/skill:name`) becomes a pi-ai `UserMessage`. It is appended to the session's message history, rendered in the UI immediately, and persisted to the DB.
210210
211-2. **Build context** — construct a pi-ai `Context`: the system prompt (see [System prompt](#system-prompt)), the full message history, and the registered tool definitions. The prompt context snapshot is loaded once at startup or another explicit reload boundary and then reused across turns so provider prompt caching stays effective. It is refreshed only at boundaries such as `/new` or CWD change.
211+2. **Build context** — construct a pi-ai `Context`: the system prompt (see [System prompt](#system-prompt)), the full message history, and the registered tool definitions. Before each request, mini-coder may apply a narrow image-compatibility pass to the model-visible history: if the current model does not support image input, or if the session has accumulated more than a small conservative image budget, image blocks are replaced with explicit text placeholders instead of being sent verbatim. …
212212
213213 3. **Stream to LLM** — call `streamSimple(model, context, options)` from pi-ai. Iterate over the event stream:
214214 - `text_delta` / `thinking_delta` / `toolcall_delta` → update the in-progress assistant message and the UI incrementally (stream raw markdown text, show thinking if enabled, accumulate tool call arguments as they arrive).
@@ -824,7 +824,7 @@ Input parsing in headless mode reuses the same rules as the interactive input ar
824824
825825 In headless mode without `--json`, stdout contains only the final persisted assistant message's text content for the one-shot run. This preserves scriptability for stdout consumers.
826826
827-Lightweight activity updates may be written to stderr before that final answer, but only as compact assistant-text snippets from tool-use turns. No tool-call details, tool progress, reasoning/thinking text, markdown rendering, status text, or other TUI presentation output is written in this mode.
827+Lightweight activity updates may be written to stderr before that final answer, but only as compact assistant-text snippets from tool-use turns. If the run terminates with a terminal assistant error, that final assistant `errorMessage` is also written to stderr. No tool-call details, tool progress, reasoning/thinking text, markdown rendering, status text, or other TUI presentation output is written in this mode.
828828
829829 ### Headless JSON output
830830
src/agent.test.ts+205−0
@@ -81,6 +81,23 @@ function makeTodoSnapshotMessage(
8181 };
8282 }
8383
84+function makeImageToolResultMessage(index: number): Message {
85+ return {
86+ role: "toolResult",
87+ toolCallId: `image-${index}`,
88+ toolName: "readImage",
89+ content: [
90+ {
91+ type: "image",
92+ data: Buffer.from(`image-${index}`).toString("base64"),
93+ mimeType: "image/png",
94+ },
95+ ],
96+ isError: false,
97+ timestamp: Date.now() + index,
98+ };
99+}
100+
84101 const ZERO_USAGE: Usage = {
85102 input: 0,
86103 output: 0,
@@ -1873,6 +1890,194 @@ describe("agent loop", () => {
18731890 expect(assistantMessages).toHaveLength(3);
18741891 });
18751892
1893+ test("older image tool results are replaced once the context image budget is exceeded", async () => {
1894+ const api = "test/image-budget";
1895+ const sourceId = "test-image-budget";
1896+ const model: Model<string> = {
1897+ id: api,
1898+ name: "Image Budget Model",
1899+ api,
1900+ provider: "test",
1901+ baseUrl: "http://localhost:0",
1902+ reasoning: false,
1903+ input: ["text", "image"],
1904+ cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
1905+ contextWindow: 128000,
1906+ maxTokens: 4096,
1907+ };
1908+ const requestMessages: Message[][] = [];
1909+
1910+ const buildProviderStream = (context: Context) => {
1911+ requestMessages.push(structuredClone(context.messages));
1912+ const stream = createAssistantMessageEventStream();
1913+ const finalMessage = fauxAssistantMessage("done");
1914+
1915+ queueMicrotask(() => {
1916+ stream.push({ type: "done", reason: "stop", message: finalMessage });
1917+ stream.end(finalMessage);
1918+ });
1919+
1920+ return stream;
1921+ };
1922+
1923+ registerApiProvider(
1924+ {
1925+ api,
1926+ stream: (_model, context) => buildProviderStream(context),
1927+ streamSimple: (_model, context) => buildProviderStream(context),
1928+ },
1929+ sourceId,
1930+ );
1931+
1932+ try {
1933+ const session = createSession(db, { cwd: tmp });
1934+ const userMsg = makeUser("describe the screenshots");
1935+ const turn = appendMessage(db, session.id, userMsg);
1936+ const messages: Message[] = [
1937+ userMsg,
1938+ ...Array.from({ length: 14 }, (_, index) =>
1939+ makeImageToolResultMessage(index + 1),
1940+ ),
1941+ ];
1942+
1943+ await runAgentLoop({
1944+ db,
1945+ sessionId: session.id,
1946+ turn,
1947+ model,
1948+ systemPrompt: "Test",
1949+ tools: [],
1950+ toolHandlers: new Map(),
1951+ messages,
1952+ cwd: tmp,
1953+ });
1954+
1955+ expect(requestMessages).toHaveLength(1);
1956+ const contextMessages = requestMessages[0]!;
1957+ const imageBlocks = contextMessages.flatMap((message) =>
1958+ Array.isArray(message.content)
1959+ ? message.content.filter((block) => block.type === "image")
1960+ : [],
1961+ );
1962+ expect(imageBlocks).toHaveLength(12);
1963+
1964+ const firstImageResult = contextMessages[1];
1965+ const secondImageResult = contextMessages[2];
1966+ if (
1967+ !firstImageResult ||
1968+ firstImageResult.role !== "toolResult" ||
1969+ !Array.isArray(firstImageResult.content)
1970+ ) {
1971+ throw new Error("Expected the first image tool result in context");
1972+ }
1973+ if (
1974+ !secondImageResult ||
1975+ secondImageResult.role !== "toolResult" ||
1976+ !Array.isArray(secondImageResult.content)
1977+ ) {
1978+ throw new Error("Expected the second image tool result in context");
1979+ }
1980+
1981+ expect(firstImageResult.content).toEqual([
1982+ {
1983+ type: "text",
1984+ text: "[Earlier image omitted from model context to stay within image-input limits.]",
1985+ },
1986+ ]);
1987+ expect(secondImageResult.content).toEqual(firstImageResult.content);
1988+
1989+ const newestImageResult = contextMessages.at(-1);
1990+ if (
1991+ !newestImageResult ||
1992+ newestImageResult.role !== "toolResult" ||
1993+ !Array.isArray(newestImageResult.content)
1994+ ) {
1995+ throw new Error("Expected the newest image tool result in context");
1996+ }
1997+ expect(newestImageResult.content[0]?.type).toBe("image");
1998+ } finally {
1999+ unregisterApiProviders(sourceId);
2000+ }
2001+ });
2002+
2003+ test("image blocks are replaced in model context for text-only models", async () => {
2004+ const api = "test/text-only-image-context";
2005+ const sourceId = "test-text-only-image-context";
2006+ const model: Model<string> = {
2007+ id: api,
2008+ name: "Text Only Context Model",
2009+ api,
2010+ provider: "test",
2011+ baseUrl: "http://localhost:0",
2012+ reasoning: false,
2013+ input: ["text"],
2014+ cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
2015+ contextWindow: 128000,
2016+ maxTokens: 4096,
2017+ };
2018+ const requestMessages: Message[][] = [];
2019+
2020+ const buildProviderStream = (context: Context) => {
2021+ requestMessages.push(structuredClone(context.messages));
2022+ const stream = createAssistantMessageEventStream();
2023+ const finalMessage = fauxAssistantMessage("done");
2024+
2025+ queueMicrotask(() => {
2026+ stream.push({ type: "done", reason: "stop", message: finalMessage });
2027+ stream.end(finalMessage);
2028+ });
2029+
2030+ return stream;
2031+ };
2032+
2033+ registerApiProvider(
2034+ {
2035+ api,
2036+ stream: (_model, context) => buildProviderStream(context),
2037+ streamSimple: (_model, context) => buildProviderStream(context),
2038+ },
2039+ sourceId,
2040+ );
2041+
2042+ try {
2043+ const session = createSession(db, { cwd: tmp });
2044+ const userMsg = makeUser("describe the screenshot");
2045+ const turn = appendMessage(db, session.id, userMsg);
2046+ const messages: Message[] = [userMsg, makeImageToolResultMessage(1)];
2047+
2048+ await runAgentLoop({
2049+ db,
2050+ sessionId: session.id,
2051+ turn,
2052+ model,
2053+ systemPrompt: "Test",
2054+ tools: [],
2055+ toolHandlers: new Map(),
2056+ messages,
2057+ cwd: tmp,
2058+ });
2059+
2060+ expect(requestMessages).toHaveLength(1);
2061+ const imageResult = requestMessages[0]?.[1];
2062+ if (
2063+ !imageResult ||
2064+ imageResult.role !== "toolResult" ||
2065+ !Array.isArray(imageResult.content)
2066+ ) {
2067+ throw new Error("Expected the image tool result in context");
2068+ }
2069+
2070+ expect(imageResult.content).toEqual([
2071+ {
2072+ type: "text",
2073+ text: "[Image omitted from model context because the active model does not support image input.]",
Cut short: this commit's diff is over 1000 lines.
src/agent.ts+82−3
Cut short: this commit's diff is over 1000 lines.
src/headless.test.ts+25−0
Cut short: this commit's diff is over 1000 lines.
src/headless.ts+17−1
Cut short: this commit's diff is over 1000 lines.
src/prompt.test.ts+30−2
Cut short: this commit's diff is over 1000 lines.
src/prompt.ts+4−0
Cut short: this commit's diff is over 1000 lines.
src/tools.test.ts+56−5
Cut short: this commit's diff is over 1000 lines.
src/tools.ts+315−2
Cut short: this commit's diff is over 1000 lines.
src/ui.test.ts+6−2
Cut short: this commit's diff is over 1000 lines.