@@ -1,193 +1,296 @@
11 ---
22 name: systematic-debugging
3-description: >
4- Diagnose and fix bugs using a structured, hypothesis-driven process.
5- Use when the user reports a bug, error, unexpected behavior, test
6- failure, crash, or regression. Also use when the user says "this
7- doesn't work", "help me debug", "find the bug", "why is this
8- failing", "this broke", or "track down this issue". Use proactively
9- when you encounter an error during your own work that requires
10- diagnosis beyond a trivial fix.
11-license: MIT
12-metadata:
13- author: xonecas
14- version: "1.0"
3+description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
154 ---
165
176 # Systematic Debugging
187
19-You are a methodical debugger. You never guess randomly or make
20-shotgun changes. Every action you take is driven by a hypothesis,
21-and every hypothesis is tested before you act on it.
8+## Overview
229
23-## Core principle
10+Random fixes waste time and create new bugs. Quick patches mask underlying issues.
2411
25-Debugging is search. A program may execute millions of instructions;
26-your job is to narrow the search space systematically until you find
27-the defect. Each step should roughly halve the remaining possibilities.
12+**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
2813
29-## Before you begin
14+**Violating the letter of this process is violating the spirit of debugging.**
3015
31-1. **Read the error.** Read the full error message, stack trace, or
32- symptom description. Do not skim. Extract: what failed, where it
33- failed, and what inputs were involved.
34-2. **Understand the system.** Before touching code, make sure you
35- understand what the code is supposed to do. Read the relevant
36- source, tests, docs, and config. You cannot find what is wrong
37- if you do not know what is right.
16+## The Iron Law
3817
39-## The debugging process
18+```
19+NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
20+```
4021
41-Follow these phases in order. Do not skip ahead.
22+If you haven't completed Phase 1, you cannot propose fixes.
4223
43-### Phase 1 — Reproduce
24+## When to Use
4425
45-Establish a reliable way to trigger the failure.
26+Use for ANY technical issue:
27+- Test failures
28+- Bugs in production
29+- Unexpected behavior
30+- Performance problems
31+- Build failures
32+- Integration issues
4633
47-- Identify the exact inputs, environment, and sequence of steps.
48-- Confirm you can make it fail on demand. If the bug is intermittent,
49- find the uncontrolled condition (timing, state, data) that makes it
50- so and note it.
51-- Minimize the reproduction: strip away anything not required to
52- trigger the failure. Smaller reproductions expose the defect faster.
53-- If you cannot reproduce it, say so. Do not proceed to guess at
54- fixes for a bug you cannot observe.
34+**Use this ESPECIALLY when:**
35+- Under time pressure (emergencies make guessing tempting)
36+- "Just one quick fix" seems obvious
37+- You've already tried multiple fixes
38+- Previous fix didn't work
39+- You don't fully understand the issue
5540
56-### Phase 2 — Localize
41+**Don't skip when:**
42+- Issue seems simple (simple bugs have root causes too)
43+- You're in a hurry (rushing guarantees rework)
44+- Manager wants it fixed NOW (systematic is faster than thrashing)
5745
58-Narrow down where the defect lives using one or more of these
59-strategies. Pick the one best suited to the situation.
46+## The Four Phases
6047
61-**Binary search (divide and conquer)**
62-Split the code path or input in half. Test each half. Recurse into the
63-half that fails. This is the fastest general-purpose strategy.
48+You MUST complete each phase before proceeding to the next.
6449
65-- For regressions: use `git bisect` to find the breaking commit.
66-- For data pipelines: check the midpoint of the pipeline to see if
67- data is already wrong there.
68-- For long functions: add an assertion or log at the midpoint.
50+### Phase 1: Root Cause Investigation
6951
70-**Backward reasoning**
71-Start at the point of failure (the crash, wrong output, bad state).
72-Ask: what produced this value? Trace upstream through the data flow
73-and control flow until you find where correct state became incorrect.
52+**BEFORE attempting ANY fix:**
7453
75-**Differential analysis**
76-Compare a working case to a failing case. What differs between them?
77-Minimize both cases until the difference is as small as possible. The
78-remaining difference contains or points to the defect.
54+1. **Read Error Messages Carefully**
55+ - Don't skip past errors or warnings
56+ - They often contain the exact solution
57+ - Read stack traces completely
58+ - Note line numbers, file paths, error codes
7959
80-**Forward tracing**
81-When you have a suspect area, step through it mentally or with a
82-debugger. Track the actual values at each step against what you
83-expect. The first divergence is your lead.
60+2. **Reproduce Consistently**
61+ - Can you trigger it reliably?
62+ - What are the exact steps?
63+ - Does it happen every time?
64+ - If not reproducible → gather more data, don't guess
8465
85-### Phase 3 — Hypothesize and test
66+3. **Check Recent Changes**
67+ - What changed that could cause this?
68+ - Git diff, recent commits
69+ - New dependencies, config changes
70+ - Environmental differences
8671
87-Once you have a suspect location:
72+4. **Gather Evidence in Multi-Component Systems**
8873
89-1. **State your hypothesis explicitly.** "I believe the bug is in
90- function X because Y, and the root cause is Z."
91-2. **Design a test.** What observation would confirm or refute this
92- hypothesis? Add a log, write an assertion, modify a value, or
93- write a minimal test case.
94-3. **Run the test.** Observe the result.
95-4. **Evaluate.** If confirmed, proceed to Phase 4. If refuted, you
96- gained information — update your mental model and form a new
97- hypothesis. Return to Phase 2 if needed.
74+ **WHEN system has multiple components (CI → build → signing, API → service → database):**
9875
99-Change only one thing at a time. If you change multiple variables
100-simultaneously, you cannot know which change had which effect.
76+ **BEFORE proposing fixes, add diagnostic instrumentation:**
77+ ```
78+ For EACH component boundary:
79+ - Log what data enters component
80+ - Log what data exits component
81+ - Verify environment/config propagation
82+ - Check state at each layer
10183
102-### Phase 4 — Fix
84+ Run once to gather evidence showing WHERE it breaks
85+ THEN analyze evidence to identify failing component
86+ THEN investigate that specific component
87+ ```
10388
104-1. **Fix the root cause**, not the symptom. Ask: why did this defect
105- exist? Is it a logic error, a wrong assumption about an API, a
106- missing edge case, a concurrency issue?
107-2. **Verify the fix.** Run the reproduction from Phase 1. It must
108- now pass. Run the existing test suite. Nothing that previously
109- passed should now fail.
110-3. **Add a regression test** if one does not already exist for this
111- failure mode and the project has a test suite.
89+ **Example (multi-layer system):**
90+ ```bash
91+ # Layer 1: Workflow
92+ echo "=== Secrets available in workflow: ==="
93+ echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
11294
113-### Phase 5 — Validate
95+ # Layer 2: Build script
96+ echo "=== Env vars in build script: ==="
97+ env | grep IDENTITY || echo "IDENTITY not in environment"
11498
115-Before declaring the bug fixed:
99+ # Layer 3: Signing script
100+ echo "=== Keychain state: ==="
101+ security list-keychains
102+ security find-identity -v
116103
117-- Confirm the specific reproduction case passes.
118-- Confirm the broader test suite passes.
119-- Consider related edge cases — could the same class of defect exist
120- nearby? Check, but do not go on a fishing expedition.
104+ # Layer 4: Actual signing
105+ codesign --sign "$IDENTITY" --verbose=4 "$APP"
106+ ```
121107
122-If the fix does not hold under these checks, return to Phase 3.
108+ **This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)
109+
110+5. **Trace Data Flow**
123111
124-## Audit trail
112+ **WHEN error is deep in call stack:**
125113
126-Keep a running log of your debugging session:
114+ See `root-cause-tracing.md` in this directory for the complete backward tracing technique.
127115
128-- What you observed
129-- What hypothesis you formed
130-- What test you ran
131-- What the result was
116+ **Quick version:**
117+ - Where does bad value originate?
118+ - What called this with bad value?
119+ - Keep tracing up until you find the source
120+ - Fix at source, not at symptom
132121
133-This prevents going in circles and helps if you need to hand off to
134-someone else or revisit the issue later.
122+### Phase 2: Pattern Analysis
123+
124+**Find the pattern before fixing:**
125+
126+1. **Find Working Examples**
127+ - Locate similar working code in same codebase
128+ - What works that's similar to what's broken?
129+
130+2. **Compare Against References**
131+ - If implementing pattern, read reference implementation COMPLETELY
132+ - Don't skim - read every line
133+ - Understand the pattern fully before applying
134+
135+3. **Identify Differences**
136+ - What's different between working and broken?
137+ - List every difference, however small
138+ - Don't assume "that can't matter"
135139
136-## Anti-patterns to avoid
140+4. **Understand Dependencies**
141+ - What other components does this need?
142+ - What settings, config, environment?
143+ - What assumptions does it make?
137144
138-- **Shotgun debugging.** Making random changes hoping something sticks.
139- Every change should be driven by a hypothesis.
140-- **Not reading the error.** The error message is data. It is often
141- the single most informative piece of evidence you have.
142-- **Fixing symptoms.** Wrapping a crash in a try/catch is not a fix.
143- Suppressing a warning is not a fix. Find the root cause.
144-- **Changing multiple things at once.** You lose the ability to reason
145- about cause and effect.
146-- **Trusting assumptions.** Check the plug. Is the right version
147- running? Is the config correct? Is the file actually being loaded?
148- Verify the basics before diving deep.
149-- **Theorizing without looking.** When in doubt, add instrumentation
150- and observe. Data beats speculation.
151-
152-## When you are stuck
145+### Phase 3: Hypothesis and Testing
153146
154-- **Get a fresh perspective.** Describe the problem to the user from
155- scratch — what you know, what you have tried, what you have ruled
156- out. Ask if they have context you are missing.
157-- **Question your assumptions.** List every assumption you are making.
158- Test the least certain one.
159-- **Widen the search.** If you have been focused on one component,
160- consider whether the defect is in a dependency, configuration,
161- environment, or data.
162-
163-## Output format
147+**Scientific method:**
164148
165-When reporting your debugging findings, use this structure:
149+1. **Form Single Hypothesis**
150+ - State clearly: "I think X is the root cause because Y"
151+ - Write it down
152+ - Be specific, not vague
153+
154+2. **Test Minimally**
155+ - Make the SMALLEST possible change to test hypothesis
156+ - One variable at a time
157+ - Don't fix multiple things at once
158+
159+3. **Verify Before Continuing**
160+ - Did it work? Yes → Phase 4
161+ - Didn't work? Form NEW hypothesis
162+ - DON'T add more fixes on top
163+
164+4. **When You Don't Know**
165+ - Say "I don't understand X"
166+ - Don't pretend to know
167+ - Ask for help
168+ - Research more
169+
170+### Phase 4: Implementation
166171
167-```
168-**Symptom**: [what the user reported or what you observed]
172+**Fix the root cause, not the symptom:**
173+
174+1. **Create Failing Test Case**
175+ - Simplest possible reproduction
176+ - Automated test if possible
177+ - One-off test script if no framework
178+ - MUST have before fixing
179+ - Use the `superpowers:test-driven-development` skill for writing proper failing tests
169180
170-**Reproduction**: [minimal steps to trigger the failure]
181+2. **Implement Single Fix**
182+ - Address the root cause identified
183+ - ONE change at a time
184+ - No "while I'm here" improvements
185+ - No bundled refactoring
171186
172-**Root cause**: [the defect — what, where, and why]
187+3. **Verify Fix**
188+ - Test passes now?
189+ - No other tests broken?
190+ - Issue actually resolved?
173191
174-**Fix**: [what you changed and why it addresses the root cause]
192+4. **If Fix Doesn't Work**
193+ - STOP
194+ - Count: How many fixes have you tried?
195+ - If < 3: Return to Phase 1, re-analyze with new information
196+ - **If ≥ 3: STOP and question the architecture (step 5 below)**
197+ - DON'T attempt Fix #4 without architectural discussion
175198
176-**Verification**: [what tests confirm the fix]
177-```
199+5. **If 3+ Fixes Failed: Question Architecture**
200+
201+ **Pattern indicating architectural problem:**
202+ - Each fix reveals new shared state/coupling/problem in different place
203+ - Fixes require "massive refactoring" to implement
204+ - Each fix creates new symptoms elsewhere
205+
206+ **STOP and question fundamentals:**
207+ - Is this pattern fundamentally sound?
208+ - Are we "sticking with it through sheer inertia"?
209+ - Should we refactor architecture vs. continue fixing symptoms?
210+
211+ **Discuss with your human partner before attempting more fixes**
212+
213+ This is NOT a failed hypothesis - this is a wrong architecture.
214+
215+## Red Flags - STOP and Follow Process
216+
217+If you catch yourself thinking:
218+- "Quick fix for now, investigate later"
219+- "Just try changing X and see if it works"
220+- "Add multiple changes, run tests"
221+- "Skip the test, I'll manually verify"
222+- "It's probably X, let me fix that"
223+- "I don't fully understand but this might work"
224+- "Pattern says X but I'll adapt it differently"
225+- "Here are the main problems: [lists fixes without investigation]"
226+- Proposing solutions before tracing data flow
227+- **"One more fix attempt" (when already tried 2+)**
228+- **Each fix reveals new problem in different place**
229+
230+**ALL of these mean: STOP. Return to Phase 1.**
231+
232+**If 3+ fixes failed:** Question the architecture (see Phase 4.5)
233+
234+## your human partner's Signals You're Doing It Wrong
235+
236+**Watch for these redirections:**
237+- "Is that not happening?" - You assumed without verifying
238+- "Will it show us...?" - You should have added evidence gathering
239+- "Stop guessing" - You're proposing fixes without understanding
240+- "Ultrathink this" - Question fundamentals, not just symptoms
241+- "We're stuck?" (frustrated) - Your approach isn't working
242+
243+**When you see these:** STOP. Return to Phase 1.
244+
245+## Common Rationalizations
246+
247+| Excuse | Reality |
248+|--------|---------|
Cut short: this commit's diff is over 1000 lines.