xonecas.io
64875
Mc

mini-coder / commit b651d24

chore: Benchmark loop and docs

xonecas2026-04-19 b651d24bfd3723bbd457ae244885802be5cbe234 parent 3beb946

2 files changed

BENCHMARK.md+73−11
@@ -4,6 +4,8 @@ Use this as the default tuning loop for `mini-coder` on Terminal-Bench.
44
55 The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small prompt / loop / tool-use changes can be judged quickly, then promote only the promising ones to bigger runs.
66
7+Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.
8+
79 ## Principles
810
911 - Keep changes small.
@@ -11,6 +13,9 @@ The goal is not to rerun the whole benchmark after every change. The goal is to
1113 - Compare against a fresh baseline, not an old leaderboard run.
1214 - Use fast suites for iteration, broad suites for promotion.
1315 - Keep structured `mc --json` logs in trial artifacts so behavior can be analyzed.
16+- Optimize for general coding-agent behavior first.
17+- Use benchmark failures to extract general behavior gaps, not to encode benchmark lore into the agent.
18+- Prefer generic improvements over task-named patches, reminders, or stop-time nudges.
1419
1520 ## Suites
1621
@@ -115,19 +120,52 @@ That gives a fresh local baseline for:
115120
116121 Do not compare a new change only against an old run from days ago if a fresh same-HEAD baseline is available.
117122
123+## Experiment quality bar
124+
125+Before changing code, write the hypothesis in two layers:
126+
127+1. the benchmark symptom
128+2. the general coding-agent behavior gap behind it
129+
130+Only run an experiment if you can answer all of these:
131+
132+- what general behavior is being improved?
133+- why should that help outside Terminal-Bench?
134+- what would make this change obviously overfit?
135+
136+Good experiment themes:
137+
138+- verification-equivalence before completion
139+- artifact-grounded verification of final outputs
140+- preferring a task-named local source of truth over approximations
141+- reducing shell thrash before first meaningful verification
142+
143+Reject or redesign experiments that:
144+
145+- depend on benchmark-specific task names, file names, package names, or tool names in product logic
146+- inject reminders or guards keyed to one benchmark noun unless that rule maps cleanly to a real product behavior
147+- only make sense because a particular verifier is known
148+- cannot be explained without citing a single task transcript
149+
150+Hard rule:
151+
152+- no task-specific nouns in agent logic unless they map to a real product feature
153+
118154 ## Iteration loop
119155
120156 For each change:
121157
122158 1. inspect the last fast / focused failures
123-2. pick **one** failure pattern
124-3. form **one** narrow hypothesis
125-4. make **one** small change
126-5. run:
159+2. translate them into **one** general behavior gap
160+3. reject benchmark-shaped ideas; if you cannot phrase the change without task-specific nouns, keep diagnosing
161+4. if the change depends on a dynamic trigger, confirm that the trigger actually appears in the target failures
162+5. form **one** narrow hypothesis
163+6. make **one** small change
164+7. run:
127165 - guardrails
128166 - fast suite
129-6. compare to baseline
130-7. decide:
167+8. compare to baseline
168+9. decide:
131169 - keep
132170 - revert
133171 - refine
@@ -157,6 +195,7 @@ Keep a change if:
157195 - the fast suite improves materially
158196 - guardrails do not regress
159197 - the result repeats on another fast-suite rerun
198+- the mechanism is still a general agent-quality improvement, not just a benchmark-specific patch
160199
161200 Revert a change if:
162201
@@ -167,6 +206,11 @@ Call it inconclusive if:
167206
168207 - the delta is tiny and does not repeat
169208
209+Also prefer:
210+
211+- a smaller, more general improvement over a larger but obviously benchmark-specific patch
212+- a clearly exercised mechanism over a theory that never fired in the target runs
213+
170214 ## Failure buckets to optimize against
171215
172216 Use these as the main buckets when inspecting logs:
@@ -179,13 +223,21 @@ Examples:
179223 - right output plus extra junk
180224 - correct core work, but violates a strict file / in-place requirement
181225
182-### Didn’t verify early enough
226+### Verification-equivalence / premature completion
183227
184228 Examples:
185229
186-- waited too long to run a test
230+- waited too long to run a meaningful check
187231 - never checked the exact contract
188-- kept exploring before the first meaningful verification
232+- stopped after a weaker non-equivalent check
233+- verified the intended design instead of the written artifact
234+
235+### Source-of-truth selection gaps
236+
237+Examples:
238+
239+- task names an exact local tool/package/interface, but the agent uses an approximation instead
240+- lower-level library reasoning replaces package-local or task-local semantics
189241
190242 ### Over-exploration / shell thrash
191243
@@ -235,6 +287,9 @@ With those artifacts, analyze things like:
235287 - time to first edit
236288 - number of `shell` / `read` / `grep` / `edit` calls
237289 - whether the agent ran a verifier-like shell command
290+- whether it used the named local source of truth when one was available
291+- whether it verified the final artifact it actually wrote
292+- whether a dynamic intervention visibly exercised
238293 - whether it had a local pass before ending
239294 - whether it kept changing files after a pass-worthy state
240295 - whether it left extra artifacts
@@ -336,11 +391,18 @@ PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
336391
337392 ## Minimal experiment log format
338393
339-Keep this in the PROGRESS.md, a final summary for each change:
394+Keep this in `PROGRESS.md`, a final summary for each completed change:
340395
396+- benchmark symptom
397+- general behavior gap
398+- why this should help outside Terminal-Bench
399+- overfitting risk / why this is still general
400+- trigger evidence, if relevant
341401 - hypothesis
342402 - files changed
403+- local verification
343404 - guardrail result
344405 - fast-suite result
345406 - focused-suite result, if run
346-- keep / revert / refine
407+- mechanism exercised?
408+- keep / revert / refine (Make the decision very visible in the file).
PROGRESS.md+3−1
@@ -1,3 +1,5 @@
11 # CURRENT TASK PROGRESS
22
3-- Started benchmark loop.
3+- Benchmark loop stopped.
4+- In-progress experiment changes reverted.
5+- Progress reset for a fresh benchmark restart.