@@ -19,17 +19,20 @@ The goal is not to rerun the whole benchmark after every change. The goal is to
1919 Use this to catch regressions on known-good behavior.
2020
2121 Tasks:
22+
2223 - `cancel-async-tasks`
2324 - `bn-fit-modify`
2425 - `sparql-university`
2526 - `extract-elf`
2627
2728 Settings:
29+
2830 - `1` attempt
2931 - `2` concurrent
3032 - `0` retries
3133
3234 Estimated wall time:
35+
3336 - about `8–10 minutes`
3437
3538 ### 2. Fast suite
@@ -37,6 +40,7 @@ Estimated wall time:
3740 Use this after almost every small change.
3841
3942 Tasks:
43+
4044 - `polyglot-rust-c`
4145 - `overfull-hbox`
4246 - `gcode-to-text`
@@ -45,11 +49,13 @@ Tasks:
4549 - `torch-tensor-parallelism`
4650
4751 Settings:
52+
4853 - `2` attempts
4954 - `2` concurrent
5055 - `0` retries
5156
5257 Estimated wall time:
58+
5359 - about `44 minutes`
5460
5561 ### 3. Focused suite
@@ -57,6 +63,7 @@ Estimated wall time:
5763 Use this only if a change looks good on the fast suite.
5864
5965 Tasks:
66+
6067 - `polyglot-rust-c`
6168 - `overfull-hbox`
6269 - `gcode-to-text`
@@ -67,11 +74,13 @@ Tasks:
6774 - `qemu-alpine-ssh`
6875
6976 Settings:
77+
7078 - `2` attempts
7179 - `2` concurrent
7280 - `0` retries
7381
7482 Estimated wall time:
83+
7584 - about `1.1 hours`
7685
7786 ### 4. Broad promotion suite
@@ -79,11 +88,13 @@ Estimated wall time:
7988 Use this after 1–3 promising changes, or overnight.
8089
8190 Options:
91+
8292 - rerun recent failures
8393 - wider local failure suite
8494 - full leaderboard-style run
8595
8696 Typical settings for rerunning recent failures:
97+
8798 - `4` attempts
8899 - `2` concurrent
89100 - `0` retries
@@ -97,6 +108,7 @@ Before changing code:
97108 3. run the fast suite again
98109
99110 That gives a fresh local baseline for:
111+
100112 - pass count
101113 - runtime
102114 - variance / noise
@@ -121,10 +133,12 @@ For each change:
121133 - refine
122134
123135 Only run the focused suite if:
136+
124137 - the fast suite improved, and
125138 - the guardrails did not regress
126139
127140 Only run the broad promotion suite if:
141+
128142 - the focused suite also looks good, or
129143 - enough promising changes have accumulated to justify it
130144
@@ -133,20 +147,24 @@ Only run the broad promotion suite if:
133147 Treat the fast suite as a noisy but useful signal.
134148
135149 For the 12-trial fast suite:
150+
136151 - `+2` or more passes: probably meaningful improvement
137152 - `-2` or more passes: probably meaningful regression
138153 - `±1`: likely noise unless it repeats
139154
140155 Keep a change if:
156+
141157 - the fast suite improves materially
142158 - guardrails do not regress
143159 - the result repeats on another fast-suite rerun
144160
145161 Revert a change if:
162+
146163 - it clearly loses fast-suite passes, or
147164 - it regresses guardrails
148165
149166 Call it inconclusive if:
167+
150168 - the delta is tiny and does not repeat
151169
152170 ## Failure buckets to optimize against
@@ -154,31 +172,41 @@ Call it inconclusive if:
154172 Use these as the main buckets when inspecting logs:
155173
156174 ### Exact-contract / cleanup misses
175+
157176 Examples:
177+
158178 - right artifact, wrong path
159179 - right output plus extra junk
160180 - correct core work, but violates a strict file / in-place requirement
161181
162182 ### Didn’t verify early enough
183+
163184 Examples:
185+
164186 - waited too long to run a test
165187 - never checked the exact contract
166188 - kept exploring before the first meaningful verification
167189
168190 ### Over-exploration / shell thrash
191+
169192 Examples:
193+
170194 - too many shell / read steps before first write
171195 - repeated discovery commands without narrowing the problem
172196
173197 ### Persistent correctness gaps
198+
174199 Examples:
200+
175201 - `torch-tensor-parallelism`
176202 - `filter-js-from-html`
177203 - `configure-git-webserver`
178204
179205 ### Long-horizon / timeout-heavy tasks
206+
180207 These are useful for promotion runs, not tight inner loops.
181208 Examples:
209+
182210 - `gpt2-codegolf`
183211 - `query-optimize`
184212 - `winning-avg-corewars`
@@ -189,9 +217,11 @@ Examples:
189217 Behavior analysis depends on structured agent logs.
190218
191219 Keep wrappers on:
220+
192221 - `mc --json -p ...`
193222
194223 Per trial, keep:
224+
195225 - result JSON
196226 - verifier output
197227 - exception type
@@ -200,6 +230,7 @@ Per trial, keep:
200230 - timestamps
201231
202232 With those artifacts, analyze things like:
233+
203234 - time to first tool call
204235 - time to first edit
205236 - number of `shell` / `read` / `grep` / `edit` calls
@@ -208,14 +239,6 @@ With those artifacts, analyze things like:
208239 - whether it kept changing files after a pass-worthy state
209240 - whether it left extra artifacts
210241
211-## Recommended first optimization targets
212-
213-Start with:
214-1. exact-contract and stopping discipline
215-2. earlier verification
216-3. less wandering during discovery
217-4. loop nudges that reduce shell thrash without adding a hard step cap
218-
219242 ## Commands
220243
221244 ### Guardrail suite
@@ -311,19 +334,9 @@ done
311334 PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
312335 ```
313336
314-## Session-start prompt
315-
316-In a new session, the quickest way to start is:
317-
318-- read `BENCHMARK.md`
319-- run the guardrail suite
320-- run the fast suite twice for baseline
321-- inspect failures and grouped behavior patterns
322-- propose one small hypothesis and one small change
323-
324337 ## Minimal experiment log format
325338
326-Keep this in the session or in the final summary for each change:
339+Keep this in the PROGRESS.md, a final summary for each change:
327340
328341 - hypothesis
329342 - files changed