@@ -1,10 +1,16 @@
1+# CORE GOAL
2+
3+**Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.**
4+
5+- Do not add fixes to fix an issue with specific terminal bench evals, focus on improving the agent's behaviour.
6+- Benchmarks run headless mode in one shot. Make sure you use `tmux` often to check that the TUI and multiturn UX is still
7+ good.
8+
19 # Benchmark workflow
210
311 Use this as the default tuning loop for `mini-coder` on Terminal-Bench.
412
5-The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small prompt / loop / tool-use changes can be judged quickly, then promote only the promising ones to bigger runs.
6-
7-Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.
13+The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small changes can be judged quickly, then promote only the promising ones to bigger runs.
814
915 ## Principles
1016
@@ -19,39 +25,8 @@ Terminal-Bench is a signal, not the product. The target is a better coding agent
1925
2026 ## Suites
2127
22-### 1. Guardrail suite
23-
24-Use this to catch regressions on known-good behavior.
25-
26-Tasks:
27-
28-- `cancel-async-tasks`
29-- `bn-fit-modify`
30-- `sparql-university`
31-- `extract-elf`
32-
33-Settings:
34-
35-- `1` attempt
36-- `2` concurrent
37-- `0` retries
38-
39-Estimated wall time:
40-
41-- about `8–10 minutes`
42-
43-### 2. Fast suite
44-
45-Use this after almost every small change.
46-
47-Tasks:
48-
49-- `polyglot-rust-c`
50-- `overfull-hbox`
51-- `gcode-to-text`
52-- `mteb-retrieve`
53-- `dna-insert`
54-- `torch-tensor-parallelism`
28+There is a full 89 test baseline run with 2 attemps in the teminal-bench folder. Use the evals in it
29+to determine your fast evals to start your optimization process and iterations.
5530
5631 Settings:
5732
@@ -59,67 +34,6 @@ Settings:
5934 - `2` concurrent
6035 - `0` retries
6136
62-Estimated wall time:
63-
64-- about `44 minutes`
65-
66-### 3. Focused suite
67-
68-Use this only if a change looks good on the fast suite.
69-
70-Tasks:
71-
72-- `polyglot-rust-c`
73-- `overfull-hbox`
74-- `gcode-to-text`
75-- `mteb-retrieve`
76-- `torch-tensor-parallelism`
77-- `filter-js-from-html`
78-- `configure-git-webserver`
79-- `qemu-alpine-ssh`
80-
81-Settings:
82-
83-- `2` attempts
84-- `2` concurrent
85-- `0` retries
86-
87-Estimated wall time:
88-
89-- about `1.1 hours`
90-
91-### 4. Broad promotion suite
92-
93-Use this after 1–3 promising changes, or overnight.
94-
95-Options:
96-
97-- rerun recent failures
98-- wider local failure suite
99-- full leaderboard-style run
100-
101-Typical settings for rerunning recent failures:
102-
103-- `4` attempts
104-- `2` concurrent
105-- `0` retries
106-
107-## Baseline procedure
108-
109-Before changing code:
110-
111-1. run the guardrail suite
112-2. run the fast suite
113-3. run the fast suite again
114-
115-That gives a fresh local baseline for:
116-
117-- pass count
118-- runtime
119-- variance / noise
120-
121-Do not compare a new change only against an old run from days ago if a fresh same-HEAD baseline is available.
122-
12337 ## Experiment quality bar
12438
12539 Before changing code, write the hypothesis in two layers:
@@ -133,13 +47,6 @@ Only run an experiment if you can answer all of these:
13347 - why should that help outside Terminal-Bench?
13448 - what would make this change obviously overfit?
13549
136-Good experiment themes:
137-
138-- verification-equivalence before completion
139-- artifact-grounded verification of final outputs
140-- preferring a task-named local source of truth over approximations
141-- reducing shell thrash before first meaningful verification
142-
14350 Reject or redesign experiments that:
14451
14552 - depend on benchmark-specific task names, file names, package names, or tool names in product logic
@@ -162,108 +69,13 @@ For each change:
16269 5. form **one** narrow hypothesis
16370 6. make **one** small change
16471 7. run:
165- - guardrails
16672 - fast suite
167-8. compare to baseline
73+8. compare to baseline or your reference run
16874 9. decide:
16975 - keep
17076 - revert
17177 - refine
17278
173-Only run the focused suite if:
174-
175-- the fast suite improved, and
176-- the guardrails did not regress
177-
178-Only run the broad promotion suite if:
179-
180-- the focused suite also looks good, or
181-- enough promising changes have accumulated to justify it
182-
183-## Decision rules
184-
185-Treat the fast suite as a noisy but useful signal.
186-
187-For the 12-trial fast suite:
188-
189-- `+2` or more passes: probably meaningful improvement
190-- `-2` or more passes: probably meaningful regression
191-- `±1`: likely noise unless it repeats
192-
193-Keep a change if:
194-
195-- the fast suite improves materially
196-- guardrails do not regress
197-- the result repeats on another fast-suite rerun
198-- the mechanism is still a general agent-quality improvement, not just a benchmark-specific patch
199-
200-Revert a change if:
201-
202-- it clearly loses fast-suite passes, or
203-- it regresses guardrails
204-
205-Call it inconclusive if:
206-
207-- the delta is tiny and does not repeat
208-
209-Also prefer:
210-
211-- a smaller, more general improvement over a larger but obviously benchmark-specific patch
212-- a clearly exercised mechanism over a theory that never fired in the target runs
213-
214-## Failure buckets to optimize against
215-
216-Use these as the main buckets when inspecting logs:
217-
218-### Exact-contract / cleanup misses
219-
220-Examples:
221-
222-- right artifact, wrong path
223-- right output plus extra junk
224-- correct core work, but violates a strict file / in-place requirement
225-
226-### Verification-equivalence / premature completion
227-
228-Examples:
229-
230-- waited too long to run a meaningful check
231-- never checked the exact contract
232-- stopped after a weaker non-equivalent check
233-- verified the intended design instead of the written artifact
234-
235-### Source-of-truth selection gaps
236-
237-Examples:
238-
239-- task names an exact local tool/package/interface, but the agent uses an approximation instead
240-- lower-level library reasoning replaces package-local or task-local semantics
241-
242-### Over-exploration / shell thrash
243-
244-Examples:
245-
246-- too many shell / read steps before first write
247-- repeated discovery commands without narrowing the problem
248-
249-### Persistent correctness gaps
250-
251-Examples:
252-
253-- `torch-tensor-parallelism`
254-- `filter-js-from-html`
255-- `configure-git-webserver`
256-
257-### Long-horizon / timeout-heavy tasks
258-
259-These are useful for promotion runs, not tight inner loops.
260-Examples:
261-
262-- `gpt2-codegolf`
263-- `query-optimize`
264-- `winning-avg-corewars`
265-- `train-fasttext`
266-
26779 ## Behavior analysis requirements
26880
26981 Behavior analysis depends on structured agent logs.
@@ -281,128 +93,15 @@ Per trial, keep:
28193 - structured `agent/mini-coder.ndjson`
28294 - timestamps
28395
284-With those artifacts, analyze things like:
285-
286-- time to first tool call
287-- time to first edit
288-- number of `shell` / `read` / `grep` / `edit` calls
289-- whether the agent ran a verifier-like shell command
290-- whether it used the named local source of truth when one was available
291-- whether it verified the final artifact it actually wrote
292-- whether a dynamic intervention visibly exercised
293-- whether it had a local pass before ending
294-- whether it kept changing files after a pass-worthy state
295-- whether it left extra artifacts
296-
297-## Commands
298-
299-### Guardrail suite
300-
301-```bash
302-GUARDRAIL_TASKS=(
303- cancel-async-tasks
304- bn-fit-modify
305- sparql-university
306- extract-elf
307-)
308-
309-cmd=(
310- harbor run -y
311- --job-name local-guardrails-$(date +%F__%H-%M-%S)
312- --jobs-dir "$PWD/terminal-bench/jobs"
313- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
314- --model openai-codex/gpt-5.4
315- --dataset terminal-bench@2.0
316- --n-attempts 1
317- --n-concurrent 2
318- --max-retries 0
319-)
320-
321-for t in "${GUARDRAIL_TASKS[@]}"; do
322- cmd+=(--include-task-name "$t")
323-done
324-
325-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
326-```
327-
328-### Fast suite
329-
330-```bash
331-FAST_TASKS=(
332- polyglot-rust-c
333- overfull-hbox
334- gcode-to-text
335- mteb-retrieve
336- dna-insert
337- torch-tensor-parallelism
338-)
339-
340-cmd=(
341- harbor run -y
342- --job-name local-fast-$(date +%F__%H-%M-%S)
343- --jobs-dir "$PWD/terminal-bench/jobs"
344- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
345- --model openai-codex/gpt-5.4
346- --dataset terminal-bench@2.0
347- --n-attempts 2
348- --n-concurrent 2
349- --max-retries 0
350-)
351-
352-for t in "${FAST_TASKS[@]}"; do
353- cmd+=(--include-task-name "$t")
354-done
355-
356-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
357-```
358-
359-### Focused suite
360-
361-```bash
362-FOCUS_TASKS=(
363- polyglot-rust-c
364- overfull-hbox
365- gcode-to-text
366- mteb-retrieve
367- torch-tensor-parallelism
368- filter-js-from-html
369- configure-git-webserver
370- qemu-alpine-ssh
371-)
372-
373-cmd=(
374- harbor run -y
375- --job-name local-focus-$(date +%F__%H-%M-%S)
376- --jobs-dir "$PWD/terminal-bench/jobs"
377- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
378- --model openai-codex/gpt-5.4
379- --dataset terminal-bench@2.0
380- --n-attempts 2
381- --n-concurrent 2
382- --max-retries 0
383-)
384-
385-for t in "${FOCUS_TASKS[@]}"; do
386- cmd+=(--include-task-name "$t")
387-done
388-
389-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
390-```
391-
39296 ## Minimal experiment log format
39397
98+You are running in a loop, make sure to keep your progress tracked so you
99+can continue between loop iterations, this is to avoid context pressure.:w
394100 Keep this in `PROGRESS.md`, a final summary for each completed change:
395101
396102 - benchmark symptom
397103 - general behavior gap
398104 - why this should help outside Terminal-Bench
399-- overfitting risk / why this is still general
400-- trigger evidence, if relevant
401105 - hypothesis
402-- files changed
403-- local verification
404-- guardrail result
405-- fast-suite result
406-- focused-suite result, if run
407-- mechanism exercised?
106+- verification method
408107 - keep / revert / refine (Make the decision very visible in the file).