xonecas.io
64875
Mc

mini-coder / commit ccb219b

chore(docs): benchmark prep

xonecas2026-04-20 ccb219bfae466569249b4a78bd7abf8d8a8636cc parent 0d42cc7

3 files changed

BENCHMARK.md+15−316
@@ -1,10 +1,16 @@
1+# CORE GOAL
2+
3+**Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.**
4+
5+- Do not add fixes to fix an issue with specific terminal bench evals, focus on improving the agent's behaviour.
6+- Benchmarks run headless mode in one shot. Make sure you use `tmux` often to check that the TUI and multiturn UX is still
7+ good.
8+
19 # Benchmark workflow
210
311 Use this as the default tuning loop for `mini-coder` on Terminal-Bench.
412
5-The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small prompt / loop / tool-use changes can be judged quickly, then promote only the promising ones to bigger runs.
6-
7-Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.
13+The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small changes can be judged quickly, then promote only the promising ones to bigger runs.
814
915 ## Principles
1016
@@ -19,39 +25,8 @@ Terminal-Bench is a signal, not the product. The target is a better coding agent
1925
2026 ## Suites
2127
22-### 1. Guardrail suite
23-
24-Use this to catch regressions on known-good behavior.
25-
26-Tasks:
27-
28-- `cancel-async-tasks`
29-- `bn-fit-modify`
30-- `sparql-university`
31-- `extract-elf`
32-
33-Settings:
34-
35-- `1` attempt
36-- `2` concurrent
37-- `0` retries
38-
39-Estimated wall time:
40-
41-- about `8–10 minutes`
42-
43-### 2. Fast suite
44-
45-Use this after almost every small change.
46-
47-Tasks:
48-
49-- `polyglot-rust-c`
50-- `overfull-hbox`
51-- `gcode-to-text`
52-- `mteb-retrieve`
53-- `dna-insert`
54-- `torch-tensor-parallelism`
28+There is a full 89 test baseline run with 2 attemps in the teminal-bench folder. Use the evals in it
29+to determine your fast evals to start your optimization process and iterations.
5530
5631 Settings:
5732
@@ -59,67 +34,6 @@ Settings:
5934 - `2` concurrent
6035 - `0` retries
6136
62-Estimated wall time:
63-
64-- about `44 minutes`
65-
66-### 3. Focused suite
67-
68-Use this only if a change looks good on the fast suite.
69-
70-Tasks:
71-
72-- `polyglot-rust-c`
73-- `overfull-hbox`
74-- `gcode-to-text`
75-- `mteb-retrieve`
76-- `torch-tensor-parallelism`
77-- `filter-js-from-html`
78-- `configure-git-webserver`
79-- `qemu-alpine-ssh`
80-
81-Settings:
82-
83-- `2` attempts
84-- `2` concurrent
85-- `0` retries
86-
87-Estimated wall time:
88-
89-- about `1.1 hours`
90-
91-### 4. Broad promotion suite
92-
93-Use this after 1–3 promising changes, or overnight.
94-
95-Options:
96-
97-- rerun recent failures
98-- wider local failure suite
99-- full leaderboard-style run
100-
101-Typical settings for rerunning recent failures:
102-
103-- `4` attempts
104-- `2` concurrent
105-- `0` retries
106-
107-## Baseline procedure
108-
109-Before changing code:
110-
111-1. run the guardrail suite
112-2. run the fast suite
113-3. run the fast suite again
114-
115-That gives a fresh local baseline for:
116-
117-- pass count
118-- runtime
119-- variance / noise
120-
121-Do not compare a new change only against an old run from days ago if a fresh same-HEAD baseline is available.
122-
12337 ## Experiment quality bar
12438
12539 Before changing code, write the hypothesis in two layers:
@@ -133,13 +47,6 @@ Only run an experiment if you can answer all of these:
13347 - why should that help outside Terminal-Bench?
13448 - what would make this change obviously overfit?
13549
136-Good experiment themes:
137-
138-- verification-equivalence before completion
139-- artifact-grounded verification of final outputs
140-- preferring a task-named local source of truth over approximations
141-- reducing shell thrash before first meaningful verification
142-
14350 Reject or redesign experiments that:
14451
14552 - depend on benchmark-specific task names, file names, package names, or tool names in product logic
@@ -162,108 +69,13 @@ For each change:
16269 5. form **one** narrow hypothesis
16370 6. make **one** small change
16471 7. run:
165- - guardrails
16672 - fast suite
167-8. compare to baseline
73+8. compare to baseline or your reference run
16874 9. decide:
16975 - keep
17076 - revert
17177 - refine
17278
173-Only run the focused suite if:
174-
175-- the fast suite improved, and
176-- the guardrails did not regress
177-
178-Only run the broad promotion suite if:
179-
180-- the focused suite also looks good, or
181-- enough promising changes have accumulated to justify it
182-
183-## Decision rules
184-
185-Treat the fast suite as a noisy but useful signal.
186-
187-For the 12-trial fast suite:
188-
189-- `+2` or more passes: probably meaningful improvement
190-- `-2` or more passes: probably meaningful regression
191-- `±1`: likely noise unless it repeats
192-
193-Keep a change if:
194-
195-- the fast suite improves materially
196-- guardrails do not regress
197-- the result repeats on another fast-suite rerun
198-- the mechanism is still a general agent-quality improvement, not just a benchmark-specific patch
199-
200-Revert a change if:
201-
202-- it clearly loses fast-suite passes, or
203-- it regresses guardrails
204-
205-Call it inconclusive if:
206-
207-- the delta is tiny and does not repeat
208-
209-Also prefer:
210-
211-- a smaller, more general improvement over a larger but obviously benchmark-specific patch
212-- a clearly exercised mechanism over a theory that never fired in the target runs
213-
214-## Failure buckets to optimize against
215-
216-Use these as the main buckets when inspecting logs:
217-
218-### Exact-contract / cleanup misses
219-
220-Examples:
221-
222-- right artifact, wrong path
223-- right output plus extra junk
224-- correct core work, but violates a strict file / in-place requirement
225-
226-### Verification-equivalence / premature completion
227-
228-Examples:
229-
230-- waited too long to run a meaningful check
231-- never checked the exact contract
232-- stopped after a weaker non-equivalent check
233-- verified the intended design instead of the written artifact
234-
235-### Source-of-truth selection gaps
236-
237-Examples:
238-
239-- task names an exact local tool/package/interface, but the agent uses an approximation instead
240-- lower-level library reasoning replaces package-local or task-local semantics
241-
242-### Over-exploration / shell thrash
243-
244-Examples:
245-
246-- too many shell / read steps before first write
247-- repeated discovery commands without narrowing the problem
248-
249-### Persistent correctness gaps
250-
251-Examples:
252-
253-- `torch-tensor-parallelism`
254-- `filter-js-from-html`
255-- `configure-git-webserver`
256-
257-### Long-horizon / timeout-heavy tasks
258-
259-These are useful for promotion runs, not tight inner loops.
260-Examples:
261-
262-- `gpt2-codegolf`
263-- `query-optimize`
264-- `winning-avg-corewars`
265-- `train-fasttext`
266-
26779 ## Behavior analysis requirements
26880
26981 Behavior analysis depends on structured agent logs.
@@ -281,128 +93,15 @@ Per trial, keep:
28193 - structured `agent/mini-coder.ndjson`
28294 - timestamps
28395
284-With those artifacts, analyze things like:
285-
286-- time to first tool call
287-- time to first edit
288-- number of `shell` / `read` / `grep` / `edit` calls
289-- whether the agent ran a verifier-like shell command
290-- whether it used the named local source of truth when one was available
291-- whether it verified the final artifact it actually wrote
292-- whether a dynamic intervention visibly exercised
293-- whether it had a local pass before ending
294-- whether it kept changing files after a pass-worthy state
295-- whether it left extra artifacts
296-
297-## Commands
298-
299-### Guardrail suite
300-
301-```bash
302-GUARDRAIL_TASKS=(
303- cancel-async-tasks
304- bn-fit-modify
305- sparql-university
306- extract-elf
307-)
308-
309-cmd=(
310- harbor run -y
311- --job-name local-guardrails-$(date +%F__%H-%M-%S)
312- --jobs-dir "$PWD/terminal-bench/jobs"
313- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
314- --model openai-codex/gpt-5.4
315- --dataset terminal-bench@2.0
316- --n-attempts 1
317- --n-concurrent 2
318- --max-retries 0
319-)
320-
321-for t in "${GUARDRAIL_TASKS[@]}"; do
322- cmd+=(--include-task-name "$t")
323-done
324-
325-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
326-```
327-
328-### Fast suite
329-
330-```bash
331-FAST_TASKS=(
332- polyglot-rust-c
333- overfull-hbox
334- gcode-to-text
335- mteb-retrieve
336- dna-insert
337- torch-tensor-parallelism
338-)
339-
340-cmd=(
341- harbor run -y
342- --job-name local-fast-$(date +%F__%H-%M-%S)
343- --jobs-dir "$PWD/terminal-bench/jobs"
344- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
345- --model openai-codex/gpt-5.4
346- --dataset terminal-bench@2.0
347- --n-attempts 2
348- --n-concurrent 2
349- --max-retries 0
350-)
351-
352-for t in "${FAST_TASKS[@]}"; do
353- cmd+=(--include-task-name "$t")
354-done
355-
356-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
357-```
358-
359-### Focused suite
360-
361-```bash
362-FOCUS_TASKS=(
363- polyglot-rust-c
364- overfull-hbox
365- gcode-to-text
366- mteb-retrieve
367- torch-tensor-parallelism
368- filter-js-from-html
369- configure-git-webserver
370- qemu-alpine-ssh
371-)
372-
373-cmd=(
374- harbor run -y
375- --job-name local-focus-$(date +%F__%H-%M-%S)
376- --jobs-dir "$PWD/terminal-bench/jobs"
377- --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
378- --model openai-codex/gpt-5.4
379- --dataset terminal-bench@2.0
380- --n-attempts 2
381- --n-concurrent 2
382- --max-retries 0
383-)
384-
385-for t in "${FOCUS_TASKS[@]}"; do
386- cmd+=(--include-task-name "$t")
387-done
388-
389-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
390-```
391-
39296 ## Minimal experiment log format
39397
98+You are running in a loop, make sure to keep your progress tracked so you
99+can continue between loop iterations, this is to avoid context pressure.:w
394100 Keep this in `PROGRESS.md`, a final summary for each completed change:
395101
396102 - benchmark symptom
397103 - general behavior gap
398104 - why this should help outside Terminal-Bench
399-- overfitting risk / why this is still general
400-- trigger evidence, if relevant
401105 - hypothesis
402-- files changed
403-- local verification
404-- guardrail result
405-- fast-suite result
406-- focused-suite result, if run
407-- mechanism exercised?
106+- verification method
408107 - keep / revert / refine (Make the decision very visible in the file).
terminal-bench/README.md+2−0
@@ -118,3 +118,5 @@ PYTHONPATH=$PWD/terminal-bench harbor run -y \
118118 --n-concurrent 1 \
119119 --max-retries 0
120120 ```
121+
122+See `TERMINALBENCH_README.md` for more information.
terminal-bench/TERMINALBENCH_README.md+158−0
@@ -0,0 +1,158 @@
1+# Running Terminal-Bench
2+
3+Harbor is installed in this system with `pipx harbor`.
4+You will need to create a custom agent for your implementation.
5+keep the benchmark jobs in `benchmark-jobs/*` folder at the root of the repo.
6+Never run harbor evals with more than 2 concurrent evals, be aware of rate limits.
7+
8+---
9+
10+[Terminal-Bench](https://tbench.ai) is a benchmark for evaluating the performance of agents on terminal-based tasks. Harbor is the official harness for running Terminal-Bench 2.0.
11+
12+To run Terminal-Bench you will first need to install [Harbor](/docs/getting-started). You'll know that it's installed correctly if you're able to run the oracle solutions for Terminal-Bench 2.0. Note that you will first need to install Docker and have it running on your machine:
13+
14+```bash
15+harbor run -d terminal-bench/terminal-bench-2 -a oracle
16+```
17+
18+You should then be able to try any of the more advanced features offered by Harbor, such as running Terminal-Bench with Claude Code on Daytona:
19+
20+```bash
21+export DAYTONA_API_KEY="<your-daytona-api-key>"
22+export ANTHROPIC_API_KEY="<your-anthropic-api-key>"
23+harbor run \
24+ -d terminal-bench/terminal-bench-2 \
25+ -m anthropic/claude-haiku-4-5 \
26+ -a claude-code \
27+ --env daytona \
28+ -n 32
29+```
30+
31+## Testing your own agent
32+
33+See our docs on [agents](/docs/agents) (pasted bellow) for more information on how to test your own agent on Terminal-Bench.
34+
35+## Submitting to the Terminal-Bench leaderboard
36+
37+Leaderboard logs are stored in [this HuggingFace repo](https://huggingface.co/datasets/alexgshaw/terminal-bench-2-leaderboard). To submit your results, open a PR there following the instructions in the README.
38+
39+## Viewing the Terminal-Bench leaderboard
40+
41+You can view the leaderboard [here](https://tbench.ai/leaderboard).
42+
43+---
44+
45+# Agents
46+
47+How to evaluate on existing agents and integrate your own. This is particularly useful for benchmarking your agent, optimizing its prompts, using it as a scaffold for RL, or using it to generate SFT datasets.
48+
49+## Existing agents
50+
51+Harbor comes with most popular agents pre-integrated. You can run the following command and reference the `--agent` flag to see a list of all available agents:
52+
53+```bash
54+harbor run --help
55+```
56+
57+Right now, Harbor includes Terminus-2, Claude Code, Copilot CLI, Codex CLI, Gemini CLI, OpenHands, Mini-SWE-Agent, and more.
58+
59+## Integrating your own agent
60+
61+Harbor supports integrating your own agent without having to modify the Harbor source code.
62+
63+There are two types of agents:
64+
65+1. **External agents** which interface with the environment through the `BaseEnvironment` interface, typically by executing bash commands via the `exec` method.
66+2. **Installed agents** which are agents that are installed directly into the container environment and are executed in headless mode. This is how most agents are integrated and comes with the advantage of bringing custom tools.
67+
68+### External agents
69+
70+To build an external agent, you need to implement the `BaseAgent` interface which involved defining the following methods:
71+
72+```python title="my_external_agent.py"
73+from harbor.agents.base import BaseAgent
74+
75+class MyExternalAgent(BaseAgent):
76+ @staticmethod
77+ def name() -> str:
78+ """The name of the agent."""
79+ pass
80+
81+ def version(self) -> str | None:
82+ """The version of the agent."""
83+ pass
84+
85+ async def setup(self, environment: BaseEnvironment) -> None:
86+ """
87+ Run commands to setup the agent & its tools.
88+ """
89+ pass
90+
91+ async def run(
92+ self,
93+ instruction: str,
94+ environment: BaseEnvironment,
95+ context: AgentContext,
96+ ) -> None:
97+ """
98+ Runs the agent in the environment. Be sure to populate the context with the
99+ results of the agent execution. Ideally, populate the context as the agent
100+ executes in case of a timeout or other error.
101+
102+ Args:
103+ instruction: The task instruction.
104+ environment: The environment in which to complete the task.
105+ context: The context to populate with the results of the agent execution.
106+ """
107+ pass
108+```
109+
110+### Installed agents
111+
112+To build an installed agent, you need to implement the `BaseInstalledAgent` interface which involves defining the following methods:
113+
114+```python title="my_installed_agent.py"
115+from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
116+from harbor.environments.base import BaseEnvironment
117+from harbor.models.agent.context import AgentContext
118+
119+class MyInstalledAgent(BaseInstalledAgent):
120+ async def install(self, environment: BaseEnvironment) -> None:
121+ """
122+ Install the agent in the environment. Use exec_as_root for system
123+ packages and exec_as_agent for user-level installs.
124+ """
125+ await self.exec_as_root(environment, command="apt-get update && apt-get install -y curl")
126+ await self.exec_as_agent(environment, command="pip install my-agent")
127+
128+ @with_prompt_template
129+ async def run(
130+ self, instruction: str, environment: BaseEnvironment, context: AgentContext
131+ ) -> None:
132+ """
133+ Run the agent in the environment. The @with_prompt_template decorator
134+ automatically applies prompt template rendering to the instruction.
135+ Use exec_as_agent to execute commands as the configured agent user.
136+ """
137+ await self.exec_as_agent(
138+ environment,
139+ command=f"my-agent run {shlex.quote(instruction)}",
140+ )
141+
142+ def populate_context_post_run(self, context: AgentContext) -> None:
143+ """
144+ Populate the context with the results of the agent execution.
145+ Called after run() completes. Typically involves parsing trajectory files.
146+ """
147+ pass
148+```
149+
150+The `exec_as_root` and `exec_as_agent` helpers handle logging, environment variable merging, `set -o pipefail`, and error handling automatically. `exec_as_agent` runs commands as the task's configured agent user (see [`agent.user` in task.toml](/docs/tasks#configuration--metadata)).
151+
152+### Running a custom agent
153+
154+To run a custom agent, you can use the following command:
155+
156+```bash
157+harbor run -d "<dataset@version>" --agent-import-path path.to.agent:SomeAgent
158+```