xonecas.io
64875
Mc

mini-coder / commit 5424197

chore: update terminal-bench benchmark runbook

xonecas2026-04-10 54241976b7d64dbf4896ffc6f93fa83aefd6c204 parent bcaf95c

2 files changed

terminal-bench/mini_coder_agent.py+3−3
@@ -12,7 +12,7 @@ from harbor.models.trial.paths import EnvironmentPaths
1212 class MiniCoderAgent(BaseInstalledAgent):
1313 """Harbor installed-agent wrapper for the published mini-coder CLI."""
1414
15- def __init__(self, *args, version: str = "0.5.0", **kwargs):
15+ def __init__(self, *args, version: str = "latest", **kwargs):
1616 super().__init__(*args, version=version, **kwargs)
1717 self._host_config_dir = Path.home() / ".config" / "mini-coder"
1818 self._host_settings_path = self._host_config_dir / "settings.json"
@@ -27,9 +27,9 @@ class MiniCoderAgent(BaseInstalledAgent):
2727 environment,
2828 command=(
2929 "if command -v apt-get >/dev/null 2>&1; then "
30- "apt-get update && apt-get install -y curl unzip ca-certificates bash git; "
30+ "apt-get update && apt-get install -y curl unzip ca-certificates bash git python3 ripgrep; "
3131 "elif command -v apk >/dev/null 2>&1; then "
32- "apk add --no-cache curl unzip ca-certificates bash git; "
32+ "apk add --no-cache curl unzip ca-certificates bash git python3 ripgrep; "
3333 "else "
3434 'echo "Unsupported package manager for mini-coder install" >&2; exit 1; '
3535 "fi"
terminal-bench/readme.md+81−92
@@ -1,47 +1,31 @@
11 # terminal-bench benchmark workspace
22
3-This folder is the scratch space for benchmarking mini-coder against Terminal-Bench 2.0 with Harbor.
3+This directory contains the Harbor wrapper and job output for benchmarking published `mini-coder` on Terminal-Bench 2.0.
44
5-Everything here is intentionally kept under `terminal-bench/` so it does it's separate from production code.
5+Everything here stays under `terminal-bench/` so benchmark work stays separate from production mini-coder code.
66
77 ## Goal
88
9-Benchmark published `mini-coder@latest` on Terminal-Bench 2.0 using Harbor, with the current local mini-coder config:
9+Run Terminal-Bench tasks against published `mini-coder@latest` with Harbor, using the current local mini-coder config:
1010
1111 - model: `openai-codex/gpt-5.4`
12-- effort: `xhigh`
1312 - auth source: `~/.config/mini-coder/auth.json`
1413 - settings source: `~/.config/mini-coder/settings.json`
14+- effort source: `~/.config/mini-coder/settings.json`
1515
16-## Key decisions
17-
18-1. **Use Harbor's custom installed-agent path**
19- - Harbor does not require multiple models.
20- - We use a custom installed agent instead of changing mini-coder core behavior.
21-
22-2. **Benchmark published mini-coder, not the local checkout**
23- - The wrapper installs `mini-coder@latest` inside the task container.
24- - This was the chosen path instead of packaging the current repo state.
25-
26-3. **Use the current mini-coder OAuth config**
27- - `openai-codex` in mini-coder is OAuth-backed.
28- - The wrapper copies the local `auth.json` and `settings.json` into the container.
29- - It uploads copies only; it does **not** mount or mutate the host files.
30-
31-4. **Keep the integration outside the product code**
32- - The Harbor agent lives here in `mini_coder_agent.py`.
33- - No tracked repo files were changed for the benchmark setup.
16+At the time of writing, `npm view mini-coder version dist-tags --json` reports `latest = 0.5.1`.
3417
3518 ## Files
3619
3720 - `mini_coder_agent.py` — Harbor installed-agent wrapper for mini-coder
38-- `jobs/` — Harbor job outputs from smoke runs
21+- `jobs/` — Harbor job outputs
22+- `readme.md` — this runbook
3923
40-## Current wrapper behavior
24+## What the wrapper does
4125
4226 `mini_coder_agent.py` does this per trial:
4327
44-1. installs system deps: `curl`, `unzip`, `ca-certificates`, `bash`, `git`
28+1. installs system deps: `curl`, `unzip`, `ca-certificates`, `bash`, `git`, `python3`, `ripgrep`
4529 2. installs Bun
4630 3. installs published `mini-coder@latest`
4731 4. finds the container user's home directory
@@ -54,112 +38,117 @@ Benchmark published `mini-coder@latest` on Terminal-Bench 2.0 using Harbor, with
5438 - `/logs/agent/mini-coder.ndjson`
5539 - `/logs/agent/mini-coder.stderr.txt`
5640
57-## Progress so far
41+## Requirements
5842
59-### Smoke run 1
43+Before running Harbor:
6044
61-Command:
45+- `harbor` must be installed and on `PATH`
46+- local mini-coder auth must be valid
47+- local mini-coder settings must exist
48+
49+Quick checks:
6250
6351 ```bash
64-PYTHONPATH=$PWD/terminal-bench harbor run -y \
65- -d terminal-bench@2.0 \
66- --agent-import-path mini_coder_agent:MiniCoderAgent \
67- -m openai-codex/gpt-5.4 \
68- -l 1 -k 1 -n 1 \
69- -o terminal-bench/jobs
52+command -v harbor
53+ls -l ~/.config/mini-coder/settings.json ~/.config/mini-coder/auth.json
54+npm view mini-coder version dist-tags --json
7055 ```
7156
72-Job dir:
73-
74-- `jobs/2026-04-09__22-14-57`
57+## Verify the wrapper imports
7558
76-Result:
77-
78-- failed immediately before the agent really started
79-- root cause: `git` was missing in the task container
80-- mini-coder stderr contained:
81-
82-```text
83-Executable not found in $PATH: "git"
59+```bash
60+PYTHONPATH=$PWD/terminal-bench \
61+ ~/.local/share/pipx/venvs/harbor/bin/python -m py_compile \
62+ terminal-bench/mini_coder_agent.py
8463 ```
8564
86-Fix applied afterward:
87-
88-- updated the wrapper install step to include `git`
89-
90-### Smoke run 2
91-
92-Same command after the `git` fix.
93-
94-Job dir:
65+## Recommended runs
9566
96-- `jobs/2026-04-09__22-17-21`
67+Run these two easy tasks first. They are the recommended smoke tests before trying slower tasks.
9768
98-Result:
69+### 1. `fix-git`
9970
100-- the run got past startup
101-- mini-coder emitted headless NDJSON thinking output
102-- Harbor job was interrupted/cancelled before completion
103-- Harbor recorded a `CancelledError`
71+Recommended first run.
10472
105-Evidence:
106-
107-- `jobs/2026-04-09__22-17-21/*/agent/mini-coder.ndjson` contains streamed model output
108-- `jobs/2026-04-09__22-17-21/*/exception.txt` shows cancellation
109-
110-## Current status
111-
112-The wrapper is past the first blocker (`git` missing) and is capable of launching mini-coder inside the Harbor task container.
113-
114-The next real checkpoint is to rerun the same 1-task smoke test and let it finish.
73+```bash
74+PYTHONPATH=$PWD/terminal-bench harbor run -y \
75+ -d terminal-bench@2.0 \
76+ --agent-import-path mini_coder_agent:MiniCoderAgent \
77+ -m openai-codex/gpt-5.4 \
78+ -i fix-git -l 1 \
79+ -k 1 -n 1 \
80+ -o terminal-bench/jobs
81+```
11582
116-## Resume commands
83+### 2. `prove-plus-comm`
11784
118-### Verify the wrapper still imports
85+Recommended alternate easy run.
11986
12087 ```bash
121-PYTHONPATH=$PWD/terminal-bench \
122- ~/.local/share/pipx/venvs/harbor/bin/python -m py_compile \
123- terminal-bench/mini_coder_agent.py
88+PYTHONPATH=$PWD/terminal-bench harbor run -y \
89+ -d terminal-bench@2.0 \
90+ --agent-import-path mini_coder_agent:MiniCoderAgent \
91+ -m openai-codex/gpt-5.4 \
92+ -i prove-plus-comm -l 1 \
93+ -k 1 -n 1 \
94+ -o terminal-bench/jobs
12495 ```
12596
126-### Run the 1-task smoke test again
97+### Generic one-task command
12798
12899 ```bash
129100 PYTHONPATH=$PWD/terminal-bench harbor run -y \
130101 -d terminal-bench@2.0 \
131102 --agent-import-path mini_coder_agent:MiniCoderAgent \
132103 -m openai-codex/gpt-5.4 \
133- -l 1 -k 1 -n 1 \
104+ -i <task-id> -l 1 \
105+ -k 1 -n 1 \
134106 -o terminal-bench/jobs
135107 ```
136108
137-### Inspect the latest Harbor job
109+## Inspecting results
110+
111+### Show the latest Harbor job tree
138112
139113 ```bash
140114 JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
141-find "$JOB" -maxdepth 3 -type f | sort
115+find "$JOB" -maxdepth 4 -type f | sort
142116 ```
143117
144-### Inspect mini-coder logs from the latest trial
118+### Show the latest top-level result summary
145119
146120 ```bash
147121 JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
148-sed -n '1,120p' "$JOB"/*/agent/mini-coder.stderr.txt
149-sed -n '1,120p' "$JOB"/*/agent/mini-coder.ndjson
122+sed -n '1,220p' "$JOB"/result.json
150123 ```
151124
152-## Known constraints / notes
125+### Show verifier output from the latest trial
153126
154-- This setup depends on the host mini-coder OAuth/login state being current.
155-- If `openai-codex` auth expires, refresh it locally with mini-coder before rerunning.
156-- Because the wrapper copies the host config into the container, the host auth/settings files are not modified by Harbor.
157-- The first two smoke jobs are kept intentionally for debugging history.
127+```bash
128+JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
129+sed -n '1,220p' "$JOB"/*/verifier/test-stdout.txt
130+sed -n '1,40p' "$JOB"/*/verifier/reward.txt
131+```
158132
159-## Next step
133+### Show mini-coder logs from the latest trial
134+
135+```bash
136+JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
137+sed -n '1,160p' "$JOB"/*/agent/mini-coder.stderr.txt
138+sed -n '1,160p' "$JOB"/*/agent/mini-coder.ndjson
139+```
140+
141+### Show the latest trial exception, if any
142+
143+```bash
144+JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
145+sed -n '1,220p' "$JOB"/*/exception.txt
146+```
160147
161-Rerun the same 1-task smoke test and let it complete. If that passes the launch phase cleanly, decide whether to:
148+## Notes
162149
163-1. keep iterating with `-l 1 -k 1`
164-2. run a small subset
165-3. run a larger TB2 batch later
150+- The wrapper benchmarks the published package, not the local checkout.
151+- The wrapper copies host mini-coder auth/settings into the container; it does not mount or mutate the host files.
152+- The task image supplies task-specific toolchains. The wrapper only adds generic helpers like `git`, `python3`, and `ripgrep`.
153+- For leaderboard-comparable runs, do **not** change timeouts or resources.
154+- If `openai-codex` auth expires, ask the user to refresh it locally with mini-coder before rerunning.