@@ -0,0 +1,165 @@
1+# terminal-bench benchmark workspace
2+
3+This folder is the scratch space for benchmarking mini-coder against Terminal-Bench 2.0 with Harbor.
4+
5+Everything here is intentionally kept under `terminal-bench/` so it does it's separate from production code.
6+
7+## Goal
8+
9+Benchmark published `mini-coder@latest` on Terminal-Bench 2.0 using Harbor, with the current local mini-coder config:
10+
11+- model: `openai-codex/gpt-5.4`
12+- effort: `xhigh`
13+- auth source: `~/.config/mini-coder/auth.json`
14+- settings source: `~/.config/mini-coder/settings.json`
15+
16+## Key decisions
17+
18+1. **Use Harbor's custom installed-agent path**
19+ - Harbor does not require multiple models.
20+ - We use a custom installed agent instead of changing mini-coder core behavior.
21+
22+2. **Benchmark published mini-coder, not the local checkout**
23+ - The wrapper installs `mini-coder@latest` inside the task container.
24+ - This was the chosen path instead of packaging the current repo state.
25+
26+3. **Use the current mini-coder OAuth config**
27+ - `openai-codex` in mini-coder is OAuth-backed.
28+ - The wrapper copies the local `auth.json` and `settings.json` into the container.
29+ - It uploads copies only; it does **not** mount or mutate the host files.
30+
31+4. **Keep the integration outside the product code**
32+ - The Harbor agent lives here in `mini_coder_agent.py`.
33+ - No tracked repo files were changed for the benchmark setup.
34+
35+## Files
36+
37+- `mini_coder_agent.py` — Harbor installed-agent wrapper for mini-coder
38+- `jobs/` — Harbor job outputs from smoke runs
39+
40+## Current wrapper behavior
41+
42+`mini_coder_agent.py` does this per trial:
43+
44+1. installs system deps: `curl`, `unzip`, `ca-certificates`, `bash`, `git`
45+2. installs Bun
46+3. installs published `mini-coder@latest`
47+4. finds the container user's home directory
48+5. uploads copies of:
49+ - `~/.config/mini-coder/settings.json`
50+ - `~/.config/mini-coder/auth.json`
51+6. runs:
52+ - `mc -p <task instruction>`
53+7. writes logs to:
54+ - `/logs/agent/mini-coder.ndjson`
55+ - `/logs/agent/mini-coder.stderr.txt`
56+
57+## Progress so far
58+
59+### Smoke run 1
60+
61+Command:
62+
63+```bash
64+PYTHONPATH=$PWD/terminal-bench harbor run -y \
65+ -d terminal-bench@2.0 \
66+ --agent-import-path mini_coder_agent:MiniCoderAgent \
67+ -m openai-codex/gpt-5.4 \
68+ -l 1 -k 1 -n 1 \
69+ -o terminal-bench/jobs
70+```
71+
72+Job dir:
73+
74+- `jobs/2026-04-09__22-14-57`
75+
76+Result:
77+
78+- failed immediately before the agent really started
79+- root cause: `git` was missing in the task container
80+- mini-coder stderr contained:
81+
82+```text
83+Executable not found in $PATH: "git"
84+```
85+
86+Fix applied afterward:
87+
88+- updated the wrapper install step to include `git`
89+
90+### Smoke run 2
91+
92+Same command after the `git` fix.
93+
94+Job dir:
95+
96+- `jobs/2026-04-09__22-17-21`
97+
98+Result:
99+
100+- the run got past startup
101+- mini-coder emitted headless NDJSON thinking output
102+- Harbor job was interrupted/cancelled before completion
103+- Harbor recorded a `CancelledError`
104+
105+Evidence:
106+
107+- `jobs/2026-04-09__22-17-21/*/agent/mini-coder.ndjson` contains streamed model output
108+- `jobs/2026-04-09__22-17-21/*/exception.txt` shows cancellation
109+
110+## Current status
111+
112+The wrapper is past the first blocker (`git` missing) and is capable of launching mini-coder inside the Harbor task container.
113+
114+The next real checkpoint is to rerun the same 1-task smoke test and let it finish.
115+
116+## Resume commands
117+
118+### Verify the wrapper still imports
119+
120+```bash
121+PYTHONPATH=$PWD/terminal-bench \
122+ ~/.local/share/pipx/venvs/harbor/bin/python -m py_compile \
123+ terminal-bench/mini_coder_agent.py
124+```
125+
126+### Run the 1-task smoke test again
127+
128+```bash
129+PYTHONPATH=$PWD/terminal-bench harbor run -y \
130+ -d terminal-bench@2.0 \
131+ --agent-import-path mini_coder_agent:MiniCoderAgent \
132+ -m openai-codex/gpt-5.4 \
133+ -l 1 -k 1 -n 1 \
134+ -o terminal-bench/jobs
135+```
136+
137+### Inspect the latest Harbor job
138+
139+```bash
140+JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
141+find "$JOB" -maxdepth 3 -type f | sort
142+```
143+
144+### Inspect mini-coder logs from the latest trial
145+
146+```bash
147+JOB=$(ls -dt terminal-bench/jobs/* | head -n1)
148+sed -n '1,120p' "$JOB"/*/agent/mini-coder.stderr.txt
149+sed -n '1,120p' "$JOB"/*/agent/mini-coder.ndjson
150+```
151+
152+## Known constraints / notes
153+
154+- This setup depends on the host mini-coder OAuth/login state being current.
155+- If `openai-codex` auth expires, refresh it locally with mini-coder before rerunning.
156+- Because the wrapper copies the host config into the container, the host auth/settings files are not modified by Harbor.
157+- The first two smoke jobs are kept intentionally for debugging history.
158+
159+## Next step
160+
161+Rerun the same 1-task smoke test and let it complete. If that passes the launch phase cleanly, decide whether to:
162+
163+1. keep iterating with `-l 1 -k 1`
164+2. run a small subset
165+3. run a larger TB2 batch later