@@ -0,0 +1,266 @@
1+# Leaderboard failures rerun analysis
2+
3+This report compares:
4+
5+- original leaderboard run: `terminal-bench/jobs/leaderboard-2026-04-10__16-06-57`
6+- failure rerun: `terminal-bench/jobs/leaderboard-2026-04-10__16-06-57-failures-rerun-2`
7+
8+Data sources used:
9+
10+- each trial `result.json`
11+- verifier `ctrf.json` and `test-stdout.txt`
12+- agent logs in `agent/mini-coder.ndjson`
13+
14+## Executive summary
15+
16+This rerun covered **29 tasks that had failed in the original leaderboard run**.
17+
18+Outcome on the rerun set:
19+
20+- **12 now pass**
21+- **16 still fail**
22+- **1 timed out before verifier completion** (`video-processing`)
23+
24+The fail -> pass tasks are:
25+
26+- `adaptive-rejection-sampler`
27+- `dna-insert`
28+- `extract-elf`
29+- `largest-eigenval`
30+- `llm-inference-batching-scheduler`
31+- `mailman`
32+- `overfull-hbox`
33+- `polyglot-c-py`
34+- `polyglot-rust-c`
35+- `qemu-alpine-ssh`
36+- `torch-tensor-parallelism`
37+- `tune-mjcf`
38+
39+The improvement is real. The strongest signal is that **two tasks now pass even though the agent still timed out**:
40+
41+- `mailman`
42+- `tune-mjcf`
43+
44+That means the required artifact was already good enough before the run ended. This matches the intended prompt change: produce the required output earlier instead of saving it for late in the run.
45+
46+## What improved
47+
48+## 1. More tasks now cross the verifier line
49+
50+Several tasks moved from obvious near-miss territory to clean pass territory:
51+
52+| task | original | rerun |
53+| ---------------------------------- | ------------------ | ------------------------------- |
54+| `llm-inference-batching-scheduler` | 1/6 tests, timeout | 6/6 tests, pass |
55+| `polyglot-rust-c` | 0/1 tests, timeout | 1/1 tests, pass |
56+| `qemu-alpine-ssh` | 0/1 tests, timeout | 1/1 tests, pass |
57+| `mailman` | 1/3 tests, timeout | 3/3 tests, pass despite timeout |
58+| `tune-mjcf` | 3/4 tests, timeout | 4/4 tests, pass despite timeout |
59+| `adaptive-rejection-sampler` | 8/9 tests | 9/9 tests |
60+| `extract-elf` | 1/2 tests | 2/2 tests |
61+| `largest-eigenval` | 2/3 tests | 3/3 tests |
62+| `torch-tensor-parallelism` | 2/3 tests | 3/3 tests |
63+| `overfull-hbox` | 3/4 tests | 4/4 tests |
64+
65+This is not just a small shift in average behavior. Multiple tasks crossed from partial verifier progress to full pass.
66+
67+## 2. The agent appears to verify a bit more, and earlier
68+
69+Using a rough heuristic over shell commands, the rerun subset shows more verifier/test-like shell usage:
70+
71+- original rerun-subset baseline: **8** test/verifier-like shell calls
72+- rerun: **18** test/verifier-like shell calls
73+
74+Within the rerun itself, the passing tasks also look more verifier-oriented than the still-failing tasks:
75+
76+- tasks that improved to pass: **0.92** test-like shell calls on average
77+- tasks that still failed: **0.41** test-like shell calls on average
78+
79+This is only a heuristic, but it lines up with the visible task outcomes: more of the successful runs found the verifier boundary and iterated against it.
80+
81+## 3. Some runs now create a passing artifact before the timeout wall
82+
83+The clearest evidence is again `mailman` and `tune-mjcf`, which both still ended with `AgentTimeoutError` but now score `1.0`.
84+
85+That is exactly the failure mode the prompt change was meant to attack: runs that do enough good work to pass, but only if the required artifact lands before the timeout.
86+
87+## Tool awkwardness: what changed
88+
89+## `edit`
90+
91+`edit` no longer looks like a major source of friction.
92+
93+Across the 29-task rerun subset:
94+
95+- **79** edit calls total
96+- only **2** edit errors total
97+
98+Those two edit errors were straightforward exact-match failures:
99+
100+- `caffe-cifar-10` — old text matched multiple locations
101+- `install-windows-3.11` — old text not found
102+
103+Important contrast with the original run: the earlier internal `edit` crash seen in `gpt2-codegolf` (`undefined is not an object`) did **not** reappear here.
104+
105+Conclusion: from the agent's perspective, `edit` is mostly behaving cleanly now.
106+
107+## `shell`
108+
109+Raw non-zero shell exits are almost unchanged:
110+
111+- original rerun-subset baseline: **68** non-zero shell results
112+- rerun: **67** non-zero shell results
113+
114+So the improvement is **not** “the agent suddenly stopped making shell mistakes”. The more useful change is in the type of shell awkwardness.
115+
116+### The earlier `printf` compatibility issue appears fixed
117+
118+The original run contained several `printf: Illegal option --` failures in agent-authored shell commands.
119+
120+I found those in the original logs, including `mailman`, `query-optimize`, and other tasks. I found **no occurrences** of that issue in the rerun logs.
121+
122+That is a direct positive signal for the shell compatibility normalization work.
123+
124+### The remaining shell awkwardness is mostly environment assumptions
125+
126+The recurring shell failures in the rerun are dominated by assumptions about available binaries rather than mini-coder shell semantics.
127+
128+Recurring examples in the rerun subset:
129+
130+- `python` not found — **8 tasks**
131+- `file` not found — **4 tasks**
132+- `No module named pip` — **3 tasks**
133+- `/usr/bin/time` not found — **2 tasks**
134+- `g++` not found — **2 tasks**
135+
136+Tasks hit by `python` not found:
137+
138+- `dna-assembly`
139+- `dna-insert`
140+- `extract-moves-from-video`
141+- `mcmc-sampling-stan`
142+- `overfull-hbox`
143+- `qemu-alpine-ssh`
144+- `torch-pipeline-parallelism`
145+- `torch-tensor-parallelism`
146+
147+This matters because some of these tasks still passed anyway. So these failures are real friction, but they are not the same class as the earlier shell-tool conflict. The agent is still making environment assumptions, but mini-coder itself is less obviously fighting the agent.
148+
149+## Remaining failures: how they still fail
150+
151+## A. Timed out without producing the required artifact
152+
153+These are still the clearest contract-first misses.
154+
155+| task | rerun verifier | main failure signal |
156+| -------------------------- | -------------------- | ------------------------------------ |
157+| `crack-7z-hash` | 0/2 | `/app/solution.txt` missing |
158+| `db-wal-recovery` | 0/7 | `/app/recovered.json` missing |
159+| `extract-moves-from-video` | 0/2 | `/app/solution.txt` missing |
160+| `make-doom-for-mips` | 0/3 | `/tmp/frame.bmp` missing |
161+| `video-processing` | verifier not reached | timed out before verifier completion |
162+
163+This is still the biggest remaining bucket of lost score.
164+
165+## B. Timed out with partial progress, but not enough to pass
166+
167+These runs clearly did real work, but did not land on the exact contract in time.
168+
169+| task | rerun verifier | main failure signal |
170+| ---------------------- | -------------- | ------------------------------------------------------------ |
171+| `caffe-cifar-10` | 3/6 | `caffe.bin` missing; solver not set to CPU mode |
172+| `install-windows-3.11` | 2/4 | no VGA config; keyboard/F1 monitor interaction still failing |
173+| `train-fasttext` | 0/2 | accuracy too low and model too large |
174+
175+`caffe-cifar-10` is especially worth noting: it improved from “timeout before verifier signal” in the original run to **3/6 passing tests** in the rerun, so it is clearly closer, but still not there.
176+
177+## C. Near misses: one bad detail blocks an otherwise strong run
178+
179+These are the rerun failures that already look close to passing.
180+
181+| task | rerun verifier | main failure signal |
182+| ---------------------------- | -------------- | ------------------------------------------------------------- |
183+| `headless-terminal` | 6/7 | interactive path breaks inside `pyte` on private SGR handling |
184+| `mcmc-sampling-stan` | 5/6 | expected sampling log/messages missing |
185+| `make-mips-interpreter` | 2/3 | output image similarity 0.8456, needs 0.95 |
186+| `torch-pipeline-parallelism` | 2/3 | backward mismatch in one layer/microbatch |
187+
188+These are not broad failures. They are verifier-focused misses on one remaining edge or correctness detail.
189+
190+## D. Wrong solution shape or algorithm mismatch
191+
192+These failures do not look tool-related. They look like the agent produced the wrong implementation or the wrong tradeoff.
193+
194+| task | rerun verifier | main failure signal |
195+| --------------------- | -------------- | -------------------------------------------------------- |
196+| `dna-assembly` | 0/1 | primer constraints wrong |
197+| `filter-js-from-html` | 0/2 | still misses many XSS cases and also modifies clean HTML |
198+| `gpt2-codegolf` | 0/1 | wrong output |
199+| `raman-fitting` | 1/3 | fitted peaks are materially wrong |
200+| `sam-cell-seg` | 1/9 | run script fails; expected output CSV missing |
201+
202+`filter-js-from-html` is the clearest example here. The rerun log shows a lot of detailed work and many tool calls, but the verifier still reports both:
203+
204+- the sanitizer misses large classes of attack vectors
205+- the sanitizer changes clean HTML that should remain unchanged
206+
207+That is a solution-fit issue, not a tool-usage issue.
208+
209+## Regressions and unstable cases
210+
211+Not everything improved.
212+
213+A few tasks got worse, or at least less stable:
214+
215+| task | original | rerun |
216+| ---------------------- | --------- | ----------------------------- |
217+| `db-wal-recovery` | 5/7 tests | 0/7 tests, timeout |
218+| `install-windows-3.11` | 3/4 tests | 2/4 tests |
219+| `video-processing` | 4/5 tests | verifier not reached, timeout |
220+
221+A few others stayed stuck at roughly the same near-miss level:
222+
223+- `headless-terminal` — 6/7 -> 6/7
224+- `make-mips-interpreter` — 2/3 -> 2/3
225+- `torch-pipeline-parallelism` — 2/3 -> 2/3
226+- `raman-fitting` — 1/3 -> 1/3
227+- `sam-cell-seg` — 1/9 -> 1/9
228+
229+So the gains are real, but they are not universal.
230+
231+## Overall conclusion
232+
233+The rerun supports the main hypothesis behind the recent prompt and tool work:
234+
235+- the agent is more likely to produce the required output early enough to matter
236+- the agent is a bit more likely to discover verifier boundaries and iterate against them
237+- the earlier `printf` shell awkwardness looks gone
238+- `edit` now looks low-friction most of the time
239+
240+The remaining misses are now less about mini-coder fighting the agent, and more about three buckets:
241+
242+1. still timing out before writing the exact required artifact
243+2. near-miss implementations with one remaining verifier-visible defect
244+3. environment assumptions in shell usage (`python`, `file`, `pip`, `g++`, `/usr/bin/time`)
245+
246+That is a healthier failure profile than the original run.
247+
248+## Ranked narrow changes to discuss next
249+
250+These are ranked discussion candidates, based on the rerun data above.
251+
252+1. **Add a narrow shell compatibility rewrite from `python` -> `python3` when `python` is absent and the rewrite is unambiguous.**
253+ - Why this ranks first: `python`-missing shell failures appeared in **8 rerun tasks**, including some otherwise strong runs. This is a narrow, repeated, machine-fixable pattern.
254+
255+2. **Add an explicit prompt rule to probe tool availability before assuming non-core binaries exist.**
256+ - Example pattern: `command -v file`, `command -v g++`, `command -v python3`, `python3 -m pip --version`.
257+ - Why: the rerun still shows repeated assumptions about `file`, `pip`, `g++`, and `/usr/bin/time`.
258+
259+3. **Strengthen the prompt rule that if the task names a required output path, the agent should create that exact artifact early, then iterate.**
260+ - Why: the biggest remaining zero-score failures still end with missing required files like `/app/solution.txt`, `/app/recovered.json`, or `/tmp/frame.bmp`.
261+
262+4. **Strengthen the prompt rule to run the narrowest verifier or test as soon as there is a plausible first artifact, instead of delaying the first verifier run.**
263+ - Why: the improved tasks correlate with more verifier-like shell usage, and many remaining failures still look under-verified until late.
264+
265+5. **Add a prompt rule for near-miss mode: once most tests are passing, stop broad exploration and focus only on the failing assertions.**
266+ - Why: several rerun failures are already at 5/6, 6/7, or 2/3 and look like they lost time to broader work instead of concentrating on the final failing detail.