@@ -1,266 +0,0 @@
1-# Leaderboard failures rerun analysis
2-
3-This report compares:
4-
5-- original leaderboard run: `terminal-bench/jobs/leaderboard-2026-04-10__16-06-57`
6-- failure rerun: `terminal-bench/jobs/leaderboard-2026-04-10__16-06-57-failures-rerun-2`
7-
8-Data sources used:
9-
10-- each trial `result.json`
11-- verifier `ctrf.json` and `test-stdout.txt`
12-- agent logs in `agent/mini-coder.ndjson`
13-
14-## Executive summary
15-
16-This rerun covered **29 tasks that had failed in the original leaderboard run**.
17-
18-Outcome on the rerun set:
19-
20-- **12 now pass**
21-- **16 still fail**
22-- **1 timed out before verifier completion** (`video-processing`)
23-
24-The fail -> pass tasks are:
25-
26-- `adaptive-rejection-sampler`
27-- `dna-insert`
28-- `extract-elf`
29-- `largest-eigenval`
30-- `llm-inference-batching-scheduler`
31-- `mailman`
32-- `overfull-hbox`
33-- `polyglot-c-py`
34-- `polyglot-rust-c`
35-- `qemu-alpine-ssh`
36-- `torch-tensor-parallelism`
37-- `tune-mjcf`
38-
39-The improvement is real. The strongest signal is that **two tasks now pass even though the agent still timed out**:
40-
41-- `mailman`
42-- `tune-mjcf`
43-
44-That means the required artifact was already good enough before the run ended. This matches the intended prompt change: produce the required output earlier instead of saving it for late in the run.
45-
46-## What improved
47-
48-## 1. More tasks now cross the verifier line
49-
50-Several tasks moved from obvious near-miss territory to clean pass territory:
51-
52-| task | original | rerun |
53-| ---------------------------------- | ------------------ | ------------------------------- |
54-| `llm-inference-batching-scheduler` | 1/6 tests, timeout | 6/6 tests, pass |
55-| `polyglot-rust-c` | 0/1 tests, timeout | 1/1 tests, pass |
56-| `qemu-alpine-ssh` | 0/1 tests, timeout | 1/1 tests, pass |
57-| `mailman` | 1/3 tests, timeout | 3/3 tests, pass despite timeout |
58-| `tune-mjcf` | 3/4 tests, timeout | 4/4 tests, pass despite timeout |
59-| `adaptive-rejection-sampler` | 8/9 tests | 9/9 tests |
60-| `extract-elf` | 1/2 tests | 2/2 tests |
61-| `largest-eigenval` | 2/3 tests | 3/3 tests |
62-| `torch-tensor-parallelism` | 2/3 tests | 3/3 tests |
63-| `overfull-hbox` | 3/4 tests | 4/4 tests |
64-
65-This is not just a small shift in average behavior. Multiple tasks crossed from partial verifier progress to full pass.
66-
67-## 2. The agent appears to verify a bit more, and earlier
68-
69-Using a rough heuristic over shell commands, the rerun subset shows more verifier/test-like shell usage:
70-
71-- original rerun-subset baseline: **8** test/verifier-like shell calls
72-- rerun: **18** test/verifier-like shell calls
73-
74-Within the rerun itself, the passing tasks also look more verifier-oriented than the still-failing tasks:
75-
76-- tasks that improved to pass: **0.92** test-like shell calls on average
77-- tasks that still failed: **0.41** test-like shell calls on average
78-
79-This is only a heuristic, but it lines up with the visible task outcomes: more of the successful runs found the verifier boundary and iterated against it.
80-
81-## 3. Some runs now create a passing artifact before the timeout wall
82-
83-The clearest evidence is again `mailman` and `tune-mjcf`, which both still ended with `AgentTimeoutError` but now score `1.0`.
84-
85-That is exactly the failure mode the prompt change was meant to attack: runs that do enough good work to pass, but only if the required artifact lands before the timeout.
86-
87-## Tool awkwardness: what changed
88-
89-## `edit`
90-
91-`edit` no longer looks like a major source of friction.
92-
93-Across the 29-task rerun subset:
94-
95-- **79** edit calls total
96-- only **2** edit errors total
97-
98-Those two edit errors were straightforward exact-match failures:
99-
100-- `caffe-cifar-10` — old text matched multiple locations
101-- `install-windows-3.11` — old text not found
102-
103-Important contrast with the original run: the earlier internal `edit` crash seen in `gpt2-codegolf` (`undefined is not an object`) did **not** reappear here.
104-
105-Conclusion: from the agent's perspective, `edit` is mostly behaving cleanly now.
106-
107-## `shell`
108-
109-Raw non-zero shell exits are almost unchanged:
110-
111-- original rerun-subset baseline: **68** non-zero shell results
112-- rerun: **67** non-zero shell results
113-
114-So the improvement is **not** “the agent suddenly stopped making shell mistakes”. The more useful change is in the type of shell awkwardness.
115-
116-### The earlier `printf` compatibility issue appears fixed
117-
118-The original run contained several `printf: Illegal option --` failures in agent-authored shell commands.
119-
120-I found those in the original logs, including `mailman`, `query-optimize`, and other tasks. I found **no occurrences** of that issue in the rerun logs.
121-
122-That is a direct positive signal for the shell compatibility normalization work.
123-
124-### The remaining shell awkwardness is mostly environment assumptions
125-
126-The recurring shell failures in the rerun are dominated by assumptions about available binaries rather than mini-coder shell semantics.
127-
128-Recurring examples in the rerun subset:
129-
130-- `python` not found — **8 tasks**
131-- `file` not found — **4 tasks**
132-- `No module named pip` — **3 tasks**
133-- `/usr/bin/time` not found — **2 tasks**
134-- `g++` not found — **2 tasks**
135-
136-Tasks hit by `python` not found:
137-
138-- `dna-assembly`
139-- `dna-insert`
140-- `extract-moves-from-video`
141-- `mcmc-sampling-stan`
142-- `overfull-hbox`
143-- `qemu-alpine-ssh`
144-- `torch-pipeline-parallelism`
145-- `torch-tensor-parallelism`
146-
147-This matters because some of these tasks still passed anyway. So these failures are real friction, but they are not the same class as the earlier shell-tool conflict. The agent is still making environment assumptions, but mini-coder itself is less obviously fighting the agent.
148-
149-## Remaining failures: how they still fail
150-
151-## A. Timed out without producing the required artifact
152-
153-These are still the clearest contract-first misses.
154-
155-| task | rerun verifier | main failure signal |
156-| -------------------------- | -------------------- | ------------------------------------ |
157-| `crack-7z-hash` | 0/2 | `/app/solution.txt` missing |
158-| `db-wal-recovery` | 0/7 | `/app/recovered.json` missing |
159-| `extract-moves-from-video` | 0/2 | `/app/solution.txt` missing |
160-| `make-doom-for-mips` | 0/3 | `/tmp/frame.bmp` missing |
161-| `video-processing` | verifier not reached | timed out before verifier completion |
162-
163-This is still the biggest remaining bucket of lost score.
164-
165-## B. Timed out with partial progress, but not enough to pass
166-
167-These runs clearly did real work, but did not land on the exact contract in time.
168-
169-| task | rerun verifier | main failure signal |
170-| ---------------------- | -------------- | ------------------------------------------------------------ |
171-| `caffe-cifar-10` | 3/6 | `caffe.bin` missing; solver not set to CPU mode |
172-| `install-windows-3.11` | 2/4 | no VGA config; keyboard/F1 monitor interaction still failing |
173-| `train-fasttext` | 0/2 | accuracy too low and model too large |
174-
175-`caffe-cifar-10` is especially worth noting: it improved from “timeout before verifier signal” in the original run to **3/6 passing tests** in the rerun, so it is clearly closer, but still not there.
176-
177-## C. Near misses: one bad detail blocks an otherwise strong run
178-
179-These are the rerun failures that already look close to passing.
180-
181-| task | rerun verifier | main failure signal |
182-| ---------------------------- | -------------- | ------------------------------------------------------------- |
183-| `headless-terminal` | 6/7 | interactive path breaks inside `pyte` on private SGR handling |
184-| `mcmc-sampling-stan` | 5/6 | expected sampling log/messages missing |
185-| `make-mips-interpreter` | 2/3 | output image similarity 0.8456, needs 0.95 |
186-| `torch-pipeline-parallelism` | 2/3 | backward mismatch in one layer/microbatch |
187-
188-These are not broad failures. They are verifier-focused misses on one remaining edge or correctness detail.
189-
190-## D. Wrong solution shape or algorithm mismatch
191-
192-These failures do not look tool-related. They look like the agent produced the wrong implementation or the wrong tradeoff.
193-
194-| task | rerun verifier | main failure signal |
195-| --------------------- | -------------- | -------------------------------------------------------- |
196-| `dna-assembly` | 0/1 | primer constraints wrong |
197-| `filter-js-from-html` | 0/2 | still misses many XSS cases and also modifies clean HTML |
198-| `gpt2-codegolf` | 0/1 | wrong output |
199-| `raman-fitting` | 1/3 | fitted peaks are materially wrong |
200-| `sam-cell-seg` | 1/9 | run script fails; expected output CSV missing |
201-
202-`filter-js-from-html` is the clearest example here. The rerun log shows a lot of detailed work and many tool calls, but the verifier still reports both:
203-
204-- the sanitizer misses large classes of attack vectors
205-- the sanitizer changes clean HTML that should remain unchanged
206-
207-That is a solution-fit issue, not a tool-usage issue.
208-
209-## Regressions and unstable cases
210-
211-Not everything improved.
212-
213-A few tasks got worse, or at least less stable:
214-
215-| task | original | rerun |
216-| ---------------------- | --------- | ----------------------------- |
217-| `db-wal-recovery` | 5/7 tests | 0/7 tests, timeout |
218-| `install-windows-3.11` | 3/4 tests | 2/4 tests |
219-| `video-processing` | 4/5 tests | verifier not reached, timeout |
220-
221-A few others stayed stuck at roughly the same near-miss level:
222-
223-- `headless-terminal` — 6/7 -> 6/7
224-- `make-mips-interpreter` — 2/3 -> 2/3
225-- `torch-pipeline-parallelism` — 2/3 -> 2/3
226-- `raman-fitting` — 1/3 -> 1/3
227-- `sam-cell-seg` — 1/9 -> 1/9
228-
229-So the gains are real, but they are not universal.
230-
231-## Overall conclusion
232-
233-The rerun supports the main hypothesis behind the recent prompt and tool work:
234-
235-- the agent is more likely to produce the required output early enough to matter
236-- the agent is a bit more likely to discover verifier boundaries and iterate against them
237-- the earlier `printf` shell awkwardness looks gone
238-- `edit` now looks low-friction most of the time
239-
240-The remaining misses are now less about mini-coder fighting the agent, and more about three buckets:
241-
242-1. still timing out before writing the exact required artifact
243-2. near-miss implementations with one remaining verifier-visible defect
244-3. environment assumptions in shell usage (`python`, `file`, `pip`, `g++`, `/usr/bin/time`)
245-
246-That is a healthier failure profile than the original run.
247-
248-## Ranked narrow changes to discuss next
249-
250-These are ranked discussion candidates, based on the rerun data above.
251-
252-1. **Add a narrow shell compatibility rewrite from `python` -> `python3` when `python` is absent and the rewrite is unambiguous.**
253- - Why this ranks first: `python`-missing shell failures appeared in **8 rerun tasks**, including some otherwise strong runs. This is a narrow, repeated, machine-fixable pattern.
254-
255-2. **Add an explicit prompt rule to probe tool availability before assuming non-core binaries exist.**
256- - Example pattern: `command -v file`, `command -v g++`, `command -v python3`, `python3 -m pip --version`.
257- - Why: the rerun still shows repeated assumptions about `file`, `pip`, `g++`, and `/usr/bin/time`.
258-
259-3. **Strengthen the prompt rule that if the task names a required output path, the agent should create that exact artifact early, then iterate.**
260- - Why: the biggest remaining zero-score failures still end with missing required files like `/app/solution.txt`, `/app/recovered.json`, or `/tmp/frame.bmp`.
261-
262-4. **Strengthen the prompt rule to run the narrowest verifier or test as soon as there is a plausible first artifact, instead of delaying the first verifier run.**
263- - Why: the improved tasks correlate with more verifier-like shell usage, and many remaining failures still look under-verified until late.
264-
265-5. **Add a prompt rule for near-miss mode: once most tests are passing, stop broad exploration and focus only on the failing assertions.**
266- - Why: several rerun failures are already at 5/6, 6/7, or 2/3 and look like they lost time to broader work instead of concentrating on the final failing detail.