xonecas.io
64875
Mc

mini-coder / commit 7beda02

checkpoint: started benchmark loop

xonecas2026-04-18 7beda02519f02b2d4562402c58b2243d6e89daba parent fac0122

4 files changed

BENCHMARK.md+32−19
@@ -19,17 +19,20 @@ The goal is not to rerun the whole benchmark after every change. The goal is to
1919 Use this to catch regressions on known-good behavior.
2020
2121 Tasks:
22+
2223 - `cancel-async-tasks`
2324 - `bn-fit-modify`
2425 - `sparql-university`
2526 - `extract-elf`
2627
2728 Settings:
29+
2830 - `1` attempt
2931 - `2` concurrent
3032 - `0` retries
3133
3234 Estimated wall time:
35+
3336 - about `8–10 minutes`
3437
3538 ### 2. Fast suite
@@ -37,6 +40,7 @@ Estimated wall time:
3740 Use this after almost every small change.
3841
3942 Tasks:
43+
4044 - `polyglot-rust-c`
4145 - `overfull-hbox`
4246 - `gcode-to-text`
@@ -45,11 +49,13 @@ Tasks:
4549 - `torch-tensor-parallelism`
4650
4751 Settings:
52+
4853 - `2` attempts
4954 - `2` concurrent
5055 - `0` retries
5156
5257 Estimated wall time:
58+
5359 - about `44 minutes`
5460
5561 ### 3. Focused suite
@@ -57,6 +63,7 @@ Estimated wall time:
5763 Use this only if a change looks good on the fast suite.
5864
5965 Tasks:
66+
6067 - `polyglot-rust-c`
6168 - `overfull-hbox`
6269 - `gcode-to-text`
@@ -67,11 +74,13 @@ Tasks:
6774 - `qemu-alpine-ssh`
6875
6976 Settings:
77+
7078 - `2` attempts
7179 - `2` concurrent
7280 - `0` retries
7381
7482 Estimated wall time:
83+
7584 - about `1.1 hours`
7685
7786 ### 4. Broad promotion suite
@@ -79,11 +88,13 @@ Estimated wall time:
7988 Use this after 1–3 promising changes, or overnight.
8089
8190 Options:
91+
8292 - rerun recent failures
8393 - wider local failure suite
8494 - full leaderboard-style run
8595
8696 Typical settings for rerunning recent failures:
97+
8798 - `4` attempts
8899 - `2` concurrent
89100 - `0` retries
@@ -97,6 +108,7 @@ Before changing code:
97108 3. run the fast suite again
98109
99110 That gives a fresh local baseline for:
111+
100112 - pass count
101113 - runtime
102114 - variance / noise
@@ -121,10 +133,12 @@ For each change:
121133 - refine
122134
123135 Only run the focused suite if:
136+
124137 - the fast suite improved, and
125138 - the guardrails did not regress
126139
127140 Only run the broad promotion suite if:
141+
128142 - the focused suite also looks good, or
129143 - enough promising changes have accumulated to justify it
130144
@@ -133,20 +147,24 @@ Only run the broad promotion suite if:
133147 Treat the fast suite as a noisy but useful signal.
134148
135149 For the 12-trial fast suite:
150+
136151 - `+2` or more passes: probably meaningful improvement
137152 - `-2` or more passes: probably meaningful regression
138153 - `±1`: likely noise unless it repeats
139154
140155 Keep a change if:
156+
141157 - the fast suite improves materially
142158 - guardrails do not regress
143159 - the result repeats on another fast-suite rerun
144160
145161 Revert a change if:
162+
146163 - it clearly loses fast-suite passes, or
147164 - it regresses guardrails
148165
149166 Call it inconclusive if:
167+
150168 - the delta is tiny and does not repeat
151169
152170 ## Failure buckets to optimize against
@@ -154,31 +172,41 @@ Call it inconclusive if:
154172 Use these as the main buckets when inspecting logs:
155173
156174 ### Exact-contract / cleanup misses
175+
157176 Examples:
177+
158178 - right artifact, wrong path
159179 - right output plus extra junk
160180 - correct core work, but violates a strict file / in-place requirement
161181
162182 ### Didn’t verify early enough
183+
163184 Examples:
185+
164186 - waited too long to run a test
165187 - never checked the exact contract
166188 - kept exploring before the first meaningful verification
167189
168190 ### Over-exploration / shell thrash
191+
169192 Examples:
193+
170194 - too many shell / read steps before first write
171195 - repeated discovery commands without narrowing the problem
172196
173197 ### Persistent correctness gaps
198+
174199 Examples:
200+
175201 - `torch-tensor-parallelism`
176202 - `filter-js-from-html`
177203 - `configure-git-webserver`
178204
179205 ### Long-horizon / timeout-heavy tasks
206+
180207 These are useful for promotion runs, not tight inner loops.
181208 Examples:
209+
182210 - `gpt2-codegolf`
183211 - `query-optimize`
184212 - `winning-avg-corewars`
@@ -189,9 +217,11 @@ Examples:
189217 Behavior analysis depends on structured agent logs.
190218
191219 Keep wrappers on:
220+
192221 - `mc --json -p ...`
193222
194223 Per trial, keep:
224+
195225 - result JSON
196226 - verifier output
197227 - exception type
@@ -200,6 +230,7 @@ Per trial, keep:
200230 - timestamps
201231
202232 With those artifacts, analyze things like:
233+
203234 - time to first tool call
204235 - time to first edit
205236 - number of `shell` / `read` / `grep` / `edit` calls
@@ -208,14 +239,6 @@ With those artifacts, analyze things like:
208239 - whether it kept changing files after a pass-worthy state
209240 - whether it left extra artifacts
210241
211-## Recommended first optimization targets
212-
213-Start with:
214-1. exact-contract and stopping discipline
215-2. earlier verification
216-3. less wandering during discovery
217-4. loop nudges that reduce shell thrash without adding a hard step cap
218-
219242 ## Commands
220243
221244 ### Guardrail suite
@@ -311,19 +334,9 @@ done
311334 PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
312335 ```
313336
314-## Session-start prompt
315-
316-In a new session, the quickest way to start is:
317-
318-- read `BENCHMARK.md`
319-- run the guardrail suite
320-- run the fast suite twice for baseline
321-- inspect failures and grouped behavior patterns
322-- propose one small hypothesis and one small change
323-
324337 ## Minimal experiment log format
325338
326-Keep this in the session or in the final summary for each change:
339+Keep this in the PROGRESS.md, a final summary for each change:
327340
328341 - hypothesis
329342 - files changed
PROGRESS.md+2−109
@@ -1,110 +1,3 @@
1-# PROGRESS
1+# CURRENT TASK PROGRESS
22
3-## Conversation summary
4-
5-We reviewed ideas from:
6-
7-- HumanLayer's advanced context engineering / 12-factor agents work
8-- several coding-agent implementations and docs, especially Pi, Codex, OpenCode, Gemini CLI, Amp, and Forge
9-
10-The main conclusion from this round:
11-
12-- mini-coder should stay fairly **shell-leaning** overall
13-- but **file reading** and **content search** should be **dedicated tools**, because shell commands are too ambiguous for those jobs
14- - reading can happen through `cat`, `sed`, `awk`, `head`, `tail`, etc.
15- - searching can happen through `grep`, `rg`, `find`, `fd`, etc.
16-- for simpler repo exploration, shell is still fine
17- - prefer `ls` for directory listing
18- - prefer `fd` for file discovery
19-
20-## Chosen direction
21-
22-The best next direction is:
23-
24-1. **Copy/adapt Pi's `read()` tool implementation** for mini-coder
25-2. **Copy/adapt Pi's `grep()` tool implementation** for mini-coder
26-3. Update mini-coder's prompt so it clearly teaches the model:
27- - use `read()` for reading files
28- - use `grep()` for content search
29- - use shell `ls` and `fd` for lightweight exploration
30-4. Keep `shell` as the fallback tool for everything else, not the primary interface for reading/searching code
31-
32-## Why this direction
33-
34-- Pi's read/search tools are the cleanest practical baseline we found
35-- Codex reinforces the value of keeping search/read more explicit than raw shell
36-- Shell remains valuable, but it is a weak interface for precise read/search intent because the model has too many equivalent command shapes to choose from
37-- `ls` and `fd` are simple enough that dedicated tools are probably not worth adding right now
38-
39-## Important research notes
40-
41-### Pi
42-
43-Useful patterns worth copying:
44-
45-- `read(path, offset, limit)`
46-- `grep(pattern, path?, glob?, ignoreCase?, literal?, context?, limit?)`
47-- clear truncation behavior
48-- clear continuation hints like `use offset=...`
49-- explicit tool guidance telling the model to prefer `read` over `cat`/`sed`
50-
51-### Codex
52-
53-Useful conceptual takeaway:
54-
55-- dedicated tools for common read/search operations are better than forcing everything through shell
56-- search can often narrow candidates first, then follow up with read
57-
58-### OpenCode / Gemini CLI / Amp
59-
60-Useful secondary ideas:
61-
62-- good tool descriptions matter a lot
63-- for open-ended exploration, subagents/delegation can help keep context cleaner
64-- Gemini's grep implementation had strong ideas, especially structured `rg --json` handling and auto-context for small result sets
65-
66-## Current recommendation for implementation
67-
68-Keep scope narrow.
69-
70-Do **not** add a big new tool surface yet.
71-
72-Implement only:
73-
74-- dedicated `read` tool
75-- dedicated `grep` tool
76-- prompt clarification that `ls` and `fd` are the preferred shell commands for exploration
77-
78-Avoid adding new `ls` / `find` / `glob` tools for now unless later testing shows they are needed.
79-
80-## UI decisions from follow-up discussion
81-
82-These details are now decided for the first `read` / `grep` implementation pass:
83-
84-- `read` and `grep` tool-call previews should use the existing shared tool frame and stream their arguments in a structured, styled format rather than raw JSON.
85-- Those previews should follow the same `/verbose` behavior as the shell tool UI.
86-- `read` results should render syntax-highlighted file content as it streams in. That rendered body is still subject to `/verbose` UI truncation.
87-- The `read` result header should include the resolved file path after the `read <-` pill on the same line.
88-- `grep` results should render a structured, styled view derived from `rg --json`, focused on session-CWD-relative filenames and matches rather than raw JSON.
89-- `grep` result bodies should also follow `/verbose` UI behavior.
90-- Relative `grep` paths and displayed relative filenames should be based on the session CWD.
91-- Keep tool concerns separate from UI concerns: tool-level truncation / continuation behavior and UI `/verbose` rendering stay independent.
92-
93-## Likely files to touch later
94-
95-When implementation starts, the likely areas are:
96-
97-- `src/tools.ts`
98-- `src/prompt.ts`
99-- maybe `src/ui/conversation.ts` if tool rendering needs special treatment
100-- maybe tests in `src/tools.test.ts`, `src/prompt.test.ts`, and `src/agent.test.ts`
101-
102-## Open question to keep in mind
103-
104-If we copy Pi's read/search behavior, we should decide whether `read()` should follow the same ignore/security expectations as search tools. One real issue seen in other agents is inconsistent behavior where search respects ignore rules but direct read bypasses them.
105-
106-## Status at the end of this context
107-
108-Nothing has been implemented yet from this decision. The chosen tool surface and the intended `read` / `grep` UI behavior are now documented and should be treated as the plan for the next implementation pass.
109-
110-This file exists only to preserve the research outcome and chosen direction so a new context can continue from here without redoing the comparison.
3+- Started benchmark loop.
benchmark-loop.sh+19−0
@@ -0,0 +1,19 @@
1+#!/usr/bin/env bash
2+set -euo pipefail
3+
4+stop() {
5+ echo "stopping loop"
6+ exit 0
7+}
8+
9+trap stop INT TERM
10+
11+while :; do
12+ echo ""
13+ echo "> Running step"
14+ echo ""
15+ mc -p "See BENCHMARK.md and PROGRESS.md first. Do the next step in the process. Once the step is complete update PROGRESS.md"
16+
17+ sleep 60
18+done
19+
terminal-bench/README.md+2−25
@@ -85,26 +85,9 @@ PYTHONPATH=$PWD/terminal-bench \
8585 terminal-bench/mini_coder_local_agent.py
8686 ```
8787
88-## Current benchmark context
88+## Examples
8989
90-Latest full leaderboard-style run kept for inspection:
91-
92-- `terminal-bench/jobs/leaderboard-2026-04-10__16-06-57`
93-- score: about **67.4%**
94-- pattern: many failures were near misses; the biggest buckets were exact-contract misses and runs that explored too long before writing the required artifact
95-
96-## Fast consistency loop
97-
98-Recommended fast feedback trial: `polyglot-rust-c`
99-
100-Why this one:
101-
102-- it already flipped on the same published build in `leaderboard-2026-04-13__15-33-42`
103-- attempt `polyglot-rust-c__4P9q8rd` failed in about **398s** because it left an extra `/app/polyglot/main` artifact
104-- attempt `polyglot-rust-c__4sRQDtD` passed in about **494s**
105-- that makes it a decent exact-contract consistency check without paying for a full leaderboard run
106-
107-I looked at `break-filter-js-from-html` too because it is faster, but that flip is confounded by safety/refusal behavior. For measuring coding changes, `polyglot-rust-c` is the cleaner signal.
90+### Fast consistency loop
10891
10992 Run the consistency check as 4 attempts and treat the result as `passes/4`:
11093
@@ -135,9 +118,3 @@ PYTHONPATH=$PWD/terminal-bench harbor run -y \
135118 --n-concurrent 1 \
136119 --max-retries 0
137120 ```
138-
139-Guidance:
140-
141-- use this loop before full leaderboard runs
142-- compare changes by the `passes/4` consistency signal, not a single attempt
143-- keep concurrency at `1` here to avoid rate-limit noise