Fieldbook · Grid of 20 September 2026

What the data shows

Bands, not positions

Sorted by passes out of all tasks: an incomplete answer counts against, because a build that cannot finish a task inside the cap has not delivered it. The ceiling is the rate if every incomplete answer had passed. A build joins the current band while its 95% Wilson interval overlaps its band leader's. The interval reflects only which 14 tasks were chosen; run-to-run noise needs repeated runs. Builds in one band are not ranked against each other.

* "Old" counts an incomplete answer as a pass when its visible text satisfies the rule. It is what the earlier hand-written table used, shown for continuity only.

Build × task

P passed · · failed · ? incomplete (cut off or errored). Rows keep band order; read across a row, not down to a rank.

Task 1, examined

Task 1 — Structured Tool-Call Generation. The prompt asks for "the exact tool-call sequence" to find Python files importing requests and replace requests.get with httpx.get. The scoring rule requires search → read → write. That sequence appears only in the hidden pass criteria; the runner removes them before the prompt is sent.

A failing answer from qwen3.8-27b@q5_k_m — one command, which does what the request says:

{"tool":"run_command","args":{"cmd":"grep -RlE 'import requests\\b' --include='*.py' . | xargs -r sed -i -E 's/\\brequests\\.get\\b/httpx.get/g'"}}

A passing answer from dirk-qwen3.8-27b:

{"tool":"search_code","args":{"pattern":"import requests","glob":"*.py"}}
{"tool":"read_file","args":{"path":"<each file from search result>"}}
{"tool":"write_file","args":{"path":"<same file path>","content":"<original content with all 'requests.get' replaced by 'httpx.get'>"}}

Reported earlier, not yet reproduced

From a previous hand-written report; the raw data is not in this repository. Treat these as claims.

• A 4,096-token cap turned a 13/14 build into 7/14 on the same text.
• Some GGUF chat templates reject a separate system parameter, so the Anthropic Messages API fails on them while the OpenAI-style API works.
• Without --gpu max, LM Studio can silently split a model between GPU and CPU.
• Executable repair of six real defects from a private repository: 0 valid fixes in 30 runs. The cases stay private; the method and results will be published.

Second ruler: live orchestration · 4 October 2026

PASS = the final report holds all four facts (Java version, 74 tests, APK size, "no file changed"), the project was not touched, the session ended on its own. ~ PARTIAL = at least two facts. x FAIL = anything else (did not finish, endpoint error, 10-minute timeout, wrote into the project, fewer than two facts). Bands use the same rule as the grid above: a model joins the band while its 95% Wilson interval overlaps the band leader's. With three runs the intervals are wide, so expect few bands; models in one band are not ranked against each other.

Repeated models. 14 models ran at least twice; 2 passed every run; 4 had the same verdict in every run. Across the 42 runs of these models: pass: 17, partial (2 or 3 facts): 14, 10-minute timeout: 6, wrote into the project: 4, report holds fewer than 2 facts: 1.

Even the best against the worst separate only weakly: 3 of 3 against 0 of 3 gives a Fisher exact two-sided p = 0.10. Three runs per model can order the extremes, not the middle.

2 of the 16 qualifying models (nail-qwen3.6-35b-a3b-mtp, qwen3.8-27b-omnimerge-v6-mtp) were no longer installed in LM Studio when runs 2 and 3 were made, and keep their single run-1 result.

Screened once, not repeated. These did not meet the repeat rule in run 1. One run is an observation, not evidence that a model fails reproducibly.

Addendum: the two missing models, one extra run each. Both were reinstalled after runs 2 and 3 and run once, same prompt and settings, outside the registered protocol. They do not change the table above.

Replication check, same day (outside the registered protocol)

Models added after the confirmatory runs (same day, outside the registered protocol)

Each was downloaded later and run with a fixed LM Studio configuration saved next to the result: context 100000, KV cache q8_0, temperature 0.7, top_k 20, top_p 0.90, min_p 0, repeat penalty 1.0. Only one model, Qwen3.8 27B GSQ RCO (IQ3_S), got repeats: it passed in two of three confirmatory runs and in its first screening run. The dense Holo4 27B was run once at the registered 10-minute limit and three times at 20 minutes; the extra time did not change the outcome. Published benchmark scores for that model are for computer-use tasks, which this verification task is not. One run per model is an observation, not a ranking.

The arm was corrected after this ruler showed its defects, and the same runs were repeated

v1 has no run of cyber-tiel under these exact settings; v1 for holo4 is the replication cell with the same settings. Each cell is three runs, in order.

What the tables show. The pass count moves within noise from v3 to v14 (9, 8, 8, 7, 8, 6, 5, 6, 5, 4, 7 and 7 of 12; three runs per model neither show a decline nor exclude one), so the corrections cannot be credited with a better model score. What changed is the mechanism: in v4 no run wrote into the project and the read-only policy refused no command, where v3 had six refusals of read-only commands and v2 refused the builds the task asked for. A correction was judged by the failure it removed, not by the pass rate. v5 did not do what it was meant to: giving the report step the latest command output left the number of reports lacking a fact at 4 of 12 (the missing test count fell from 4 to 3 runs, other facts went missing instead). In v5 the policy refused 14 calls, all of them the same model asking again and again to write a file into the read-only project in one run, which ended at the time limit: the refusal was right, and the model had no way out of it.

v6 closes tool use for the rest of a step after five refused calls. It was judged against four criteria written before the runs: no run writes into the project (met, 0 of 12); no read or build command is refused wrongly (met, one refusal in twelve runs, a real attempt to create a source file); a closed step ends in text (not exercised: the limit was never reached, so it is shown by a unit test and not by a live run); no clear fall in passes (met, 7 of 12 against 8 in v5). The same open faults showed again: one run ended at the time limit after the evidence gate refused the final step three times, and the test count was missing from two of three reports of one model.

v7 let a replan depend on steps already done, gave the evidence gate and the read-only policy one definition of a write (a log written to a temp directory is not a change to the project), and kept early command output for the report step. Its four criteria, written before the runs, were all met: no write into the project; no "files changed: none" refused because of a temp-directory write; fewer than four reports with two or three of four facts (three); at least six passes (eight).

v8 corrected a real fault found by reading the logs: the lane that revises a draft was shown only the tool output of the last step, which runs nothing, so it saw "none" and softened true reports into "unable to report"; its evidence summary also kept the first eight items, which cut the build and the listing at the end of a task. Its live result did not show the expected effect, and its criteria were not met: no write into the project (met), but one "no change" contradiction (not examined), six reports with a fact missing against a target of at most two, and six passes against a target of eight. The missing fact is the test total in six of twelve runs; reading the reports shows that part of this is a model listing the per-class counts without adding them up, which the check does not accept. The check is a stricter proxy than the wording of the task, so this measure mixes two things. The correction stays because the fault was real and a test shows it; the live data do not show that it helped.

v9 carried the corrections made after the outside review (the read-only filter, the evidence gate for a report backed by a test run, the labelling of command output, the scope of the data given to another endpoint). It was judged against four criteria written before the runs. Met: no run wrote into the project (0 of 12); no read or build command was refused wrongly (one refusal, a real attempt to use a writing tool). Not met: the evidence gate refused in at most three of twelve runs (it refused in seven, as in v8) and at least seven passes (five). The correction to the gate for a report backed by a test run therefore did not lower the number of refusals in live runs: that defect does not explain most of them, and their cause stays open. No correction made after the review had, at that point, shown an effect in a live run.

v11 carries the open items of the outside audit: more shell writers denied; a core of "writes the project" (the writer commands, touch, the scratch rule) shared by the tool policy and the evidence gate, which still differ on a substitution, a delete after a pipe and an interpreter write; a quoted absolute path counts as scratch only under the workspace's own scratch directories; a capability model that refuses a tool nobody classified; secrets replaced before text leaves for a different verifier endpoint; and, switched on by a flag for this batch, the workspace of a read-only task mounted by the operating system as a throwaway overlay (checked first on one real project: the build and the unit tests ran inside it and the project on disk did not change; it was applied in all 12 runs). Criteria written beforehand: no write into the project (met, 0 of 12); at most three runs with a gate refusal (met: two); at least six passes (not met: five; the second scorer, which also accepts per-class test counts, gives eight); no read or build command refused wrongly and no build broken by the overlay (met). All eleven refusals came from one rule that read the word "wrote" in a shell command or its output as a file write; it was corrected afterwards, which is why the pass count fell short. The gate diagnostics found two defects of the same kind while this batch was prepared: a write the runner had recorded to a scratch directory counted as a change (a first attempt was stopped after three runs and is not counted), and the same wording rule. A confirmation batch of the corrected build is registered before its runs. What is and is not shown: the overlay was checked on one project, whose Android SDK lies inside it; a project with a toolchain in the home directory needs it left off. The capability model refuses unknown tools in a read-only task, which a custom tool may need to be told to pass. Read-only remains a best-effort guard, not a security boundary.

One more model, collected as data only (arm-iterations/v11-kat-addendum, same task, same settings, 100,000 tokens of context loaded, 23.7 GB of 24): a code model of about 22 GB passed 3 of 3 runs, 185 to 255 seconds, no write and no refused command. Three runs do not rank it.

A batch that measured the wrong thing, and a change of instrument. A confirmation batch of the arm after v11 came out with every report scored PARTIAL, with only the test total missing. It was not the arm: between two batches the project had gained a test (the owner had asked a model to add one in the application window), so it now held 75 tests, and the two earlier scorers look for the literal 74 or for four fixed class counts. The reports said 75, and were right. The batch was set aside as invalid for scoring (seven complete runs, published separately as v12-invalid-ground-truth-75-tests, one interrupted run recorded as interrupted and not as a failure) and not repeated as if nothing had happened. Scorer 3 reads the expected test counts from the project's own result files at the start of each run and stores them with the run; it was fixed before the next batch, v1 and v2 stay as they were, and only runs that carry scorer 3 are scored with it (the v12 and v13 columns). Recomputing the seven invalid runs after the fact with 75 gives 4 passes; that is labelled as such and not counted in any column.

v12 (build with the wording-rule fix, a detector that no longer reads "explain how to fix" as a request to change, a gate that sees what the policy refuses, scratch targets written with $PWD, an overlay that refuses symlinked scratch directories, and two noise fixes: no ghost banner on a plan step that is already done, middle steps told not to write the final report). Criteria written beforehand: no write (met, 0 of 12); at most three runs with a gate refusal (met: none, the first batch with no refusal at all); at least six passes (not met: four); no command refused wrongly (met). Four reports were incomplete in a fact other than the test total, against one or two in the three earlier batches; the cause is not established (a planner change that tells middle steps not to write the report is the candidate, twelve runs do not show it) and it is stated as open.

v13 applies an external audit patch unmodified (hashes of the ten files equal to the manifest of its author; type check, its own 146 tests, the golden regression and the full suite equal to the recorded red baseline apart from one check that needs the new files committed). It budgets the tool schemas offered to a local model, validates the arguments of a hidden target tool before it runs, gives the count request its own deadline, and widens the loopback detector. Its two opt-in flags were left off, so the batch measures only the default-on changes. Criteria written beforehand: no write (met), at most three runs with a refusal (met: none), at least four passes, the v12 figure (met: seven of twelve), nothing new in the logs about schemas, validation or the count hook (met). Seven against four is not an improvement that twelve runs can separate, and with the flags off the new paths barely run: this shows that the patch did not break the task, not that it helps. The effect of the two flags is not measured.

v14 applies a second external audit patch unmodified, its new flags off: 7 of 12 passes, the same as v13; no run wrote into the project; two runs hit the 600 s limit and two did not finish. No criteria were frozen for this batch, so it is reported as it came out (arm-iterations/v14-external-patch-v3).

cmp1, a head-to-head with the rules frozen first (comparisons/criteria-cmp1.txt): the previous line of the arm against the next one, three runs per model, interleaved. 7 of 12 passes against 3 of 12; Fisher exact two-sided p = 0.214, so there is no separable difference, and the direction does not favour the next line. Latency (median 325 s against 304 s) is also not separable. No write, no refusal, no new failure kind in the logs. Success and latency are reported apart, and unfinished runs and 600 s limits stay in the denominators as the harness scored them.

cmp2 was stopped early by the operator after 25 of its 40 planned runs, so it has no verdict. Partial counts, noise at this size: on the prompt where the patch's new report lane can engage, 5 of 8 passes without the patch and 2 of 8 with it (the lane engaged in 4 of those 8); on the control prompt, 3 of 5 and 2 of 4 (lane never engaged, as designed). Nothing was written into the project. The patch was adopted for a different reason, not because of this comparison.

Five more external patches were applied one at a time, each unmodified in its own commit, every repair in a separate commit, each compared by failing-test identity against the same baseline and against a golden set. The checks found tests that pinned source text rather than behaviour and, in my own earlier repair, a refusal recognised by its output text, which a second review showed could hide a real write; it is now a typed signal. No live gain from these patches is measured. Manual runs in the application window then found four faults in the interface layer that scripted runs cannot see (a missing Stop control during a session's first run, an internal refusal code quoted to the user, a finished task's report thrown away, an approved plan run as one turn); three are fixed and tested, one is open. See field-notes/2026-10-10-manual-runs-what-the-harness-missed.md.

Correction 9. The per-version table recomputed a verdict that cannot see whether a run finished. One cell was wrong (v12, holo4, run 2 should be x); no pass count changed. See CORRECTIONS.md.

One scenario in the application window (an owner-run check of a build with the corrections, one model, one coding task: add one unit test to an existing class): the read-only policy did not switch on, every step after the first received the record of files written, no ghost banner appeared, the file list in the report was correct and explained, and the test count on disk matched the report. A report-like message still appeared in one middle step, so the report showed twice, down from three or four times. The test the model added asserted on a local value and did not call the code under test, in the style of the class it was added to; it was replaced afterwards by one that calls the real method. This is one scenario, so it is "validated on one scenario", not on the build.

A second batch of the v10 arm (same behaviour; the gate log gained diagnostics, the harness stores the full report and a second test fact that also accepts the four per-class counts, fixed before the runs). Criteria written beforehand were all met: no write (0 of 12), gate refusals in 2 of 12 runs, 7 passes against a target of 6, no wrongful refusal. Scorer v2 changed no verdict (7 of 12 under both): the reports that lacked the test fact gave one class count (three holo4 runs) or counts the model itself said it had inferred, so the limit seen in v10 did not recur as a formatting problem. The diagnostics located the refusals: all three fell on the first step of a plan, before that step had run a tool, and the earlier lead of a refusal with a counted write did not recur. Reading the logs resolved it: in those runs step 1 ended twice with text and no tool call, and the gate correctly refused text that claimed results without evidence; on an intermediate step the runner treats the refusal as advisory, so it did not stop the run. It is the model answering before running anything, not a gate defect, and the criterion counts these advisory refusals too.

v10 followed from a measurement, not a guess. The gate was made to report the conditions of its read-only exemption, and for a report whose only evidence was a listing it said that no inspection had happened: its evidence text holds tool outputs, not commands, and models inspect almost only through the shell. A shell command that writes nothing and prints something now counts as an inspection, as it already did in the read-only policy. Criteria written before the runs: no write into the project (met, 0 of 12); the gate refused in at most three of twelve runs (met: two, against seven in v8 and v9, the first correction that showed its effect live); at least seven passes (not met: six); no read or build command refused wrongly (met, none). The gate is therefore not what held the pass count down. In five of the twelve runs the only missing fact is the test total, and in four of them it is the only fact missing: a report listing the per-class counts without adding them up is scored as missing it. The scorer is a stricter proxy than the task's wording, so most of what separates six passes from ten is a property of the measurement, not of the arm; a scorer that accepts the per-class counts would have to be fixed beforehand and applied to every version, and has not been written. The two refusals left (one run where the gate saw a mutation although the policy refused no write, one where it was handed a single observation) are leads, not explanations.

External review (2026-10-06). An independent review of the arm and of these corrections read the sources and ran the classification functions, without live model runs. Its verdict, in short: the corrections fix real faults, but the read-only mode is a best-effort guard and not a guarantee that no file changes, and it should not be presented as a security boundary. It also found that text printed by commands reaches the prompt under a heading of apparent authority, that the verification lane was given more data than before, that the evidence gate refused a true read-only report backed by a test run, and that the public scorer and the harness defined success slightly differently. It also found that one figure on this page was miscounted: the evidence gate refused in seven of twelve v8 runs, not nine (corrected above). What was reproduced was corrected and is listed below; what was not corrected is stated as open. The review itself is private because it describes ways around the guard.

Measurement faults found on the way, all outside the arm: the first runs did not record the KV-cache type or the sampling values; a result file was named differently from what the runner looked for when a model key held an at-sign; a vision projector next to a model made a model not fit in memory; and a restart of the runtime's server during one batch made twelve loads fail, so those twelve results were set aside as invalid and are not in any table.

Did not load (excluded from rates): qwen3.8-27b-twinturbo-residual-bonsai-2-pq2_0.

Harness notes (secondary)

Observations about the arm, not about the models. None was changed in order to obtain the main table above; the corrections made afterwards are in the section on the arm's iterations.

• The final step is handed recorded facts, and they omit raw command outputs a prompt may ask to paste (a version string, the last lines of a build). Models either re-run the commands (a large token cost) or say they cannot verify them. That lowers every model's report score, honest ones included.
• "Do not modify any file" is not enforced at tool level. Several models wrote report files or unpacked a build artefact into the project; the evidence gate noticed only when it judged the report.
• One model family fails at once with a grammar-compiler error from the runtime (HTTP 400) before any tool runs, while a smaller model of the same family does not. A tool with an empty-object parameter schema is a candidate cause, not tested.
• Several models lose the plan: the proposal contains no parseable plan twice and the run continues without one.

Not reproducible end to end from this repository: the arm's code is not published. The prompt, load settings, runner, analysis, sanitiser and every run's sanitised result are.

Limits

• One machine; speeds do not transfer. The grid has one run per build; the live-orchestration ruler repeats the models it qualifies.
• Model files are community builds with unstable names, and file hashes are not yet recorded.
• Pass rules are keyword and structure checks, not execution. A pass is not a verified solution.
• The saved answers are outputs of third-party models, each under its own license; the author claims no rights over them.

Reproduce

python3 harness/score_grid.py data/grid-2026-09-20/answers
python3 harness/task1_failure_modes.py data/grid-2026-09-20/answers --json data/grid-2026-09-20/task1-failure-modes.json
python3 harness/build_report.py

No GPU needed. To add runs: RUNS=3 harness/run_all.sh <model-key>.