Run It or It Didn't Happen: Benchmarks for Small-Model Tool Routing

This paper, by Lewis Tham and Cayden Chik, proposes a benchmark protocol with one unglamorous rule: a published number is admissible only if a command re-runs it green. Honest negatives are recorded next to the wins, and no claim ships whose witness cannot be found and run. We applied it to a small tool-routing model and then argued, narrowly, about which benchmarks a tool layer should be judged by.

Research paper, June 2026. Describes the design at the time. Status as of 12 October 2026: the 0.8B model, its adapters and the witness scripts below lived in our June 2026 research repository, which is not public, and the small model is not part of the hosted Unbrowse service. The tool layer the paper describes as resolve, execute and capture is, as of that date, unbrowse.discover, unbrowse.run and the cloud browser (unbrowse.browse.*), documented at /docs. The "execute, don't guess" rule did carry over: as of that date the hosted service bills verified successes only, and publishes a public route only after a clean replay proves it.

Abstract

We report a benchmark protocol with one rule: a published number is admissible only if a command re-runs it green, honest negatives are recorded alongside wins, and no claim ships whose witness cannot be located and run. Applied to a 0.8B-parameter tool-routing agent, the protocol yielded nine reproduced wins and five honest negatives.

On the task classes the architecture targets (writing correct code, retrieving facts the model cannot know, following settled procedures, and routing to distilled specialists) the accuracy lifts are large, seed-stable and reproducible live: served code correctness 68% to 100%, knowledge via retrieval 0% to 95%, hard reasoning via a specialist 50% to 92%, and skill following 63% to 93%.

We then make a narrow, falsifiable argument. A benchmark that scores a model's own multi-hop reasoning (for example BrowseComp, where this model scores 0/15) is the wrong instrument for a system whose thesis is to offload reasoning to tools, retrieval and execution. The right instrument is answer correctness given tools. We are explicit about the boundary: this holds for tool-completable tasks and not for genuinely reasoning-bound ones, where the model's ceiling is real and unmoved.

1. The protocol: a number is a runnable witness

The failure this protocol exists to prevent is the silent overclaim: a benchmark number that reads as proven but cannot be re-run. Three rules.

1. Witness or it didn't happen. Every published metric maps to a script that exits 0 exactly when the claim holds, scoring a disjoint held-out set through the real generation-and-execution loop, not a mocked stub.
2. Honest negatives are first-class. A measured failure is recorded with its real numbers and never hidden. A benchmark suite that only contains wins is suspect.
3. No fabricated green; verify the harness. A judge or scorer that fails silently (say, returning a tie on every exception) is indistinguishable from a real null until probed. So the harness itself is tested before its verdict is trusted.

The protocol caught a real defect in our own record while we were writing this note. Four headline numbers cited witnesses that were absent from the primary repository. The protocol forced the question; the witnesses were found in a sibling repository, re-run live, and one of them (code correctness) was then stress-tested for reliability (section 3). That is the protocol working, not failing: the discipline surfaced a real gap, witness locality, that prose alone had hidden.

2. Self-improvement results

The same served 0.8B model is used throughout. The lift comes from routing (to a tool, a retrieved fact, a distilled specialist), not from a larger model. Each row below is a witness that starts the server with the relevant adapter and scores a disjoint held-out set. All four were re-run live and green in June 2026.

(a) Correction: the honest baseline. An earlier statement of the code row read "25% to 100%". That 25% was measured against a weak code-adapter baseline, a degraded artifact that scored below the raw instruct model. When the baseline is the raw base model (the true "before" state, and the only one a bare clone can reproduce without a fixed artifact), it scores 67.9%, and the improved adapter reaches 100%. The execute-don't-guess effect is real (+32 points to a perfect score, seed-stable), but the honest lift is 68 to 100, not 25 to 100. The dramatic version depended on an unfair baseline. Making the witness reproducible from a bare clone is what forced the correction: the discipline doing its job on our own number.

(b) Resolved flag: hard reasoning. The hard-reasoning row uses a trained r1 baseline (50%). We measured whether the raw base model beats it. On the hard families (n-choose-k, lcm) the raw base scores 4.2% (1/24), far below the r1 baseline. So, unlike the code row, where a degraded trained baseline inflated the lift, the trained baseline here is conservative: the true raw-base lift is roughly 4% to 92%, larger than 50 to 92. The row understates rather than overclaims. Witness: specialist_basebaseline.py.

The unifying mechanism is execute, don't guess. The small model is not asked to be the oracle. It is asked to write the program, retrieve the fact or follow the procedure that produces the answer, and a deterministic executor then checks it.

3. Reliability: a number you can stand on, with its scope

A single 100% on 28 items is not yet trustworthy. We hardened the code-correctness claim two ways.

Seed stability. Across four seeds of 42 items each, the improved model scores 100% (plus or minus 0), against 32.7% (plus or minus 2) for the weak-adapter baseline. The "25%" was a low seed of that baseline; the lift is stable, not anecdotal.

Generalization scope, the honest boundary. On three unseen families (gcd, digit product, vowel count: code-writable, not trained) the lift is only 11 points, 67% to 78%, well below the in-distribution margin. So the large gain is in-distribution and family-specific. The base model already writes adequate code for novel families.

The reliable restatement: on the trained families, served code correctness is 100%, seed-stable, up from 68% for the raw base model; out of distribution it transfers only partially. That sentence overclaims nothing and survives scrutiny.

4. The reframe: what to measure for a tool-routing system

A tool-routing agent's product value is this: given a user intent, does the right answer come out, cheaply? That is what section 2 measures. It is not the same quantity as can the model, with no tools, reason its way to the answer over several hops?, which is what raw web-agent reasoning benchmarks score.

Our honest score on the second is a negative: 0/15 on BrowseComp [1]. We do not paper over it. But the negative is a property of the model's unaided reasoning, and the architecture's whole thesis is to not rely on that: route to retrieval and execution instead. Scoring such a system by its unaided reasoning is like scoring a calculator by the operator's mental arithmetic. It measures the part the design exists to remove.

So the claim is a reframe, deliberately narrow:

For a system whose thesis is execute-don't-guess, the load-bearing benchmark is answer correctness given tools, not the model's unaided reasoning. On the former we won decisively and reproducibly; on the latter we report an honest negative, because it scores a capability we offload by design.

"Don't need to beat," not "don't matter": the tool layer is not the agent

The sharper statement is not that raw-reasoning benchmarks stopped mattering. It is that they are not our benchmark to beat. BrowseComp scores an agent solving multi-hop web tasks end to end. Unbrowse is not that agent. It is the tool layer the agent calls. In June 2026 we described that layer as three calls: resolve (intent to a ranked route), execute (route to real data) and capture (a browser when no route exists). As of 12 October 2026, the same three roles are unbrowse.discover, unbrowse.run and the cloud browser.

The benchmark for a tool layer is tool reliability: when the agent routes to a tool, does the correct answer come out? That is the execute-don't-guess suite in section 2, plus the route-ranking results in our companion paper, Stop Picking Routes by Vibes, which also records where its ranker loses.

So the division of labour answers the question "do we need to beat it?": the reasoning is the agent's job; the reliable tool call is Unbrowse's. A capable agent that reasons well, plus reliable tools, is what clears BrowseComp. The 0.8B experiments show that even a small agent clears the tool-completable subset once the tools are reliable. The 0/15 is the small model's reasoning score. It does not measure whether the tool layer delivers, and the tool layer is what Unbrowse ships.

This is a claim we have to earn, not just assert. It holds only if Unbrowse is honestly positioned and measured as a tool layer. Where a product instead promises end-to-end research from the tiny model alone, the agent-reasoning score is fair game, and the honest path is to pair Unbrowse with a capable agent, never to pretend the 0.8B model did the reasoning. With that honesty kept, the answer to "why don't those benchmarks need to be beaten for us?" is: because they grade the caller, and we are the tool the caller holds.

The same argument runs through our April 2026 paper, Internal APIs Are All You Need [2]: an agent should call the first-party request a site already makes instead of reasoning its way through rendered pages. For a plain-language summary, see the arXiv paper explained.

5. The boundary: where the reframe is false

The reframe is not "reasoning benchmarks don't matter." They matter exactly as the measure of the residual that tools cannot cover. Two limits hold.

• Reasoning-bound tasks. When a task genuinely requires multi-hop inference that no tool or retrieval can shortcut, the small model's ceiling is real and unmoved. Distillation cannot transfer a capability the teacher itself lacks, and retrieval cannot fetch a fact that must be derived. There, the raw-reasoning score is the right one, and ours is low.
• Out-of-distribution generation. Section 3 shows the code-correctness gain is in-distribution. A benchmark of novel task families is a legitimate, harder yardstick that we only partly pass.

A reader should leave with the bounded version, never the dismissal: benchmarks that measure raw model reasoning are the wrong yardstick for the value a tool-routing system delivers, and the right yardstick for the capability it does not.

6. Reproduce

Every number above mapped to a command in our June 2026 research repository. That repository is internal, so these are listed as a record of what backed each number, not as commands you can run.

Each execute-don't-guess witness started the served 0.8B model with the relevant adapter and scored a disjoint held-out set through the real generation-and-execution loop. The mechanism results (route ranking, intent-type classification, the energy-ranker gates, content-addressed cache reuse) needed no model serving. The honest negative had its own gate, run to show the limit rather than hide it. A machine-readable ledger recorded the full set: nine reproduced wins, five honest negatives.

7. Conclusion

The contribution is two things, neither of which is a leaderboard victory.

First, a benchmark protocol that makes a number mean "re-runs green", and that, applied to our own record, caught a real witness-locality defect and an unfair baseline rather than rubber-stamping them.

Second, a reframe: for an execute-don't-guess system, measure the answer it delivers given tools, where a 0.8B model reached 100%, 95%, 92% and 93% on its target task classes, and record honestly the raw-reasoning score it does not chase (0/15). We do not claim the older benchmarks stopped mattering. We claim they measure a different quantity from the one this architecture optimizes, and we are willing to be judged by the witnesses, not the prose.

The one previously open item was resolved by measurement: the hard-reasoning row's baseline is a trained r1 specialist, and the raw base scores only 4.2% on those families, so that row is conservative, not inflated.

To try the tool layer itself, connect the remote MCP server at https://unbrowse.ai/mcp; setup is in the docs.

References

1. J. Wei et al. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516, 2025. https://arxiv.org/abs/2504.12516
2. L. Tham et al. Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures. arXiv:2604.00694, 2026. https://arxiv.org/abs/2604.00694
3. L. Tham, C. Chik. Stop Picking Routes by Vibes: Fractal Cell Energy for Route Selection over an Agent's Tool Library. Unbrowse, June 2026. https://unbrowse.ai/blog/stop-picking-routes-by-vibes