Claim C-029
The ladder against Nemotron 3 Ultra alone
Both contestants run every step on Token Factory, on the same tasks, with the same tests deciding whether a task passed. The ladder routes each step by its type; Ultra alone does every step itself.
Dollars per task run
Where the ladder’s dollars went
Tokens per task run
The same results as tables, per suite
Recorded runs
Replay a task
Every model call, gate and review of a recorded task, in order. The escalation demo starts the edits on Nemotron 3 Nano, so a failed gate is likely.
The policy
The ladder
Three rules in stepswitch.toml decide which model takes each step.
When a change fails the gates twice, the task moves up to Ultra.
| Model | Steps | Price per million tokens (in / cached / out) |
|---|
Per step type
Model calls of each step type in the complete runs: which model the ladder used, and what a call cost on average compared with Ultra doing the same kind of step.
| Step type | Ladder model | Calls (ladder / Ultra) | Tokens per call (ladder / Ultra) | Dollars per call (ladder / Ultra) | Tool errors (ladder / Ultra) |
|---|
Claim C-030
Two reviewers, two model families
In the ladder, Nemotron 3 Ultra reviews each change and gpt-oss-120b gives a second opinion on the same diff. The change is approved only when neither objects.
Not part of the comparison
More runs on the same tasks
Separate runs on the same 21 tasks. Each is a recorded run you can replay; none is a claim.
Edits starting on Lightning (exploratory)
The ladder with edit, run and triage starting on Lightning: Super takes them after the first escalation, Ultra after a second. One run per task, to see whether the edit loop, where Super spends most of the ladder’s dollars, can start one rung lower.
Lightning alone, and the pilot
The ladder’s smallest model doing every step itself, after the main runs, and the five-task pilot that calibrated the dollar caps.
Provenance
Where the numbers come from
Each run, the commit it ran on, its policy, and the command that re-runs it.