← Evals

Solve rate

Solved runs out of all runs for each agent version, with a 95% Wilson interval. Runs cut short by the machine (a killed process, the LLM server unreachable, the engine timing out) are left out.

• bdd945f 61% 11/18 39%–80%

Against another eval of the same questions

backend edge main bdd945f six multi-llm branch multi-llm main

Within noise: a difference this size could easily be chance with 18 and 18 runs. Fisher's exact test, two-sided, p = 0.318.

Tokens and time

Medians per run. Tokens come from RubyLLM's usage records; tokens per second is completion tokens over the run's wall time.

Every run

One dot per run, green solved, red stuck and grey interrupted, larger for more rounds; a hollow dot has no checks recorded. Hover or tap a dot for its top errors, and open it for the question's page.

Why checks fail

Each version's failed checks, as shares of the whole, by the hint in Question::HINTS their error matches first, or by the error's class when no hint matches. A check that repeats the error before it, in a streak of 3 or more, counts as a repeated failing move.

• repeated failing move 8
• no equation determines the … from what is known 37
• undefined method '…' for an instance of Physics::PhysicalObject 14
• gives nothing called … 12
• deduces from “…%{…”, which it never says 11
• which none of its laws name/give to anything 11
• it takes no argument 8
• which the solution's laws leave out…; or take one 8
• 8 other categories 23

• bdd945f 132 in 18 runs

Most frequent errors

1. 14no equation determines the … from what is knownRuntimeError: no equation determines the force on a wire on a from what is known
2. 13no equation determines the … from what is knownRuntimeError: no equation determines the force on a wire on wire a from what is known
3. 12no equation determines the … from what is knownRuntimeError: no equation determines the mass of ice from what is known
4. 10undefined method '…' for an instance of Physics::PhysicalObjectNoMethodError: undefined method 'time_taken' for an instance of Physics::PhysicalObject; time_taken is a quantity of Velocity, through UniformCircularMotion, not a method: reach it from a motion, as <motion>.time_taken
5. 8deduces from “…%{…”, which it never saysArgumentError: copper_block_on_ice deduces from “Specific heat of copper = %{copper_heat}”, which it never says
6. 8it takes no argumentNoMethodError: undefined method 'distance_from_the_wire' for an instance of Physics::PhysicalObject; distance_from_the_wire is a quantity of MagneticFieldOfCurrents, not a method: reach it from an object, as <object>.distance_from_the_wire; it takes no argument: write <object>.distance_from_the_wire, with qualifiers by keyword (during:, by:) if any
7. 6which none of its laws name/give to anythingNameError: two_forces_at_right_angles takes force1 as force on body by pusher1, which none of its laws name
8. 6which the solution's laws leave out…; or take oneNoMethodError: undefined method 'separation' for an instance of Physics::PhysicalObject; separation is a quantity of CoulombsLaw, ElectricField, TravellingWave, which the solution's laws leave out (add one to using:), not a method: reach it from an object, as <object>.separation, qualified as needed by from: <object>; or take one the solution's laws already name for that kind of quantity: radius, length_of_the_coil, distance_from_the_wire
9. 5undefined method '…' for an instance of Physics::PhysicalObjectNoMethodError: undefined method 'distance_from' for an instance of Physics::PhysicalObject; did you mean <object>.distance_from_the_wire?
10. 3gives nothing called …ArgumentError: two_parallel_wires gives nothing called distance_from_the_wire
11. 3gives nothing called …ArgumentError: two_parallel_wires gives nothing called field_at_a
12. 3gives …, which its solution never takesArgumentError: two_parallel_wires gives separation, which its solution never takes

Where it ran

• agent 17b9257 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  bdd945f on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  edge-thanveer · http://127.0.0.1:8083 · llama.cpp b11371-99b95488c

  model

  qwen3.5:9b · Q4_K_M · 49152 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on battery, 62% → on battery, 60%

  time

  14 Oct 10:10 → 14 Oct 10:14

• agent 17b9257 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  bdd945f on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  edge-thanveer · http://127.0.0.1:8083 · llama.cpp b11371-99b95488c

  model

  qwen3.5:9b · Q4_K_M · 49152 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 18% → on mains, 30%

  time

  14 Oct 10:32 → 14 Oct 10:40

• agent 17b9257 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  bdd945f on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  edge-thanveer · http://127.0.0.1:8083 · llama.cpp b11371-99b95488c

  model

  qwen3.5:9b · Q4_K_M · 49152 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 38% → on mains, 40%

  time

  14 Oct 10:47 → 14 Oct 10:49