← Evals

Solve rate

Solved runs out of all runs for each agent version, with a 95% Wilson interval. Runs cut short by the machine (a killed process, the LLM server unreachable, the engine timing out) are left out.

• f0f5267 71% 5/7 36%–92%, 1 interrupted
• f0f5267 60% 3/5 23%–88%, 1 interrupted

f0f5267 against f0f5267

Within noise: a difference this size could easily be chance with 7 and 5 runs. Fisher's exact test, two-sided, p = 1.0.

What changed between the two versions, in the agent and the engine: model_parameters.

Against another eval of the same questions

backend edge backend edge-thanveer main bdd945f six multi-llm branch

Within noise: a difference this size could easily be chance with 12 and 18 runs. Fisher's exact test, two-sided, p = 1.0.

Tokens and time

Medians per run. Tokens come from RubyLLM's usage records; tokens per second is completion tokens over the run's wall time.

Every run

One dot per run, green solved, red stuck and grey interrupted, larger for more rounds; a hollow dot has no checks recorded. Hover or tap a dot for its top errors, and open it for the question's page.

Why checks fail

Each version's failed checks, as shares of the whole, by the hint in Question::HINTS their error matches first, or by the error's class when no hint matches. A check that repeats the error before it, in a streak of 3 or more, counts as a repeated failing move.

• repeated failing move 19
• no equation determines the … from what is known 32
• ArgumentError 31
• gives nothing called … 19
• undefined local variable or method '…' for an instance of Physics::Solving 17
• undefined method '…' for an instance of Physics::PhysicalObject 10
• NoMethodError 8
• which the solution's laws leave out…; or take one 6
• 7 other categories 15

• f0f5267 84 in 8 runs
• f0f5267 73 in 6 runs

Most frequent errors

1. 20no equation determines the … from what is knownRuntimeError: no equation determines the force on a wire on wire a from what is known
2. 12no equation determines the … from what is knownRuntimeError: no equation determines the time taken of braking from what is known
3. 11no equation determines the … from what is knownRuntimeError: no equation determines the mass of ice from what is known
4. 8undefined local variable or method '…' for an instance of Physics::SolvingNameError: undefined local variable or method 'wire_b' for an instance of Physics::Solving
5. 7undefined method '…' for an instance of Physics::PhysicalObjectNoMethodError: undefined method 'time_taken' for an instance of Physics::PhysicalObject; time_taken is a quantity of Velocity, through UniformCircularMotion, not a method: reach it from a motion, as <motion>.time_taken
6. 7ArgumentErrorArgumentError: wrong number of arguments (given 0, expected 1)
7. 6gives nothing called …ArgumentError: copper_block_on_ice gives nothing called ice_mass
8. 6undefined local variable or method '…' for an instance of Physics::SolvingNameError: undefined local variable or method 'ice_mass' for an instance of Physics::Solving
9. 5is a word Ruby already usesArgumentError: round is a word Ruby already uses: call it something else
10. 5which the solution's laws leave out…; or take oneNoMethodError: undefined method 'separation' for an instance of Physics::PhysicalObject; separation is a quantity of CoulombsLaw, ElectricField, TravellingWave, which the solution's laws leave out (add one to using:), not a method: reach it from an object, as <object>.separation, qualified as needed by from: <object>; or take one the solution's laws already name for that kind of quantity: length_in_the_field, radius_of_the_path
11. 4gives nothing called …ArgumentError: two_parallel_wires gives nothing called length_section
12. 4ArgumentErrorArgumentError: car_braking takes distance as (position of car at end of braking - position of car at start of braking), a formula of quantities; don't work it out, name the quantity and the engine finds it: that formula is what the laws give for car.displacement(during: braking).magnitude, use car.displacement(during: braking).magnitude for distance

Where it ran

• agent 65a7143 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  f0f5267 on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  http://localhost:11434/v1 · Ollama 0.33.3

  model

  phryzby · a823692233f8 · Q4_K_M · 16384 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 20% → on mains, 9%

  time

  14 Oct 05:32 → 14 Oct 05:50

• agent 65a7143 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  f0f5267 on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  http://localhost:11434/v1 · Ollama 0.33.3

  model

  phryzby · a823692233f8 · Q4_K_M · 16384 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 10% → unknown

  time

  14 Oct 05:51

• agent 8f71ceb · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--multi-llm-main

  commit

  f0f5267 on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  http://localhost:11434/v1 · Ollama 0.33.3

  model

  phryzby · a823692233f8 · Q4_K_M · 16384 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 23% → on mains, 4%

  time

  14 Oct 06:13 → 14 Oct 06:31