← Evals

Solve rate

Solved runs out of all runs for each agent version, with a 95% Wilson interval. Runs cut short by the machine (a killed process, the LLM server unreachable, the engine timing out) are left out.

• bd007a8 75% 3/4 30%–95%

Against another eval of the same questions

pr72 v4 branch rerun 1 pr72 v4 branch rerun 2 pr72 v4 branch rerun 3 pr72 v4 main rerun 1 pr72 v4 main rerun 3

Within noise: a difference this size could easily be chance with 4 and 4 runs. Fisher's exact test, two-sided, p = 1.0.

Tokens and time

Medians per run. Tokens come from RubyLLM's usage records; tokens per second is completion tokens over the run's wall time.

Every run

One dot per run, green solved, red stuck and grey interrupted, larger for more rounds; a hollow dot has no checks recorded. Hover or tap a dot for its top errors, and open it for the question's page.

Why checks fail

Each version's failed checks, as shares of the whole, by the hint in Question::HINTS their error matches first, or by the error's class when no hint matches. A check that repeats the error before it, in a streak of 3 or more, counts as a repeated failing move.

• repeated failing move 5
• which none of its laws name/give to anything 13
• undefined method '…' for an instance of Physics::PhysicalObject 12
• NoMethodError 8
• no equation determines the … from what is known 2
• undefined local variable or method '…' for an instance of Physics::Solving 2
• ArgumentError 1
• SyntaxError 1
• 2 other categories 2

• bd007a8 46 in 4 runs

Most frequent errors

1. 14undefined method '…' for an instance of Physics::PhysicalObjectNoMethodError: undefined method 'time_taken' for an instance of Physics::PhysicalObject; time_taken is a quantity of Velocity, through UniformCircularMotion, not a method: reach it from a motion, as <motion>.time_taken
2. 7NoMethodErrorNoMethodError: undefined method 'time_taken' for an instance of Physics::Path
3. 5which none of its laws name/give to anythingNameError: two_forces_push takes force1 as force on body by pusher1, which none of its laws name
4. 2no equation determines the … from what is knownRuntimeError: no equation determines the time taken of braking from what is known
5. 2undefined local variable or method '…' for an instance of Physics::SolvingNameError: undefined local variable or method 'motion' for an instance of Physics::Solving; did you mean options?
6. 1which none of its laws name/give to anythingNameError: two_forces_push chooses a root of #<struct Physics::Pointing size=#<Physics::Whole:0x00007284ee7fb288 @name=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pushed: ""}>, @written=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pushed: ""}>>>, which none of its laws name
7. 1SyntaxErrorSyntaxError: solution.rb:7: syntax errors found: unexpected ':', ignoring it: unexpected ':', expecting end-of-input
8. 1gives … but never says where in what it asksArgumentError: a_whirled_stone gives turns but never says where in what it asks
9. 1which none of its laws name/give to anythingNameError: two_forces_push chooses a root of #<struct Physics::Pointing size=#<Physics::Whole:0x00007f9050bfba70 @name=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pushed: ""}>, @written=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pushed: ""}>>>, which none of its laws name
10. 1which none of its laws name/give to anythingNameError: two_forces_push chooses a root of #<struct Physics::Pointing size=#<Physics::Whole:0x00007167b783ecb8 @name=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pusher1: "p1", pusher2: "p2", pushed: ""}>, @written=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pusher1: "p1", pusher2: "p2", pushed: ""}>>>, which none of its laws name
11. 1which none of its laws name/give to anythingNameError: two_forces_push takes force1 as force on body by pusher1 during pushed, which none of its laws name
12. 1which none of its laws name/give to anythingNameError: two_forces_push chooses a root of #<struct Physics::Pointing size=#<Physics::Whole:0x00007cece26de8a8 @name=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pusher1: "p1", pusher2: "p2", pushed: ""}>, @written=#<struct Physics::Instance quantity=:force, variable=:F, owner=:body, qualifiers={by: :pusher1, during: :pushed}, labels={body: "", pusher1: "p1", pusher2: "p2", pushed: ""}>>>, which none of its laws name

Where it ran

• agent 17b9257 · engine 453fb0d

  machine

  omarchy

  worktree

  /home/fazzledev/Work/phryzby-app--pr72-main-eval

  commit

  bd007a8 on HEAD

  clean

  yes

  ruby · rails

  3.4.7 · 8.1.4

  llm server

  edge-thanveer · http://127.0.0.1:8083 · llama.cpp b11371-99b95488c

  model

  qwen3.5:9b · Q4_K_M · 49152 context

  gpu

  Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] (rev c1); Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c2)

  power

  on mains, 79% → on mains, 79%

  time

  14 Oct 15:12 → 14 Oct 15:16