bayes for days
sds comparison-vehicles.md
9.7 kB · 184 lines
Markdown
at main

Vehicles score honestly: what moved #

What this changes #

Three weighted columns read exactly 0.000 for any target the hit tables did not describe, and a combat vehicle was one. Measured on the same volley from the same shooter, against the fixture's Mek and against a tank built from it:

expected damage p_breach value_destroyed at_stake p_mission_kill
Mek 12.500 0.100 0.0231 123.00 0.000
tank, before 12.500 0.000 0.000 0.00 0.000
tank, after 12.500 0.193 0.0000 65.00 0.387

Reproduced by features::firing::tests::the_same_volley_scores_a_tank_and_a_mek_on_the_same_columns.

value_destroyed stays at 0.0000 for both targets on a volley this size, and that is not the old failure wearing the same number: at_stake moved from 0.00 to 65.00, so the denominator is now a measurement rather than an absence, and a volley that actually destroys a location scores against it. On a six-by-twenty volley the same tank reads value_destroyed 21.14.

The tank's p_breach is now higher than the Mek's, which is the shape of the model rather than a thumb on the scale: a vehicle's four arcs are four of its locations, so the front table puts every roll but the turret's three onto two places, and the same volley opens one sooner.

What I predicted, before looking #

Stated here before the run was read.

  1. The vehicle share of target selection rises. Three positively-weighted columns went from 0.000 to positive for vehicles and did not move for Meks, so a menu holding both should tip toward the vehicle. I expected a rise of at least 10 points.
  2. answered does not move. Nothing here changes which decisions the bot can answer, only what it scores them at. A drop would mean I had introduced a refusal.
  3. The win rate against Princess is not measurable at this sample size. The harness prints its own power calculation: about 385 decided games to separate 5 points, 97 to separate 10. I could run 24 an arm. So any win-rate difference in this document is noise being reported honestly, not a result.

What actually moved #

Two arms of 24 matches each over scenarios/mixed, the same 12 scenarios, sds against Princess. Before is origin/main's bot; after is this branch.

before after
FIRING decisions 548 534
answered 548 (100.0%) 534 (100.0%)
both kinds on the menu 167 138
chose the vehicle 30 49
vehicle share 0.1796 0.3551

Shift +17.5 points. Pooled SE 0.0504, z = +3.48.

Prediction against outcome #

  1. "The vehicle share rises by at least 10 points." Confirmed - +17.5, and the direction is the one the mechanism predicts. Three positively-weighted columns went from 0.000 to positive for vehicles and did not move for Meks, so a menu holding both tips toward the vehicle. Nothing here was tuned to produce it: no weight was touched on this branch.
  2. "answered does not move." Confirmed - 100.0% in both arms. The change alters what decisions score at, not which the bot can answer.
  3. "The win rate is not measurable at this sample size." Confirmed, and it is not claimed. The harness prints its own power calculation: about 385 decided games to separate 5 points, 97 for 10. Each arm is 24. No win-rate difference is reported in this document, and any that appears in the run summaries is noise.

Per scenario, because a pooled share can hide one scenario carrying it #

scenario before after
2v2-jihad-s5220000 3/16 0.188 5/10 0.500 up
2v2-succession-wars-s5020000 5/15 0.333 5/6 0.833 up
2v2-succession-wars-s5020001 0/18 0.000 4/20 0.200 up
4v4-clan-invasion-s5140000 6/22 0.273 11/16 0.688 up
4v4-clan-invasion-s5140001 5/13 0.385 5/5 1.000 up
4v4-jihad-s5240001 4/17 0.235 4/12 0.333 up
4v4-succession-wars-s5040000 3/27 0.111 9/31 0.290 up
4v4-succession-wars-s5040001 4/39 0.103 6/38 0.158 up

Eight of eight move the same way. A sign test on the direction alone gives p = 0.008, and it needs no assumption about the pooled variance. One scenario went from never choosing a vehicle (0 of 18) to choosing one a fifth of the time, which is the collapsed-column signature disappearing.

Found after this was measured: one of its own columns is mis-scaled #

Some unknown share of the +17.5 points is an artefact rather than honest scoring, and this was found after the run.

value_destroyed divides by the target's value at stake, which wire::Location::value builds from location contents - guns, actuators, 8 for an engine slot, 8 for a gyro, 20 for a cockpit. Those flags come from Observation.contents reading Mek.SYSTEM_* and Mek.ACTUATOR_* out of a location's critical slots, and a combat vehicle has no critical slots at all: probed, systemSlots=0 against an Atlas's 31, because MegaMek models a vehicle's criticals as unit-level results rather than as contents of a location.

So a vehicle's denominator is guns and ammunition alone. Measured over 10,533 real firing candidates, value_destroyed reads a mean of 0.0797 against a vehicle and 0.0048 against a Mek - about seventeen times - and reaches 0.99 where a Mek tops out at 0.25.

What that does and does not do to the result above. It does not make the shift wrong: a vehicle genuinely read 0.000 on three columns before this branch and does not now, and the direction of the change is not in question. What it means is that "the basis now prefers vehicles for defensible reasons" is not established - one of the reasons is still about our reader rather than about the machine, and this run cannot say how much of 17.5 points it accounts for. The fix is a scale decision rather than a translation and is written up in plan/unit-types.md; until it lands, value_destroyed should not carry a fitted weight over a mixed suite.

What this measurement does not control #

  • The denominators are not matched pairs. 167 mixed menus before and 138 after, because a different bot moves differently and therefore sees different targets. The share is conditional on the menu that arose, not on a fixed set of decisions, and no pairing is claimed.
  • It is one suite. Twelve scenarios across three eras, 21 Meks to 15 vehicles.
  • It says the preference moved, not that it is now correct. Whether the bot should favour a vehicle at 0.36 rather than 0.18 is a tactical judgement this branch deliberately does not make. What it removes is the reason to distrust the old number: 0.1796 was produced with three of the target's scoring columns reading zero for a modelling reason, and 0.3551 is produced with them measured.

The base moved after the arms ran, and this is what that means #

Both arms ran with origin/main at 837dd20. While they were running, the battle-armor line landed - 63 commits - and this branch is now rebased onto 46e44b0. So the measurement compares 837dd20 against 837dd20 + this branch, not against the tree the branch now sits on.

It is still the right comparison for attributing the shift, because it isolates this change against the base it was written on: the +17.5 points are this branch's doing and nothing else's. What it cannot say is what the number would be on today's main, where physical attacks are now virtual weapons in the firing menu and target selection has another input.

Two things were checked rather than assumed before letting the figure stand:

  • features::firing::ceiling did not change. It was the known hazard - a saturating denominator that would move every expected_damage in the project. git diff of the function body between 837dd20 and 46e44b0 is empty; the battle-armor line pinned the seam with a test rather than fixing it. So the denominator under both arms is the same one.
  • hitloc.rs was untouched on main, so the location model these arms exercise is the one this branch ships.

A re-measurement on the new base is worth doing and is not done here: it needs 24 more matches an arm, and the machine was under memory pressure with another agent's benchmark running. Sequencing that is not this branch's call.

Arm validity #

  • Arm A ran at 3d1d541. Everything committed since is test code or doc comments - git diff 3d1d541..HEAD -- crates/ touches only the new volley.rs test fixture and doc lines - so the arm measures this branch's behaviour.
  • Arm B ran the origin/main binary, copied into the repository so the match container could see it. The first attempt at this arm passed a host path outside the container mount: the seat never started, 162 of 162 decisions defaulted with zero candidates, and the run completed and printed a win rate anyway. It was discarded, sds bench now refuses such a path, and the replacement arm was smoke-tested on one game - 32 of 32 answered, 9,617 candidates - before 24 were committed to it.

sds spread, the epic's own instrument #

The exit criterion is that no feature reads a constant for one unit type and a range for another. In the firing phase after this change, over 319,547 candidate readings: p_breach mean 0.310 with 9% at zero and 9% flat, value_destroyed 12% at zero, p_mission_kill mean 0.077 with 35% flat. All three vary across candidates in decisions where previously they could not vary at all for a vehicle.