Vehicles score honestly: what moved #
What this changes #
Three weighted columns read exactly 0.000 for any target the hit tables did
not describe, and a combat vehicle was one. Measured on the same volley from the
same shooter, against the fixture's Mek and against a tank built from it:
| expected damage | p_breach |
value_destroyed |
at_stake |
p_mission_kill |
|
|---|---|---|---|---|---|
| Mek | 12.500 | 0.100 | 0.0231 | 123.00 | 0.000 |
| tank, before | 12.500 | 0.000 | 0.000 | 0.00 | 0.000 |
| tank, after | 12.500 | 0.193 | 0.0000 | 65.00 | 0.387 |
Reproduced by
features::firing::tests::the_same_volley_scores_a_tank_and_a_mek_on_the_same_columns.
value_destroyed stays at 0.0000 for both targets on a volley this size, and
that is not the old failure wearing the same number: at_stake moved from 0.00
to 65.00, so the denominator is now a measurement rather than an absence, and a
volley that actually destroys a location scores against it. On a six-by-twenty
volley the same tank reads value_destroyed 21.14.
The tank's p_breach is now higher than the Mek's, which is the shape of
the model rather than a thumb on the scale: a vehicle's four arcs are four of
its locations, so the front table puts every roll but the turret's three onto
two places, and the same volley opens one sooner.
What I predicted, before looking #
Stated here before the run was read.
- The vehicle share of target selection rises. Three positively-weighted
columns went from
0.000to positive for vehicles and did not move for Meks, so a menu holding both should tip toward the vehicle. I expected a rise of at least 10 points. answereddoes not move. Nothing here changes which decisions the bot can answer, only what it scores them at. A drop would mean I had introduced a refusal.- The win rate against Princess is not measurable at this sample size. The harness prints its own power calculation: about 385 decided games to separate 5 points, 97 to separate 10. I could run 24 an arm. So any win-rate difference in this document is noise being reported honestly, not a result.
What actually moved #
Two arms of 24 matches each over scenarios/mixed, the same 12 scenarios, sds
against Princess. Before is origin/main's bot; after is this branch.
| before | after | |
|---|---|---|
| FIRING decisions | 548 | 534 |
| answered | 548 (100.0%) | 534 (100.0%) |
| both kinds on the menu | 167 | 138 |
| chose the vehicle | 30 | 49 |
| vehicle share | 0.1796 | 0.3551 |
Shift +17.5 points. Pooled SE 0.0504, z = +3.48.
Prediction against outcome #
- "The vehicle share rises by at least 10 points." Confirmed - +17.5, and
the direction is the one the mechanism predicts. Three positively-weighted
columns went from
0.000to positive for vehicles and did not move for Meks, so a menu holding both tips toward the vehicle. Nothing here was tuned to produce it: no weight was touched on this branch. - "
answereddoes not move." Confirmed - 100.0% in both arms. The change alters what decisions score at, not which the bot can answer. - "The win rate is not measurable at this sample size." Confirmed, and it is not claimed. The harness prints its own power calculation: about 385 decided games to separate 5 points, 97 for 10. Each arm is 24. No win-rate difference is reported in this document, and any that appears in the run summaries is noise.
Per scenario, because a pooled share can hide one scenario carrying it #
| scenario | before | after | |||
|---|---|---|---|---|---|
2v2-jihad-s5220000 |
3/16 | 0.188 | 5/10 | 0.500 | up |
2v2-succession-wars-s5020000 |
5/15 | 0.333 | 5/6 | 0.833 | up |
2v2-succession-wars-s5020001 |
0/18 | 0.000 | 4/20 | 0.200 | up |
4v4-clan-invasion-s5140000 |
6/22 | 0.273 | 11/16 | 0.688 | up |
4v4-clan-invasion-s5140001 |
5/13 | 0.385 | 5/5 | 1.000 | up |
4v4-jihad-s5240001 |
4/17 | 0.235 | 4/12 | 0.333 | up |
4v4-succession-wars-s5040000 |
3/27 | 0.111 | 9/31 | 0.290 | up |
4v4-succession-wars-s5040001 |
4/39 | 0.103 | 6/38 | 0.158 | up |
Eight of eight move the same way. A sign test on the direction alone gives p = 0.008, and it needs no assumption about the pooled variance. One scenario went from never choosing a vehicle (0 of 18) to choosing one a fifth of the time, which is the collapsed-column signature disappearing.
Found after this was measured: one of its own columns is mis-scaled #
Some unknown share of the +17.5 points is an artefact rather than honest scoring, and this was found after the run.
value_destroyed divides by the target's value at stake, which
wire::Location::value builds from location contents - guns, actuators, 8 for
an engine slot, 8 for a gyro, 20 for a cockpit. Those flags come from
Observation.contents reading Mek.SYSTEM_* and Mek.ACTUATOR_* out of a
location's critical slots, and a combat vehicle has no critical slots at
all: probed, systemSlots=0 against an Atlas's 31, because MegaMek models a
vehicle's criticals as unit-level results rather than as contents of a location.
So a vehicle's denominator is guns and ammunition alone. Measured over 10,533
real firing candidates, value_destroyed reads a mean of 0.0797 against a
vehicle and 0.0048 against a Mek - about seventeen times - and reaches 0.99
where a Mek tops out at 0.25.
What that does and does not do to the result above. It does not make the
shift wrong: a vehicle genuinely read 0.000 on three columns before this
branch and does not now, and the direction of the change is not in question.
What it means is that "the basis now prefers vehicles for defensible reasons"
is not established - one of the reasons is still about our reader rather than
about the machine, and this run cannot say how much of 17.5 points it accounts
for. The fix is a scale decision rather than a translation and is written up in
plan/unit-types.md; until it lands, value_destroyed should not carry a
fitted weight over a mixed suite.
What this measurement does not control #
- The denominators are not matched pairs. 167 mixed menus before and 138 after, because a different bot moves differently and therefore sees different targets. The share is conditional on the menu that arose, not on a fixed set of decisions, and no pairing is claimed.
- It is one suite. Twelve scenarios across three eras, 21 Meks to 15 vehicles.
- It says the preference moved, not that it is now correct. Whether the bot should favour a vehicle at 0.36 rather than 0.18 is a tactical judgement this branch deliberately does not make. What it removes is the reason to distrust the old number: 0.1796 was produced with three of the target's scoring columns reading zero for a modelling reason, and 0.3551 is produced with them measured.
The base moved after the arms ran, and this is what that means #
Both arms ran with origin/main at 837dd20. While they were running, the
battle-armor line landed - 63 commits - and this branch is now rebased onto
46e44b0. So the measurement compares 837dd20 against 837dd20 + this branch, not against the tree the branch now sits on.
It is still the right comparison for attributing the shift, because it
isolates this change against the base it was written on: the +17.5 points are
this branch's doing and nothing else's. What it cannot say is what the number
would be on today's main, where physical attacks are now virtual weapons in the
firing menu and target selection has another input.
Two things were checked rather than assumed before letting the figure stand:
features::firing::ceilingdid not change. It was the known hazard - a saturating denominator that would move everyexpected_damagein the project.git diffof the function body between837dd20and46e44b0is empty; the battle-armor line pinned the seam with a test rather than fixing it. So the denominator under both arms is the same one.hitloc.rswas untouched on main, so the location model these arms exercise is the one this branch ships.
A re-measurement on the new base is worth doing and is not done here: it needs 24 more matches an arm, and the machine was under memory pressure with another agent's benchmark running. Sequencing that is not this branch's call.
Arm validity #
- Arm A ran at
3d1d541. Everything committed since is test code or doc comments -git diff 3d1d541..HEAD -- crates/touches only the newvolley.rstest fixture and doc lines - so the arm measures this branch's behaviour. - Arm B ran the
origin/mainbinary, copied into the repository so the match container could see it. The first attempt at this arm passed a host path outside the container mount: the seat never started, 162 of 162 decisions defaulted with zero candidates, and the run completed and printed a win rate anyway. It was discarded,sds benchnow refuses such a path, and the replacement arm was smoke-tested on one game - 32 of 32 answered, 9,617 candidates - before 24 were committed to it.
sds spread, the epic's own instrument #
The exit criterion is that no feature reads a constant for one unit type and a
range for another. In the firing phase after this change, over 319,547 candidate
readings: p_breach mean 0.310 with 9% at zero and 9% flat, value_destroyed
12% at zero, p_mission_kill mean 0.077 with 35% flat. All three vary across
candidates in decisions where previously they could not vary at all for a
vehicle.