attacker_self_damage: two rounds, and what they establish #
Both rounds ran three arms of 24 paired matches on scenarios/berserker-lance,
self-play, --explore 0, identical (scenario, seed) pairs, --jobs 2. Every
arm within a round shares one binary and one jar, checksum-verified before and
after (ARTIFACTS_STILL_FROZEN both times). Round 1 ran on the pre-#320 base;
round 2 on 3926149. The two rounds are not a series and their
answered/decisions figures are not comparable - #320 turns a refused move
into a retry, so a decision that used to default may now succeed.
Round 1: the column was inert by construction #
attacker_self_damage was expected self-damage in points over the machine's total
remaining health. It did nothing, and the reason is not behavioural:
Across 82 physical-phase candidates it read at most 0.0094, contributing at
most -0.038 to a score whose other terms run 1 to 4. A healthy Mek missing a
kick expects about 0.07 points of self-damage; two hundred points of armour and
structure underneath makes that vanish. The denominator was one no other column
uses - expected_damage divides by the output ceiling, the p_ columns are
probabilities - and the hand-authored weights are all sized against those.
That finding is a reading of recorded feature values, not a comparison, so it survives every rebase since. The arms it came with do not, and are not carried forward.
Prediction 1's pass was false #
Registered: long-odds kicks fall below 9%. Treatment: 8.9%. Floor arm, with no feature at all: 6.8%. The threshold was cleared by a control, and its baseline had been pooled across two arms that differ by the variable under test.
Round 2: a working column, an instrument fix, and an underpowered design #
The column is now P(miss) x P(fall | psr_base) - the chance the swing puts us
down. It spans 0.005 to 0.535 and discriminates about fifteenfold between a
healthy machine and a damaged one, so it can move a decision where the old one
could not.
The metric changed too, and had to. psr_base was only observable when MegaMek
printed it, which is when a kick missed - about forty samples an arm, and
conditioned on the outcome of the very roll under test. It is now written onto
every decision, giving 284 declared kicks an arm instead of ~40, and
measuring which kicks the bot chose rather than which ones missed.
The three predictions #
| # | registered | result |
|---|---|---|
| 1 | long-odds kicks fall below 9% | cannot attribute |
| 2 | total kicks drop under 20% | held, uninformative |
| 3 | risky-kick share falls a third, beating the floor | failed |
1 - cannot attribute. Treatment 6.1% against a control of 11.7%, which reads as a large effect until the floor arm is looked at: 6.8%, on the same config as the control. Signal -5.6 points, floor -4.9 points. The same shape as round 1's false pass, caught the same way.
2 - held but uninformative. Kicks fell 10.0% (290 to 261), inside the registered 20%. But the floor arm fell 18.6% (290 to 236) with nothing changed, so "the drop was small" is a statement about noise.
3 - failed, and this is the honest call. On the new per-kick metric the share
of kicks taken by machines at psr_base 6 or worse went 21.8% to 17.0%, a
relative drop of 21.9% against a registered floor of a third. The point
estimate beats the single floor observation (-4.8 points against +2.9), and the
direction is the one the feature predicts. But a paired bootstrap over matches
puts the signal at -4.79 points, 95% CI [-11.09, +2.35] - including zero,
with the floor at +2.86 [-5.51, +12.26].
Both registered conditions failed: the drop was smaller than a third, and it is not separable from zero at this sample size.
What this establishes, and what it does not #
The instrument fix worked and is the reusable part. The old metric's floor was 23 points against a 13-point threshold - unable to answer its own question. The new one has a point-estimate floor of 2.9 points on seven times the sample. What remains is match count, not instrumentation.
The feature is not shown to work. The direction is right on the column built for it, and the effect clears the one floor reading available, and neither of those is a result while the interval spans zero. Roughly 48 to 72 matches an arm would be needed for an effect of this size, against the 24 run here.
It is also not shown to be a flat penalty. That was the failure mode prediction 3 existed to detect: a term that suppresses kicking generally rather than risky kicking. The evidence is against it - the risky share fell while the total fell less - but the same interval that fails to establish the effect fails to establish its shape.
And it does not answer jmm's question about a family of features. Round 1 could not, because the column never engaged. Round 2 cannot, because a term that does not separate from zero says nothing about whether a charge's self-damage belongs in the same column as a kick's. That question is where it was.
Base #
Round 2 describes 3926149. Twenty commits landed during it, none touching the
physical path - checked, not assumed: zero mentions of physical, kick or PSR
across 6f3806d, 3ef77b0 and d8ae312, and the one commit touching
sds-bot/src/unit.rs is memo plumbing. But eleven of them touch stands.rs and
change the movement sweep, which decides how often units reach contact and in
what condition - so the population of kickers is base-specific even though the
physical menu is not. All three arms share the base, so the comparison holds
inside itself and the numbers are not portable.
A floor arm measures how much a metric moves with the config identical. It cannot measure how much it moves when the base changes underneath. Nothing here answers that, for either round.