sds
bayes for days
Star
Fork
Watch
sds prediction-attacker-self-damage.md
5.5 kB · 97 lines
Markdown
at main

Prediction for attacker_self_damage, written before any arm was run #

Recorded 2026-08-30, before a single match under the new basis.

Design #

Same scenarios/berserker-lance suite, self-play, --explore 0, paired on identical (scenario, seed) pairs, three arms of 24 as before. Both arms run the same binary and the same basis; they differ only in a weights file, with attacker_self_damage at -4.0 in one and 0.0 in the other. That isolates the weight rather than the feature, so nothing about the comparison depends on two builds.

The third arm repeats the control for the floor, because a seed does not reproduce a match and self-play was measured at only 16 of 24 identical.

What I expect, with numbers #

The bot currently declares a kick whenever MegaMek offers one - 17 of 17 in the logs - because hold fire scores nought and any swing scored above it. The feature exists to let the bad ones lose.

  1. Kicks at long odds roughly halve. Kicks needing 10 or more were 19%, 11% and 21% of all kicks across the three Advance arms - about 17% pooled. I predict under 9% with the weight on. This is the number the feature is for and the one I would call it a failure on.
  2. Total kicks fall by much less - I predict a drop of under 20%. Most kicks are at short odds against healthy machines, where the term is small and should not stop anything.
  3. The drop concentrates on damaged machines. A kick from a unit at psr_base 6 or worse falls 37-75% of the time; at 2-5 it falls 5%. I predict the decline in kicks is at least twice as large among units at 6+ as among units at 2-5. If the term suppresses both equally it is behaving as a flat penalty on kicking and not as a risk measure, which is the failure mode I would most expect and the hardest to see in a headline.
  4. Falls from missed kicks fall. 14 falls over 68 missed-kick PSRs in 72 matches. I predict fewer falls per match, and I do not predict the fall rate per missed kick moves - that is MegaMek's dice and the machine's state, not something the bot can change by declining.
  5. Melee rate drops slightly, and that is a cost, not a defect. The whole point is refusing some attacks. If melee rate is unchanged the feature is inert.

What would falsify it #

Any of these differences being inside the floor the repeat arm measures, the same way Advance's +3.79 was checked against a floor of +0.17. Prediction 3 is the one that distinguishes "the bot learned to judge a kick" from "the bot kicks less"; if 1 and 2 hold but 3 does not, I will report it as a flat penalty rather than claim the risk model works.

What I am not predicting #

Anything about win rate. Declining a bad kick should help and I have no basis for a number, and plan/harness.md puts separating a 5-point win-rate difference at about 385 decided games.

How prediction 3 will be measured, decided before any data #

psr_base is on the wire but not in the decision log, and the only place it appears in a run's output is MegaMek's own line for a kick that missed - the roll is printed because the roll happened. There is no per-unit piloting column in game_actions_0.tsv either. So the population I can attribute a piloting base to is missed kicks, not all kicks.

Prediction 3 is therefore measured as: among missed kicks, the share whose PSR base is 6 or worse, compared between arms. Across the three Advance arms that was 26 of 68, about 38%. If the term is doing risk modelling, the on arm suppresses the risky kicks and this share falls; if it is a flat penalty, the share holds while the total drops.

Two limitations, stated now rather than when the numbers are inconvenient:

  • It is a proxy for the attempted-kick distribution, not that distribution. A missed kick is a kick, but whether a kick misses depends on to-hit, and damage worsens to-hit and piloting base together - so the missed-kick sample is mildly enriched for damaged machines in both arms. It is the same bias in both, which is why a between-arm comparison survives it and an absolute reading of "how often does the bot kick while damaged" would not.
  • Falls and PSR counts are small. 68 missed-kick rolls over 72 matches, so 24 matches an arm gives something like 20. A share of 20 has a standard error near 11 points, which is wide enough that only a large move in prediction 3 will clear the floor. If it does not clear it, the honest report is that this design cannot separate the two hypotheses at this sample size - not that the term is a flat penalty.

One refinement, made on the first match of the first arm and before any comparison. A piloting roll often cites more than one reason - was kicked; missed a kick is common, because the machine that kicked was usually kicked back. Counting only rolls whose sole reason is a missed kick threw away most of the sample: the first match had three missed kicks and nought such rolls.

So the two questions get two filters. What base was a kicker at counts any roll citing a missed kick, because the base printed is the machine's own and the other reasons do not change it - that is prediction 3's population. Did the kick cause this fall keeps the sole-reason filter, because a machine that was also kicked could have been put down by either, and a combined roll cannot say which. The fall counts are therefore an undercount, deliberately.