I started building Steel Onslaught because I wanted a concrete way to explain the architecture I work on. It is a browser-based mech game where LLM pilots decide how to move, what to shoot, and when to use utility cards. AI helped me get to a playable game quickly, but once I started trying to balance it, the speed advantage mostly disappeared. One mech kept getting demolished, and I spent a while doing what I suspect a lot of us do: changing the prompt, watching a few matches, and coming up with an explanation that seemed to fit what I had just seen.



The problem was that several of those explanations were wrong. For example, we made the objective instructions more prominent without changing the arena, rewards, loadouts, or pilot personas. In the first 30-versus-30 test, the default version produced no objective captures and a 15-15 win split. The prominent version produced 34 captures and a 28-2 split. At first that looked like an obvious success, but most of those captures came from four unusually long matches. We repeated the test with fresh matched seeds, interleaved conditions, and a different inference server. Across 58 complete pairs, prominent-only objective awards beat default-only awards 9-1, and red's win rate moved from 20.7% to 43.1%. That was a real improvement, but it still did not create parity, and it did not prove the simple explanation we started with.



We tried increasing the objective reward to four points because I expected a stronger incentive to make the pilots pursue it more consistently. It did not improve the result, so we left the change out. That negative result was probably more useful than another successful demo because it forced us to separate what we had actually measured from the story we wanted to tell about it.



The part of the project I value most now is not that AI helped produce a game. It is that the architecture gives us a way to be wrong without losing the evidence. Every match writes an event ledger and can be replayed. Experiments use matched seeds, failed runs remain in the record, and we write down predictions and stopping rules before running a battery. The attached video is one illustrative replay, not proof of the aggregate result. It shows the system at work, but the conclusions come from the repeated experiments.



If you are building AI-driven game behavior, I would be interested in hearing how you moved from watching a few convincing runs to deciding that a behavior change was actually repeatable.

Building the game was fast. Figuring out why the AI behaved the way it did was not.