One script runs every test before a release

The question that started this was asked out loud, a little annoyed: we have done a crap ton of testing in the past. Is there any way of making sure we perform this test every time we upgrade?

There wasn’t, honestly. The tests existed. Months of hardening had produced waves of them. But the knowledge of which checks had blessed which build lived in chat scrollback and habit, and that’s how old bugs get back in. Someone wrote the test; nobody re-ran it.

Now there’s a script, and a rule. The script chains every automated check the project trusts and ends with a verdict line. It builds the release binaries. Runs the unit suite, a thousand tests. Runs the engine-parity batteries in stock PowerShell, the same ones that get pasted into the app for comparison. Verifies the module contract Quick Fix depends on hasn’t drifted. Runs the test-generation gate. Confirms every function still carries its numbered design blueprint, with a ratchet so the count of documented legacy gaps can fall but never grow. Scans dependencies for known CVEs. Scans the source and the full git history for leaked secrets. Last line: RELEASE GATE: PASS, or not. The rule is that nothing gets republished or packaged without it.

Two policies keep the gate from rotting. Flaky tests are handled in the open: a test can be listed as a known flake only after passing three consecutive isolated runs, it gets retried alone when it fails in the full suite, and an isolated failure counts as real. The flake list sits in the script with dates on each entry, visible debt instead of folklore. And new test infrastructure joins the gate the day it exists. That rule is written in blood, mildly: our end-to-end UI rig once sat outside the gate for ten days, built and working and unrun, while a bug it would have caught shipped past a “gate is green” claim. The gap wasn’t a missing test. It was a missing line in the script.

The gate ends by printing what a script can’t do: paste the parity battery into the app and compare tallies, run the guided GUI checklist, sweep the themes with human eyes, check color and progress rendering by looking at them. Printing the manual list sounds trivial. In practice it’s the difference between “someone should do these” and the terminal currently telling you to.

Runtime is a few minutes, which is the design requirement hiding under everything else. A gate that costs an afternoon gets skipped the week it matters most. This one is cheap enough to run on a hunch, so it runs.

← All posts