How to Fix a Flaky Frame-Budget Performance Regression Test

A perf test that fails because one frame spiked on a busy runner is noise, not signal. Warmup plus percentile-based thresholds make the test catch real regressions only.

How to fix it

1. Warm up before measuring

Run the workload for a few hundred frames before recording so JIT, caching, and allocations settle. Cold-start numbers are not representative of steady-state cost.

2. Assert on percentiles, not max

Record many frame times and assert the 95th percentile stays under budget. A single GC spike should not fail the test, but a sustained regression will move the percentile.

3. Compare to a baseline with tolerance

Store a baseline and fail only when the new median exceeds it by a margin (for example 15%). Shared CI runners are too noisy for absolute millisecond gates.

Catching the ones you can't reproduce

The hardest version of this to fix is the one you can't reproduce — it only happens on a player's hardware, OS, driver, or save state, under conditions that simply aren't present on your machine. A report that says “it crashed” or “it froze” gives you nothing to act on, so the bug survives release after release while quietly costing you players.

Automatic error capture closes that gap. Each failure arrives with its full stack trace, the device and OS, the build number, and a breadcrumb trail of what the player did right before it broke, so even a failure you have never seen becomes a specific, reproducible issue. Fold identical failures into one signature ranked by how many players each hits, and your worklist sorts itself worst-first instead of arriving as a stream of vague complaints.

This is where a tool like Bugnet earns its place. Its SDK captures every error automatically with the full stack trace plus device, OS, memory, build, and game-state context, folds duplicates into one grouped issue with an occurrence count, and ties each to the build it first appeared on — so you fix the problem that hurts the most players first and confirm it is gone when its signature disappears from the next release.