What causes an Unreal automation run not returning a failure exit code in CI?

The editor process exits with code zero even when tests fail, so CI sees a green job, and the run does not request the test report or fail the editor on failures.

How do I catch this when it only happens for some players?

Capture it automatically from the player's device with the full stack trace, the device and OS, the build, and a breadcrumb trail of what they did. That turns a vague complaint into a specific, reproducible issue you can fix without owning the hardware it happens on.

How to Fix Unreal Automation Runs Not Returning a CI Exit Code

Quick answer: Run automation with the flags that fail the editor on test failure and write a report, then parse the report so CI fails when any test failed.

UE's editor often exits zero even with failing tests. You must request the report and either use the test-failure exit flag or parse the JSON to fail the job.

How to fix it

1. Request a report and fail on failure

Run with -ExecCmds="Automation RunTests <filter>; Quit" plus -ReportExportPath, and use -testexit / the failure flag so the editor returns nonzero when tests fail.

2. Parse the report in CI

Read the exported index.json report and fail the pipeline if any test state is not Success, which is reliable even when the editor exit code is unhelpful.

3. Run unattended

Add -unattended -nopause -nullrhi so no dialog blocks the run on a headless agent and it does not need a GPU, then capture the log for diagnosis.

Catching the ones you can't reproduce

The hardest version of this to fix is the one you can't reproduce — it only happens on a player's hardware, OS, driver, or save state, under conditions that simply aren't present on your machine. A report that says “it crashed” or “it froze” gives you nothing to act on, so the bug survives release after release while quietly costing you players.

Automatic error capture closes that gap. Each failure arrives with its full stack trace, the device and OS, the build number, and a breadcrumb trail of what the player did right before it broke, so even a failure you have never seen becomes a specific, reproducible issue. Fold identical failures into one signature ranked by how many players each hits, and your worklist sorts itself worst-first instead of arriving as a stream of vague complaints.

This is where a tool like Bugnet earns its place. Its SDK captures every Unreal Engine error automatically with the full stack trace plus device, OS, memory, build, and game-state context, folds duplicates into one grouped issue with an occurrence count, and ties each to the build it first appeared on — so you fix the problem that hurts the most players first and confirm it is gone when its signature disappears from the next release.

Most of the time the fix is small. Seeing the failure clearly is the part that actually costs you.