Test Result Aggregation

As a test suite runs, ABQ workers will individually output and report the results of the tests they have been assigned.

It can be convenient to consolidate all of your test suite's results in one place for easier review. ABQ provides an abq report command to aggregate test results in a single reporting step, after your test run completes.

abq report takes a run id via --run-id or the ABQ_RUN_ID environment variable, and any number of reporters via the --reporter CLI flag. For example, to consolidate all results of the test run with run id my-run-id, and output RWX v1 JSON and JUnit XML reports, execute

export ABQ_RUN_ID=my-run-id
abq report --reporter rwx-v1-json=results.json --reporter junit-xml=results.xml

abq report aggregates and correctly reports retries, whether they be automatic or manual retries.

No worker for a given run id may be running when abq report is executed, including manual retries of a worker. If a worker is running when abq report is executed, abq report will exit with an error. We suggest configuring your CI so that abq report steps only run after all workers have explicitly completed.