ABQ 1.5: Replay test runs locally

We've just released ABQ 1.5, which adds support for replaying test runs from CI locally. It also adds support for new test batching options, which can improve ABQ's performance on some test suites.

Local test replay

Because ABQ utilizes a queue for parallelization, it can be hard to reproduce CI failures locally when those failures are dependent on the exact order that tests are executed in. ABQ 1.5 introduces local test replay for managed queues, which allows you to exactly replay the order in which a CI node ran tests. Local test replay dramatically improves the debugging experience when resolving CI failures!

In support of this feature, we added support for personal access tokens so that you can communicate with your ABQ queue from your local machine. You can generate and save a personal access token using the new abq login command.

Once you're using your personal access token, you can replay any test run from a CI node by copying the command output at the end of that node's test run. Look for the following output in your CI logs:

Run the following command to replay these tests locally:


	abq test --run-id <run-id> --worker 0 --num 1 -- <your-test-command>

Test distribution strategy

ABQ 1.5 also introduces a new by-file test distribution strategy. By default, ABQ distributes individual test cases, so multiple test cases from the same file can run on different nodes. When --test-strategy by-file is provided to ABQ, however, ABQ will always ensure every test case in a file is distributed to the same node. This strategy is useful for test suites that have files with very slow hooks that run before all the test cases in a file, such as RSpec integration test suites with before(:all) hooks that perform expensive setup.

In the future, we hope to automatically enable by-file for test suites that would benefit from it. We also plan to add support for using the by-file strategy only for files that need it instead of applying it across your entire test suite.