ABQ 1.3 Generally Available, Open Source Release

Last year, we began building a universal test runner that executes test suites as distributed workloads. It’s universal in that it’s one CLI that can be used with many existing testing frameworks. After a thorough beta testing period over the past several months, we’re excited to announce the generally available, open source release of ABQ. It’s the best tool for splitting test suites into parallel jobs in CI.

Reasons for Queue-Based Parallelization

As soon as a test suite takes more than a few minutes to run, engineers begin looking into splitting it into multiple CI jobs.

The easiest way to achieve parallel execution is to manually partition test files into separate groups to be executed independently. Often, it becomes difficult to create evenly balanced partitions using this approach.

A better option is to automate grouping the test files based on historical timings. Many frameworks have a third party library available to record timings, but it requires some curation to generate timing files and check them in. In some cases, with high variance in test runtime or CI job startup time, partitions can still become unbalanced.

Both of these approaches can be thought of as pre-partitioning the files before execution. Unfortunately, pre-partitioning can result in unbalanced jobs due to variance in test runtime or worker startup time. They also can require effort to maintain, such as updating timing files.

The best way to avoid these issues is to take the approach that engineers would take to distribute any other workload: using a message queue. This approach ensures that all workers run as evenly as possible, resulting in the fastest possible feedback loop. Variance in performance doesn’t impact the overall runtime of a queue-based approach like it does with a pre-partitioned approach.

A Universal Test Runner

We designed ABQ to bring message queue based test distribution to many languages and testing frameworks. While ABQ’s distribution mechanism is complex, it only takes a little bit of work to build native test bindings which add the capabilities we need for ABQ to control the native test frameworks.

One aspect we love about this approach is that ABQ wraps the native framework rather than replacing it. This means that you can continue using existing test commands and CLIs. For example, you can wrap the RSpec CLI (abq test -- bundle exec rspec), npm tasks (abq test -- npm run test:ci), or npx commands (abq test — npx playwright test). ABQ is agnostic to the command it’s running, only requiring the native bindings to be available.

Native bindings are currently available for RSpec, Jest, Playwright, and pytest.

Feature Highlights

ABQ distributes test suites using a highly efficient protocol, and it provides a universal set of capabilities often needed when managing test suites as part of a CI process. Some highlights include:

• Automated retries of failed tests with the --retries flag, ensuring that failures are legitimate and not due to flakiness
• abq report which will generate an aggregate list of test failures, enabling engineers to look in one place for the list of failures instead of needing to click into each individual job in CI
• Network overhead happens asynchronously, ensuring that the queue protocol adds minimal overhead to test runtimes
• Support for running multiple processes on a single machine with the -n flag
• Persisted dispatch history, ensuring that retried jobs work reliably on CI
• Universal test result reporters, like JUnit XML and RWX v1 JSON
• Built as a standalone binary with all dependencies statically linked for easy installation
• A high degree of compatibility with the underlying test framework, enabling features like using the native test frameworks reporters

For a look under the hood, see the ABQ native runner protocol 0.2 which describes how native test processes communicate with ABQ.

Getting Started

ABQ can be added to an existing test suite and CI build in minutes. To run in CI, you can take the self-hosted approach of using the abq binary to start a hosted queue, or you can use managed queues run by RWX, which you can set up at rwx.com/abq.

For more on ABQ, see the documentation.

We’re happy to help with any integrations. Say hello on Discord or reach out at [email protected]

Credits

Much thanks to Ayaz Hafiz for leading the development of ABQ.