Retrying tests with ABQ

ABQ understands that sometimes you need to re-run certain tests, or workers, in order to resolve flakes and unblock your team. ABQ supports automated and manual retries when you need them.

Automated test retries

Automated retries of tests can be a great way to unblock your team in the presence of flaky tests.

ABQ supports configuring automated retries for any supported test framework. You can specify the maximum number of times a test should be retried with the --retries flag in the ABQ CLI. A test will be retried until its first success, or all its retries have been exhausted.

If the --retries flag is not specified, ABQ will not retry tests. ABQ does not currently support configuring retry counts on a per-test basis.

An example with Ruby and RSpec

Consider the following flaky Ruby RSpec test

RSpec.describe do
  it "is flaky" do
    expect(rand(2)).to eq(1)
  end
end

To run this test through ABQ with two retries, make sure you've installed ABQ's RSpec integration and pass --retries 2 when invoking ABQ.

abq test --retries 2 -- bundle exec rspec flaky.rb

This test run might look like

FF.

--- is flaky: FAILED ---

  1) is flaky
     Failure/Error: expect(rand(2)).to eq(1)

       expected: 1
            got: 0

       (compared using ==)
     # ./flaky.rb:6:in `block (2 levels) in '

(completed in 5 ms [worker 0])

--- is flaky: FAILED (attempt 2) ---

  1) is flaky
     Failure/Error: expect(rand(2)).to eq(1)

       expected: 1
            got: 0

       (compared using ==)
     # ./flaky.rb:6:in `block (2 levels) in '

(completed in 5 ms [worker 0])


Finished in 0.69 seconds (0.00 seconds spent in test code)
1 tests, 0 failures, 1 retried

Indicating that after 2 failures, this flaky test ultimately passed.

Manual test retries

Retrying your CI test jobs may be necessary in the presence of new flakes, or sporadic startup failures. ABQ supports retrying individual workers for a given run-id without re-running your whole test suite.

Retrying an individual worker will only retry the tests allocated to that worker in the first attempt. The test results produced in the worker retry will be recorded and reported as retries during test result aggregation.

For example, consider an RSpec/Ruby test suite run through ABQ with two workers:

# on worker machine 0
ABQ_RUN_ID=my-rspec-test abq test --worker 0 -- bundle exec rspec

# on worker machine 1
ABQ_RUN_ID=my-rspec-test abq test --worker 1 -- bundle exec rspec

If only worker 1 fails, re-running the same command

# retry on worker machine 1
ABQ_RUN_ID=my-rspec-test abq test --worker 1 -- bundle exec rspec

will run only the tests originally assigned to worker 1, and subsequent calls to abq report will aggregate test results executed by the only execution of worker 0, and both executions of worker 1.