ABQ with GitHub Actions

Access Token

You'll need an access token set in an environment variable named RWX_ACCESS_TOKEN.

For these examples, we assume this is also set as a repository or organization secret named RWX_ACCESS_TOKEN.

Installing the CLI

Use the setup-abq action to install the ABQ CLI.

- uses: rwx-research/setup-abq@v1
  with:
    access-token: ${{ secrets.RWX_ACCESS_TOKEN }}

The abq test command will pick up tests from a queue created for your test run, and execute them. You'll need to configure every abq test job with all of the setup necessary to run your tests. For example, if you install Ruby and depend on other Docker services, you'll set those up before running abq test.

An ABQ test run can consist of one or more abq test workers. Each worker will locally report the results of tests it's been assigned as it runs them. You can also configure ABQ to aggregate all of your workers' test results at the end of a test run.

Here's a full example of configuring an ABQ test run, using GitHub's matrix configuration to determine how much the test suite is parallelized. In this example, the matrix parameter shard is used to determine the --worker parameter of abq test in each job. After all workers in the abq-test job run, abq-report runs a single step that aggregates and reports the results of all tests your workers ran.

Be sure to replace YOUR_TEST_COMMAND with the command you normally use to run your tests in CI, including any arguments.

abq-test:
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      shard: [0, 1]
  # Optionally start service containers needed to run your project's tests.
  services:
    db:
      image: postgres
  steps:
    - uses: actions/checkout@v3
    - uses: rwx-research/setup-abq@v1
      with:
        access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
    # Install dependencies necessary to run your project's tests
    - uses: ruby/setup-ruby@v1
      with:
        bundler-cache: true
    - run: abq test --worker ${{ matrix.shard }} -- YOUR_TEST_COMMAND
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}

# Optionally, report the aggregated test results.
abq-report:
  runs-on: ubuntu-latest
  # Run the aggregation step if your test suite succeeded or failed, but not if
  # it was cancelled.
  needs: [abq-test]
  if: ${{ success() || failure() }}
  steps:
    - uses: rwx-research/setup-abq@v1
      with:
        access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
    - run: abq report --reporter rwx-v1-json=suite-results.json
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}

You can pass any command as YOUR_TEST_COMMAND, as long as it appears after a double hyphen. For example, if you're using RSpec and have other flags that you're using, you could call:

abq test -- bundle exec rspec --tag integration

abq test for Rails

Here's an example of what a full ABQ test job may look like for a Rails project.

abq-test:
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      shard: [0, 1]
  services:
    postgres:
      image: postgres:14
      ports:
        - '5432:5432'
      env:
        POSTGRES_USER: postgres
        POSTGRES_PASSWORD: password
  env:
    RAILS_ENV: test
  steps:
    - uses: actions/checkout@v3
    - uses: rwx-research/setup-abq@v1
      with:
        access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
    - uses: ruby/setup-ruby@v1
      with:
        bundler-cache: true
    - run: bundle exec rails db:prepare
    - run: bundle exec rake assets:precompile
    - run: abq test --worker ${{ matrix.shard }} -- bundle exec rspec
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}

# Optionally, report the aggregated test results.
abq-report:
  runs-on: ubuntu-latest
  needs: [abq-test]
  if: ${{ success() || failure() }}
  steps:
    - uses: rwx-research/setup-abq@v1
      with:
        access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
    - run: abq report --reporter rwx-v1-json=suite-results.json
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}

Run ID

Different abq test processes for a test run are connected to each other through a run-id. On GitHub Actions, ABQ automatically sets run-id to ${{ github.run_id }}. The default value is sufficient as long as you're only running a single test suite through ABQ in your GitHub Actions workflow.

If you're running multiple test suites through ABQ in a single GitHub Actions workflow, then you'll need to configure the run-id so that each test suite has a separate run-id with its own abq test processes.

We recommend suffixing the default value with a test suite name, such as ${{ github.run_id }}-rspec.

If your configured run-id is consistent between GitHub Actions job attempts, retrying a single worker job will work seamlessly, re-running just the tests allocated to that worker.

For the ABQ CLI, you can either pass the run id as a --run-id argument, or you can set it in an environment variable named ABQ_RUN_ID.

Here's an example of a GitHub Actions workflow that runs multiple test suites through ABQ.

name: 'CI'
on:
  - push
jobs:
  abq-test-rspec:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1]
    env:
      ABQ_RUN_ID: ${{ github.run_id }}-rspec
    steps:
      - uses: actions/checkout@v3
      - uses: rwx-research/setup-abq@v1
        with:
          access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
          run-id: ${{ env.ABQ_RUN_ID }}
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - run: abq test --worker ${{ matrix.shard }} -- bundle exec rspec
        env:
          RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}
          ABQ_RUN_ID: ${{ env.ABQ_RUN_ID }}

  abq-test-jest:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1]
    env:
      ABQ_RUN_ID: ${{ github.run_id }}-jest
    steps:
      - uses: actions/checkout@v3
      - uses: rwx-research/setup-abq@v1
        with:
          access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
          run-id: ${{ env.ABQ_RUN_ID }}
      - uses: actions/setup-node@v3
        with:
          node-version: 16
          cache: 'npm'
      - run: npm install
      - run: abq test --worker ${{ matrix.shard }} -- npm run test:ci
        env:
          RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}
          ABQ_RUN_ID: ${{ env.ABQ_RUN_ID }}

# Optionally, report the aggregated test results.
abq-report:
  runs-on: ubuntu-latest
  needs: [abq-test-rspec, abq-test-jest]
  if: ${{ success() || failure() }}
  steps:
    - uses: rwx-research/setup-abq@v1
      with:
        access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
    - id: Report RSpec results
      run: abq report --reporter rwx-v1-json=suite-results.json
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}
        ABQ_RUN_ID: ${{ github.run-id }}-rspec
    - id: Report Jest results
      run: abq report --reporter rwx-v1-json=suite-results.json
      env:
        RWX_ACCESS_TOKEN: ${{ secrets.RWX_ACCESS_TOKEN }}
        ABQ_RUN_ID: ${{ github.run-id }}-jest