OSS Guide

The Captain CLI can be used in OSS mode without a subscription to Captain Cloud. To enable this behavior, set cloud.disabled in your .captain/config.yml file.

Managing Flaky Tests

You can mark tests as flaky to provide more fine-grained control over which tests the Captain CLI will retry. Specifically, if you track your flaky tests, you can configure test-suites.*.retries.flaky-attempts to target your retries specifically to tests which you've deemed to be flaky.

Marking a test as flaky requires you to use captain add flake along with the identity components relevant to your test framework (see the "Identifying Tests" section). For example, RSpec works best if you pass the file and description of the test:

captain add flake your-project-rspec \
  --file ./test/controller_spec.rb \
  --description "SomeClass#some_method performs some action"

Once the flaky test is added, be sure to commit the changes to ./captain/*/flakes.yaml.

To remove a flaky test, you can either remove it manually by editing the file or you can use captain remove flake:

captain remove flake your-project-rspec \
  --file ./test/controller_spec.rb \
  --description "SomeClass#some_method performs some action"

Once the flaky test is removed, be sure to commit the changes to ./captain/*/flakes.yaml.

Managing Quarantined Tests

If you run your tests through captain run or captain quarantine, marking a test as quarantined will tell the Captain CLI that it should ignore any failed or timed out test results and prevent them from failing the test suite. Instead of failing the test suite, the Captain CLI will indicate in its output which tests were quarantined.

Marking a test as quarantined requires you to use captain add quarantine along with the identity components relevant to your test framework (see the "Identifying Tests" section). For example, RSpec works best if you pass the file and description of the test:

captain add quarantine your-project-rspec \
  --file ./test/controller_spec.rb \
  --description "SomeClass#some_method performs some action"

Once the quarantined test is added, be sure to commit the changes to ./captain/*/quarantines.yaml.

To remove a quarantined test, you can either remove it manually by editing the file or you can use captain remove quarantine:

captain remove quarantine your-project-rspec \
  --file ./test/controller_spec.rb \
  --description "SomeClass#some_method performs some action"

Once the quarantined test is removed, be sure to commit the changes to ./captain/*/quarantines.yaml.

Managing Timings

Partitioning your test suite with captain run works best with up-to-date timing information on the test files in your test suite. We recommend keeping these up-to-date by running captain update results periodically. The command takes one or more test results artifacts and records how long each test file in those artifacts took. For example:

captain update results your-project-rspec ./tmp/rspec.json

Once the timing file has been updated, be sure to commit the changes to ./captain/*/timings.yaml.

Depending on the characteristics of your test suite, you can either run your entire suite serially (to produce a single test results artifact) or you can run your test suite in parallel, upload the artifacts to some intermediate storage (e.g. using actions/upload-artifact on GitHub Actions), and then, once all of the parallel steps have finished, download all of those artifacts (e.g. using actions/download-artifact on GitHub Actions) and pass them to captain update results.