config.yaml

.captain/config.yaml is the Captain CLI's primary mode of configuration. We recommend keeping the majority of your configuration within the file to have a better experience when invoking the CLI.

Location

It's typical to place your configuration file at .captain/config.yaml at the root of your repository. In some situations, you may find that you want separate .captain directories in sub-directories of your repository (if, for example, you have a monorepo setup). Because of this, the Captain CLI will walk from the working directory (where the CLI was invoked) up until it finds a configuration file. This allows you to have a directory structure like the following:

repo/
├─ .captain/
│  ├─ config.yaml
├─ service_a/
│  ├─ .captain/
│  │  ├─ config.yaml
├─ service_b/

If you were to invoke the CLI from repo/service_a, repo/service_a/.captain/config.yaml would be used. If you were to invoke the CLI from repo/service_b or from repo, repo/.captain/config.yaml would be used.

Examples

Before diving in to the reference, below are select examples of CLI configurations. To find examples specific to your test framework of choice, take a look at the test framework documentation.

Basic Example

test-suites:
  your-suite:
    command: bundle exec rspec
    output:
      reporters:
        rwx-v1-json: ./tmp/rwx.json
    results:
      language: Ruby
      framework: RSpec
      path: ./tmp/rspec.json

Retry Example

test-suites:
  your-suite:
    command: bundle exec rspec
    output:
      reporters:
        rwx-v1-json: ./tmp/rwx.json
    results:
      language: Ruby
      framework: RSpec
      path: ./tmp/rspec.json
    retries:
      attempts: 1
      flaky-attempts: 2
      max-tests: 1%
      command: 'bundle exec rspec {{ tests }}'

Partition Example

test-suites:
  your-suite:
    command: bundle exec rspec
    output:
      reporters:
        rwx-v1-json: ./tmp/rwx.json
    results:
      language: Ruby
      framework: RSpec
      path: ./tmp/rspec.json
    partition:
      command: bundle exec rspec {{ testFiles }}
      globs:
        - spec/**/*_spec.rb

Options

`cloud`

cloud:
  disabled: true

cloud manages options around the integration with Captain Cloud. Specifically disabled can be used to explicitly disable communication with Captain's API.

Default: false

`flags`

flags:
  retries: 1 # for example

flags accepts a map of flag to value and allows you to set any flag supported by the CLI within the configuration file. Generally, we expect nearly all CLI options to be configurable within the config file. If you find that isn't the case, you can use this field as a workaround and open an issue if you think it should be supported as a first-class feature of the config file.

Default: {}

`output.debug`

output:
  debug: true

Setting output.debug to true enables debug-level logging. It's generally not recommended to enable this during normal usage of the CLI as it will be quite verbose.

Default: false

`test-suites`

test-suites:
  your-suite:
    # ...

The test-suites section contains a map of your test suite identifier to the options for that specific test suite. When running any Captain CLI command which accepts the --suite-id flag, it will use the options scoped to that suite in this file.

Test suite identifiers can contain any alphanumeric character, -, and _.

`test-suites.*.command`

test-suites:
  your-suite:
    command: bundle exec rspec

Setting test-suites.*.command configures the command that the Captain CLI will invoke to run your test suite. To find framework-specific examples, take a look at the test framework documentation.

Commands configured by this option:

captain run Required
captain quarantine Required

`test-suites.*.fail-on-upload-error`

test-suites:
  your-suite:
    fail-on-upload-error: true

Setting test-suites.*.fail-on-upload-error to true configures the Captain CLI to return a non-zero exit code when the test results upload to Captain Cloud fails.

This option only affects the Captain CLI when used in Cloud mode. We'd recommend trying Captain Cloud to improve your experience when using the Captain CLI.

Default: false

Commands configured by this option:

captain run

`test-suites.*.output`

test-suites:
  your-suite:
    output:
      # ...

The test-suites.*.output section contains options pertaining to the output that the Captain CLI can produce based on your test results. For example, a summary of the results or a JSON/XML artifact.

`test-suites.*.output.quiet`

output:
  quiet: true

Setting test-suites.*.output.quiet to true disables most default standard output. This includes any output emitted by your test suite. If your test suite is particularly verbose in its output, you may find yourself setting this option and test-suites.*.output.print-summary when retries are configured as your test suite's stdout will otherwise be printed each time a retry is attempted.

Default: false

Commands configured by this option:

`test-suites.*.output.print-summary`

test-suites:
  your-suite:
    output:
      print-summary: true

Setting test-suites.*.output.print-summary will configure the Captain CLI to produce a textual summary of your test results. This is useful when retrying your tests with the Captain CLI as your underlying command will be invoked multiple times and will not provide a representative summary of the outcome by itself.

Default: false

Commands configured by this option:

`test-suites.*.output.reporters`

test-suites:
  your-suite:
    output:
      reporters:
        rwx-v1-json: ./tmp/path/to/output.json
        junit-xml: ./tmp/path/to/output.xml
        markdown-summary: ./tmp/path/to/output.md
        github-step-summary:

Setting test-suites.*.output.reporters will produce a single test results artifact representing the outcome of the tests in your test suite. The option accepts a map of the reporter you are enabling to the path it should write its output.

The available reporters are:

rwx-v1-json: Produces an artifact adhering to the RWX v1 Test Results Schema.
junit-xml: Produces a JUnit XML-compatible artifact.
markdown-summary: Produces a summary of your results in Markdown.
github-step-summary: Writes the output of the markdown-summary to the $GITHUB_STEP_SUMMARY in GitHub Actions.

Default: {}

Commands configured by this option:

`test-suites.*.results`

test-suites:
  your-suite:
    results:
      # ...

The test-suites.*.results section contains information about the results produced by your test suite. For example, the language and framework producing the results or the path to the results themselves.

`test-suites.*.results.{language,framework}`

test-suites:
  your-suite:
    results:
      language: Python
      framework: pytest

Setting test-suites.*.results.language and test-suites.*.results.framework tells the Captain CLI how to parse the results produced by your test suite. In many cases this is optional, but it is required for certain test frameworks. For framework-specific documentation, take a look at the test framework documentation.

Valid combinations of language and framework include:

Language: .NET Framework: xUnit (see docs)
Language: Elixir Framework: ExUnit (see docs)
Language: Go Framework: Ginkgo (see docs)
Language: Go Framework: go test (see docs)
Language: JavaScript Framework: Cucumber JS (see docs)
Language: JavaScript Framework: Cypress (see docs)
Language: JavaScript Framework: Jest (see docs)
Language: JavaScript Framework: Karma (see docs)
Language: JavaScript Framework: Mocha (see docs)
Language: JavaScript Framework: Playwright (see docs)
Language: JavaScript Framework: Vitest (see docs)
Language: PHP Framework: PHPUnit (see docs)
Language: Python Framework: pytest (see docs)
Language: Python Framework: unittest (see docs)
Language: Ruby Framework: Cucumber Ruby (see docs)
Language: Ruby Framework: minitest (see docs)
Language: Ruby Framework: RSpec (see docs)

Both of these are required when one of them is set.

Commands configured by this option:

`test-suites.*.results.path`

test-suites:
  your-suite:
    results:
      path: './tmp/results/*.xml'

Setting test-suites.*.results.path tells the Captain CLI where to look for the results produced by your test suite.

Commands configured by this option:

`test-suites.*.retries`

test-suites:
  your-suite:
    retries:
      # ...

The test-suites.*.retries section contains settings related to retrying failures in your test suite. By default, retries will not be attempted- you have to configure this section if you'd like to retry failures.

`test-suites.*.retries.command`

test-suites:
  your-suite:
    retries:
      command: 'bundle exec rspec {{ tests }}'

Setting test-suites.*.retries.command specifies the templated command which the Captain CLI will use to execute a subset of the tests within your test suite when they fail. For the frameworks that support retries, we have pre-defined template variables you can specify.

For frameworks without built-in retry support (or for advanced custom scripting), we also expose jsonFilePath which will substitute a path to a file where we've written an artifact adhering to the RWX v1 Test Results Schema. This file contains the tests which need to be retried. For example, you can specify a retry command like so: bin/your-script {{ jsonFilePath }}

Commands configured by this option:

captain run

`test-suites.*.retries.attempts`

test-suites:
  your-suite:
    retries:
      attempts: 2

Setting test-suites.*.retries.attempts tells the Captain CLI how many times a given failed test should be retried. For example, if set to 2 we will run your entire suite once. Then, for any tests that failed, they will be given 2 additional chances to pass. This option applies to every failing test.

This option can be used either by itself or in combination with test-suites.*.retries.flaky-attempts. When set in combination, test-suites.*.retries.flaky-attempts takes precedence over this option for the tests which are flaky.

If this is set, test-suites.*.retries.command is required.

Commands configured by this option:

captain run

`test-suites.*.retries.flaky-attempts`

test-suites:
  your-suite:
    retries:
      flaky-attempts: 2

Setting test-suites.*.retries.flaky-attempts tells the Captain CLI how many times a given failed test should be retried if that failed test is marked as flaky either in the Captain Cloud Dashboard or in .captain/*/flakes.yaml. For example, if set to 2 we will run your entire suite once. Then, for any tests that failed and are known to be flaky, they will be given 2 additional chances to pass.

This option can be used either by itself or in combination with test-suites.*.retries.attempts. When set in combination, this option takes precedence over test-suites.*.retries.attempts for the tests which are flaky.

If this is set, test-suites.*.retries.command is required.

When you enable Cloud mode, you don't have to manually update your flakes for this behavior! As soon as Captain Cloud detects a new flaky tests, every new invocation of the Captain CLI will automatically apply the number of attempts from this option to the new flaky test. All without opening a PR!

Commands configured by this option:

captain run

`test-suites.*.retries.max-tests`

test-suites:
  your-suite:
    retries:
      max-tests: '1.5%'

Setting test-suites.*.retries.max-tests limits the situations under which the Captain CLI will retry your tests to save compute and avoid retrying when changes which break the entire test suite are made. This can be set to either a percentage (e.g. 1.5%) or an integer (e.g. 15).

When set to a percentage, the Captain CLI will not retry your tests if more than that percentage of the tests failed. For example, if the max is 1.5%, you run 1000 tests through the Captain CLI, and 15 of them fail, the tests will be retried. If 16 were to fail, though, the tests would not be retried.

When set to an integer, the Captain CLI will not retry your tests if more than that number of the tests failed. For example, if the max is 15 tests and 15 of them fail, the tests will be retried. If 16 were to fail, though, the tests would not be retried.

Default: ♾️

Commands configured by this option:

captain run

`test-suites.*.retries.fail-fast`

test-suites:
  your-suite:
    retries:
      fail-fast: true

Setting test-suites.*.retries.fail-fast will cause the Captain CLI to exit as early as possible when it's clear that further retries will not result in the test suite passing. This is useful when using both test-suites.*.retries.attempts and test-suites.*.retries.flaky-attempts.

For example, consider a configuration with 1 attempt and 3 flaky-attempts. If we run the test suite and have a non-flaky test fail and a flaky test fail, we'll start by retrying each of them. If the non-flaky test fails again, there are no further attempts to be made so it's not possible for the test suite to pass and it's not necessary to make further attempts on the flaky test to know that.

Default: false

Commands configured by this option:

captain run

`test-suites.*.retries.pre-retry-commands`

test-suites:
  your-suite:
    retries:
      pre-retry-commands:
        - 'bin/setup-database'

Setting test-suites.*.retries.pre-retry-commands configures additional commands to run prior to running the retry command. This is run before every invocation of the retry command.

Default: []

Commands configured by this option:

captain run

`test-suites.*.retries.post-retry-commands`

test-suites:
  your-suite:
    retries:
      post-retry-commands:
        - 'bin/truncate-database'

Setting test-suites.*.retries.post-retry-commands configures additional commands to run after running the retry command. This is run after every invocation of the retry command.

Default: []

Commands configured by this option:

captain run

`test-suites.*.retries.intermediate-artifacts-path`

test-suites:
  your-suite:
    retries:
      intermediate-artifacts-path: './tmp/intermediate-artifacts'

Setting test-suites.*.retries.intermediate-artifacts-path configures the Captain CLI's retry behavior to move the intermediate artifacts produced during retry attempts into the directory you specify.

Normally when retrying your failing tests, your test suite will repeatedly produce test results artifacts as it does in normal usage. When no intermediate artifacts path is specified, the Captain CLI cleans these up between each invocation of your retry command so that the command is working with a clean environment and the files from the previous invocation do not taint the results of the next invocation. When an intermediate artifacts path is specified, the Captain CLI will instead move the artifacts into directories within the path you specify so that you can analyze them later.

For example, a suite with 2 retry attempts configured might produce an intermediate artifacts directory that looks like:

repo/
├─ tmp/
│  ├─ intermediate-artifacts/
│  │  ├─ original-attempt/
│  │  │  ├─ artifact.json
│  │  ├─ retry-1/
│  │  │  ├─ command-1/
│  │  │  │  ├─ artifact.json
│  │  │  ├─ command-2/
│  │  │  │  ├─ artifact.json
│  │  ├─ retry-2/
│  │  │  ├─ command-1/
│  │  │  │  ├─ artifact.json
│  │  │  ├─ command-2/
│  │  │  │  ├─ artifact.json

If you find yourself reaching for this option, you might want to try the RWX v1 JSON Reporter. It merges all of those intermediate artifacts and provides a single artifact that's representative of the outcome of the original suite, all retries, and quarantining.

Default: []

Commands configured by this option:

captain run

`test-suites.*.partition.command`

test-suites:
  your-suite:
    partition:
      command: 'bundle exec rspec {{ testFiles }}'

Setting test-suites.*.partition.command specifies the templated command which Captain CLI will use to execute a partition of your tests.

Upon expressing intent to partition, the resulting command will be used in place of the default test-suites.*.command. This is done by supplying captain run with --partition-index and --partition-total flags.

Commands configured by this option:

captain run

`test-suites.*.partition.delimiter`

test-suites:
  your-suite:
    partition:
      delimiter: ','

Setting test-suites.*.partition.delimiter specifies the characters used to separate the partition of test files that are ultimately passed to your partition command.

Default: ' '

Commands configured by this option:

captain run

`test-suites.*.partition.globs`

test-suites:
  your-suite:
    partition:
      globs:
        - 'spec/**/*_spec.rb'
        - 'test/**/*_test.rb'

Setting test-suites.*.partition.globs informs Captain CLI's run command what files you intend to partition. You can specify multiple glob patterns.

The resulting files produced by these globs are combined with recorded test file timings to split your test suite into approximately even partitions.

Files that have timing information are placed into partitions using a first-fit descending strategy. Files that have no timing information will round-robin the partitions. Both strategies are deterministic.

Commands configured by this option:

captain run