captain run
Executes a test suite and parses test result artifacts. This command is the entry point for other Captain CLI features such as test retries, quarantining, and the production of test result artifacts.
Captain CLI enhances the test runner you are already using.
This is achieved by orchestrating the configured run command or argument supplied to captain run.
Prior to running your command, Captain CLI retrieves a run configuration. Depending on the features you opt-into and the features supported by your framework, this includes things like quarantined tests and known flaky tests.
Check out Cloud mode to have Captain manage the run configuration on your team's behalf. Leverage the Cloud UI to make changes to the run configuration without ever needing to open a PR or worry about how a particular test is identified.
Depending on the outcome of your tests, Captain CLI uses this metadata to modify the exit code returned.
When using Captain's quarantined functionality, failures from tests under quarantine won't cause your build to fail.
Rather retry than quarantine? Captain CLI has you covered, providing you insight into just how many retries it took for a test to pass. From here, you can configure powerful targeted retry policies to avoid wasting compute retrying tests that won't pass.
Usage
captain run [suite-id] [FLAGS]
Examples
Running a test suite named example-suite-id using the options configured in .captain/config.yaml:
captain run example-suite-id
Passing an additional positional argument to the command configured for example-suite-id:
captain run example-suite-id ./tests/test.js
Passing an additional flag to the command configured for example-suite-id:
captain run example-suite-id -- --detectOpenHandles
Running your tests, parsing and uploading test results without a configuration file:
captain run example-suite-id --test-results jest-result.json -c jest
Please see supported frameworks for more in-depth framework specific examples.
General Options
--command <COMMAND>
Required. The command you use to run your tests. You can also specify this with the shorthand -c <COMMAND>.
--config-file <PATH_TO_CONFIG>
Defaulted. Overrides the location of the Captain CLI config file.
--framework <FRAMEWORK>
Configurable. The framework of the test suite.
This value is required if --language is set.
These can be set to anything, but Captain has specific handling for supported frameworks.
--language <LANGUAGE>
Configurable. The programming language of the test suite.
This is required if --framework is set.
These can be set to anything, but Captain has specific handling for supported frameworks.
--help
Displays more information about a command or subcommand.
--intermediate-artifacts-path <PATH>
The path to store intermediate artifacts under. Intermediate artifacts will be removed if not set.
--print-summary
Configurable. Prints a summary of all tests to the console.
--quiet
Configurable. Disables most default output.
--reporter <TYPE=OUTPUT_PATH>
Configurable. One or more TYPE=OUTPUT_PATH (e.g rwx-v1-json=path/to/ouput.json) pairs to enable different reporting options.
Available reporters are rwx-v1-json, junit-xml, markdown-summary, and github-step-summary.
--suite-id <SUITE_IDENTIFIER>
Required. The identifier of the test suite. This value should be unique to your Captain organization.
When neither --suite-id nor the environment variable CAPTAIN_SUITE_ID are specified, then the first positional argument will be treated as the suite ID. For example, if you run captain run example-suite-id then example-suite-id will be used for the suite ID.
--test-results <STRING>
Required. A filepath to the native test runner's test results output. This option supports globs for multiple result files.
Captain Cloud Options
--branch <BRANCH_NAME>
Sometimes Required1. The branch name of the commit being built.
Use this flag or set the environment variable CAPTAIN_BRANCH.
--build-url <BUILD_URL>
Sometimes Required1. The URL of the build results.
Use this flag or set the environment variable CAPTAIN_BUILD_URL.
--commit-message <STRING>
Sometimes Required1. The git commit message of the commit being built.
Use this flag or set the environment variable CAPTAIN_COMMIT_MESSAGE.
--fail-on-upload-error
Configurable. Return a non-zero exit code in case the test results upload fails.
--sha <COMMIT_SHA>
Sometimes Required1. The git commit sha hash of the commit being built.
Use this flag or set the environment variable CAPTAIN_SHA.
--title <TITLE>
A descriptive title for the test suite run, such as the commit message or build message. This is used only when viewing results in the Captain Cloud web interface.
If using a supported CI provider, this will be set automatically.
Otherwise, you may use this flag or set the environment variable CAPTAIN_TITLE.
--who <STRING>
Sometimes Required1. The person or machine that triggered the build.
Use this flag or set the environment variable CAPTAIN_WHO.
Captain OSS Options
--update-stored-results
If set, captain will update its internal storage files under .captain with the latest test results, such as flaky tests and test timings.
Retry Options
--fail-retries-fast
Configurable. If set, your test suite will fail as quickly as we know it will fail.
For example, with --retries 1 and --flaky-retries 5, you might have a non-flaky test that we stop retrying after 1 additional attempt.
In this situation, we know the tests overall will fail so we can stop retrying to save compute.
Similarly if you only set --flaky-retries 1, we can stop retrying if any non-flaky tests fail because we won't retry them.
--flaky-retries <INTEGER>
Configurable. The number of times failing flaky tests should be retried.
This takes precedence over --retries if the test is known to be flaky.
For instance, --flaky-retries 2 would mean a maximum of 3 attempts of any flaky test.
This value defaults to -1.
--post-retry <STRING>
Configurable. Commands to run immediately after Captain CLI retries a test. You may supply this option multiple times to run multiple commands post retry.
--pre-retry <STRING>
Configurable. Commands to run immediately before Captain CLI retries a test. You may supply this option multiple times to run multiple commands pre retry.
--max-tests-to-retry <STRING>
Configurable. If set, retries will not be run when there are more than N tests to retry or if more than N% of all tests need retried. Supports both static numbers or percentage.
For instance, --max-tests-to-retry 15 or --max-tests-to-retry 1.5%
--retries
Configurable. The number of times failed tests should be retried.
For example, --retries 2 would mean a maximum of 3 attempts of any given test.
This value defaults to -1.
--retry-command string
Configurable. The command that will be run to execute a subset of your tests while retrying.
This flag is required if --retries or --flaky-retries is provided.
Please see supported frameworks for framework specific examples.
Partition Options
--partition-command
Configurable. The command that will be run to execute a partition of your tests.
This flag is required if --partition-index or --partition-total is provided.
--partition-delimiter
Configurable. A string used to delimit test files passed to the native test framework.
This value defaults to a space.
--partition-globs <STRING>
Configurable. A file path glob used to inform captain of the files you wish to partition.
--partition-index <INTEGER>
Given test suite that is split into N partitions, this represents a particular instance of a partition.
This value is 0-indexed.
This value is used in tandem with --partition-total.
For example, --partition-index 0 --partition-total 2 means you are executing the first of 2 partitions.
This value defaults to -1.
Use this flag or set the environment variable CAPTAIN_PARTITION_INDEX.
--partition-total <INTEGER>
The desired number of partitions.
This value is used in tandem with --partition-index.
For example, --partition-index 0 --partition-total 2 means you are executing the first of 2 partitions.
This value defaults to -1.
Use this flag or set the environment variable CAPTAIN_PARTITION_TOTAL.