Self Hosting

RWX provides managed hosting of ABQ, but ABQ queues can also be self-hosted.

ABQ binaries are self-contained with no dynamic dependencies. ABQ queues have the following infrastructure dependencies:

  • must be run with access to a filesystem
  • must be able to bind to at least three network ports

ABQ versioning policy

ABQ employs the following versioning policy:

  • abq application interfaces (like argument flags, and the externally-visible behavior of ABQ) obeys semantic versioning.
  • Binary interfaces (i.e. the internal ABQ protocol) are only guaranteed to be consistent at a single specific version. Binary interfaces may not follow semantic versioning.

The interaction between an ABQ queue and workers is only tested for a single abq binary version. Moreover, abq will exit with an error if mismatched binary versions are used to facilitate a test run.

Self-hosting must ensure that abq test clients match exactly the version of a hosted queue.

Starting a queue

An ABQ queue can be started with abq start. ABQ binds to three ports, all of which are assigned arbitrarily if they are not explicitly specified:

  • --port: the port to bind the queue server to
  • --work-port: the port to bind the work server to
  • --negotiator-port: the port to bind the queue negotiator to

abq start also takes flags for what IP address to bind to, and what IP address to publicize. These flags may be different if, for example, you want to bind to a private IP address.

  • --bind: the IP address to bind server ports to. When not specified, the unspecified address 0.0.0.0 is used.
  • --public-ip: the IP address to advertise server ports as being on. When not specified, the unspecified address 0.0.0.0 is used.

For example, the following command launches ABQ with queue, work, and negotiator ports on 7000, 8000, and 9000 respectively, with public IP 172.180.0.1:

abq start \
  --port 7000 \
  --work-port 8000 \
  --negotiator-port 9000 \
  --public-ip 172.180.0.1

Currently, ABQ test runs are bound to the lifetime of the queues they run under. An ABQ queue must live as long as any test runs tied to it are on-going.

Authentication tokens

If you'd like to guard access to ABQ behind an authentication token, you can create an authentication token with abq token new:

abqs_jIyfJfIGgaOzIzvI0XmV7jzZEnWv6E

abq start should take the authentication token via --user-token:

abq start \
  --user-token abqs_jIyfJfIGgaOzIzvI0XmV7jzZEnWv6E

and abq test should specify the authentication token via --token:

abq test \
  --token abqs_jIyfJfIGgaOzIzvI0XmV7jzZEnWv6E \
  -- <your test command>

Note that the specified --user-token applies to all test suite runs against a given ABQ queue. Authentication tokens are not dynamic per run ID.

TLS

ABQ supports encrypting network traffic via TLS. When TLS is configured, all network communication between a queue and workers will be encrypted.

When TLS is configured, ABQ workers expect to communicate with the SAN DNS name abq.rwx. To create a self-signed SSL certificate with SAN abq.rwx, you can use the following command:

openssl \
  req -x509 -newkey rsa:4096 \
  -keyout server.key -out server.crt \
  -days 365 -sha256 -nodes \
  -subj '/CN=abq.rwx' \
  -extensions san \
  -config <( \
    echo '[req]'; \
    echo 'distinguished_name=req'; \
    echo '[san]'; \
    echo 'subjectAltName=DNS:abq.rwx' )

To start an ABQ queue with TLS, pass the filepath to the generated certificate and key:

abq start \
  --tls-cert server.crt \
  --tls-key server.key

abq test invocations that connect to this queue should pass the TLS certificate as well:

abq test \
  --tls-cert server.crt \
  -- <your test command>

Queue load characteristics

Resources needed by an ABQ queue will vary by the size and number of workloads.

In general, a queue with 200 concurrent test runs is expected to be performant and under-utilized (typically max 50%) when configured to run with

  • 2 commodity CPUs
  • 1 GB memory
  • 1 GB disk

Since ABQ queues persists information about completed test runs, for the purposes of retries and result aggregation, memory and disk usage may steadily increase the longer a queue is active.