Implementing a Remote Debugger with Node and tmux

On most CI/CD platforms, the developer experience of debugging tasks is painful.

Part of the problem stems from having to git push to run workflows, which we've written about and solved by providing a Mint CLI which can be used to start runs.

The other part of the problem is that CI/CD tasks are usually running on a different operating system and with a different configuration than local development. Often, local development is done on macOS, but CI/CD tasks are running on a Linux distro like Ubuntu. Although engineers can run the same bash scripts locally as on CI/CD, sometimes the differences result in code that "worked on my machine" not working in CI.

Without direct access to the machine running the code, debugging can be difficult. You often have to guess at what's wrong and add statements to print additional information or inspect file system state. The feedback loop is usually very slow.

This entire experience can be made much better with a remote debugger. We built one into Mint using a familiar construct: setting a breakpoint. It's as easy as calling mint-breakpoint from a task and then using the local CLI to run mint debug {debugId}.

Approach

We wanted to provide as much flexibility as possible to engineers debugging tasks. Some CI platforms only allow opening shells at the beginning or end of a task's execution. We wanted to allow opening the shell at any point, which is why we came up with the breakpoint approach. It can be set anywhere. You can call mint-breakpoint directly in the run definition of a task, or you can call it from within another script.

Client Implementation

Here's how the mint-breakpoint script works.

It first generates an identifier for the debug session.

1
debugid="mintdebug_$(uuidgen)"

It then starts a tmux process, using that identifier.

1
tmux -L "$debugid" new-session -d -s "$debugid" /bin/bash

Next, it needs to trigger starting an SSH server to allow remote connections. Mint tasks run inside containers, but the SSH server runs outside the container on the host. Therefore we need a mechanism from inside the container to notify the host that it needs to start an SSH server. We do this by writing to a pipe to communicate from inside the container to outside of it.

1
2
3
pipe="$MINT_BREAKPOINT_DIR/pipe"
msg='{"debugId":"'"$debugid"'"}'
echo "$msg" >"$pipe"

Finally, mint-breakpoint needs to wait for the debugging session to complete. It does this by polling tmux to see if the session is still open.

1
2
3
while (tmux -L "$debugid" ls 2>/dev/null || true) | grep -q "$debugid"; do
  sleep "$sleep_seconds"
done

Server Implementation

With the breakpoint set and a tmux session running, the server then needs to start an ssh session to enable connections. It does this using the ssh2 node package.

When a connection is established, the host runs a process to connect to the tmux session running inside the container:

1
/bin/bash -c /usr/bin/tmux -L ${debugId} attach-session -t ${debugId}

It then uses node-pty to directly attach the ssh session to the bash process.

1
2
3
4
5
6
7
8
9
10
11
// spawn the tmux attach process
pty = spawn(...);

pty.onData((data: string) => channels.stdout.write(data));

pty.onExit(({ exitCode }: { exitCode: number }) => {
  channels.exit(exitCode);
  conn.end();
});

channels.stdin.on('data', (data: string) => pty.write(data));

CLI Implementation

The last piece of the implementation is facilitating an easy way to open an ssh client.

We achieve this in the Mint CLI. It's open source, so you can see exactly how it works:

https://github.com/rwx-research/mint-cli/blob/26ae56c731df14bfb94ce76969abb06f4ab4b9e3/internal/cli/service.go#L37

It uses the RWX Cloud authentication to fetch the necessary keys so that starting the ssh session is as easy as running

1
mint debug {debugId}

Demo

Here's a 95 second video on how easy it is to remotely debug tasks with Mint. It's the best developer experience in CI/CD.

Follow Along

We write a lot about software engineering best practices and CI/CD pipelines in Mint. Follow along on X at @rwx_research, LinkedIn, or our email newsletter