A self-hosted CI runner pool on a client’s logistics platform had zero runners online for 6 hours 38 minutes, and nothing reported it. Every deploy, the infrastructure apply and the restore drill depended on that pool, and so did the watchdog built to catch exactly this, so it waited in the queue behind the outage it was meant to detect. A health check that shares a failure domain with the thing it checks only works when you don’t need it.
What happened
Every runner in the pool went offline: 0 of 5. At the peak, 28 workflow runs were stuck in the queue, including an Android and an iOS release. All 57 self-hosted jobs in the repository targeted that one Azure virtual machine scale set.
The pool had two automated controls: a watchdog workflow that checks runner health, and a reimage workflow that repairs bad instances. Both declared the same runs-on labels as every other job, along the lines of:
runs-on: [self-hosted, linux, azure]
With one sick instance, that works. With the pool empty, the detector and the fix queue alongside the deploys they exist to protect.
We found this during an audit re-run the day after. The census job that measures the estate also runs in CI, so the pass began with the estate unmeasurable: a second instance of the same pattern.
How the evidence erased itself
The watchdog’s scheduled runs did more than queue. They cancelled each other.
The watchdog set cancel-in-progress: true on its concurrency group, so each hourly run cancelled the previous one, which was still waiting for a runner. The reimage workflow correctly set cancel-in-progress: false, because two reimages at once would be dangerous. But by default GitHub Actions keeps only one pending run per concurrency group: when a new run arrives, the existing pending run is cancelled and replaced. So with one run holding the group, each new arrival cancels the one waiting behind it.
In the 12 hours covered by the last 100 runs, the record read:
| Result | Runs |
|---|---|
| Success | 37 |
| Cancelled | 30 |
| Queued | 28 |
| Failure | 3 |
Someone reading that history the next morning would see no alerts, few failures and thirty cancellations, which is exactly what a person tidying up looks like. The record of the outage was being erased while the outage ran.
When the pool healed, the new runners weren’t equivalent
The scale set recovered on its own. Four new instances registered, and at least one came up without the Azure CLI. It reported online and took work. Jobs that did not touch Azure ran fine on it. Jobs that did failed with:
Unable to locate executable file: az
Across the last 60 runs, that instance had six failures; every other instance combined had one. The failures included a backend deploy, the estate census, and the pool’s own watchdog and reimage workflow. Hours after the pool caused an outage, an unhealthy member of the pool knocked out the controls meant to keep it healthy.
The cause was in the provisioning template. Cloud-init installed the CLI before registering the runner, so the order was right. But the install was best effort:
- [ bash, -lc, "command -v az >/dev/null || curl -sSL <installer-url> | bash" ]
A failed download did not stop cloud-init, and registration ran next anyway. (Without pipefail, a failed curl piped into bash can still exit 0.) Two lines above, a comment in the same file called the Azure CLI “a hard requirement, not a convenience”. The comment was right, but the code treated the CLI as optional.
The error also misleads: it reads like a PATH problem in the workflow, so the cost lands on whoever wrote the workflow, not on the broken machine.
What to change
Take the health check out of the failure domain
At minimum, whatever acts on total pool loss must run somewhere that does not depend on the pool. Our first recommendation was to move the reimage workflow onto a GitHub-hosted runner. Options, roughly in order of independence:
- A GitHub-hosted runner for the watchdog and the repair job, which need little compute.
- A schedule outside CI, such as an Azure Function or a cron job on unrelated infrastructure, that calls the GitHub API for runner status.
- An external probe that alerts through a channel that does not depend on CI.
Make concurrency fit the job
For a watchdog, cancel-in-progress: true turns every missed run into a cancellation that looks deliberate. Prefer false, and know the default limit of one pending run per group. On GitHub.com, the newer queue: max option keeps up to 100 pending runs instead (it cannot be combined with cancel-in-progress: true):
concurrency:
group: runner-watchdog
cancel-in-progress: false
queue: max
Don’t rely on timeout-minutes to surface the problem. It limits how long a job runs, not how long it waits for a runner; a self-hosted job can sit in the queue for up to 24 hours before GitHub cancels it, and that also shows up as “cancelled”. This is why the watchdog should not be queuing at all.
Bake and test runner images
Install tools into a versioned image, not at boot from the internet. If you must install at boot, fail closed:
set -euo pipefail
command -v az >/dev/null || install_azure_cli # your installer, failing on any error
az version >/dev/null # refuse to register if this fails
./register-runner.sh
Then run a smoke job against each new image before it joins the pool, checking every tool your workflows assume. A runner that cannot authenticate is worse than no runner: it looks like capacity, and its failures land on the workflow.
Alert on queue age, not just runner count
Runner count tells you whether machines exist. Queue age tells you whether work is moving, which is what users feel. A job whose labels no online runner carries does not fail. It waits. On the same platform, a later release build sat in the queue for 58 minutes for that reason before it was cancelled and dispatched again.
Useful signals, all available from the GitHub API:
- Age of the oldest queued run, against a threshold that fits your pipeline.
- Online runners per label set, not just in total.
- The rate of cancelled runs for workflows people rarely cancel by hand.
- Failure rate per runner. One instance with six failures against one everywhere else is a broken machine, not flaky tests.
Exercise the path for when the pool is what broke
The restore drill on this platform had run and passed. What had never been tested was running it with no runners, and the written manual recovery procedure had never been exercised. If your recovery objective assumes CI, CI is part of your disaster-recovery scope.
The general rule
A watchdog has to survive the failure it watches for, whether it guards CI runners, a log pipeline or a cluster. Ask of each control: “If the thing it watches is completely gone, does this still run, and does anyone hear about it?”
How we can help
We audit CI and platform operations by measuring the live estate, not by reading the configuration. If your pipelines depend on a single runner pool, talk to us about where your watchdogs run and what they would miss.
