A self-hosted CI runner pool on a logistics platform we operate had zero runners online for 6 hours 38 minutes, and nothing reported it. The watchdog built to catch exactly that ran on the same pool, so it waited in the queue behind the outage it was supposed to detect. A health check that shares a failure domain with the thing it checks only works when you don’t need it.
Read More