Platform Reliability & FinOps

A failover exercise we ran for a client hit three defects, and any one of them would have stopped a real recovery. All three had already been found in the previous exercise fifteen months earlier, worked around on a tag that sat on no branch, and never fixed. A workaround that does not merge is not a fix: it is a known defect waiting for the next disaster.

Twice on the same SQL Server estate on Azure VMs, the database engine kept running while the service around it failed. In the first incident, customers lost access for 71 minutes because a backup job was retrying against a credential that did not exist. In the second, seven databases went without full backups, and nothing alerted until their transaction-log backups stopped and point-in-time recovery was already lost. In both cases “is the service running?” was the wrong check.

A client’s AWS bill went from roughly $1.7K in one month to $7.8K the next. Nothing in the application had changed. Three security and logging features, each reasonable on its own, had been connected so that each one’s output became another’s input. Logging needs to be designed as a system, because a single console change can turn it into a loop that pays for itself to grow.

Transferring a repository to another GitHub organization looks like a single button, and for history, issues and pull requests it is. What breaks is everything outside the repository that refers to it by owner and name: cloud logins, tokens, infrastructure code and deployment controllers. The fix is to find those references first and change them in the right order, so nothing fails on transfer day.

A self-hosted CI runner pool on a logistics platform we operate had zero runners online for 6 hours 38 minutes, and nothing reported it. The watchdog built to catch exactly that ran on the same pool, so it waited in the queue behind the outage it was supposed to detect. A health check that shares a failure domain with the thing it checks only works when you don’t need it.