A failover exercise we ran for a client hit three defects, and any one of them would have stopped a real recovery. All three had been found in the previous exercise fifteen months earlier and worked around on a tag that sat on no branch, so the main configuration never changed. A workaround that does not merge is not a fix. It is a known defect waiting for the next disaster.
The exercise
The estate runs in East US 2, with Central US as the failover region. The failover plan is infrastructure as code: the Terraform that describes production is applied to the second region, workspace by workspace, to rebuild the landing zone, network and five application workloads at production sizes. Data comes back through Azure Backup cross-region restore. Against the current code, governance and networking applied cleanly. The application layer failed four times before it went through.
Why imported resources hide failover bugs
All three defects had the same root cause, and it applies to most estates that adopted Terraform after they were built.
The production resources had been imported into Terraform, not created by it. An imported resource only goes through the read and update paths: Terraform compares configuration with reality, finds nothing to change, and moves on. The create path, meaning the validation and API calls that build a resource from nothing, had never run. A failover build is nothing but creates, so the exercise was the first time that code really ran. In a real incident, the first time would have been the disaster.
A clean terraform plan against production tells you the configuration matches what exists. It tells you nothing about whether the same configuration can build the estate from nothing.
The three defects
| # | Defect | Error at create time | The old workaround |
|---|---|---|---|
| 1 | IOPS, throughput and sector-size attributes set on Premium_LRS managed disks | The azurerm provider refuses them: they apply only to Ultra Disk and Premium SSD v2 | Comment out the disk resource |
| 2 | Boot-diagnostics storage URIs hard-coded to accounts in the primary region | Azure returns StorageAccountLocationMismatch: a custom diagnostics account must be in the VM’s region | Set every URI to null |
| 3 | A standalone managed disk with the same name as the OS disk the VM creates | Azure returns ConflictingUserInput: the disk already exists, and only attaching it is supported | Comment out the disk resource |
None could appear in the primary region. The provider checks disk attributes when it creates a disk, and these disks had only been imported. The diagnostics accounts sat in the VMs’ own region. And the duplicate name never clashed because both disks already existed at import. In a fresh region, Terraform created the standalone disk first, and the VM then tried to create its OS disk under the same name.
Why the workarounds never merged
The old workarounds were unconditional, which is probably why they stayed on a tag. Nulling the boot-diagnostics URIs on the main branch would have moved production off its dedicated diagnostics accounts. Commenting out a managed resource is worse: Terraform plans to destroy what it no longer sees. Nobody merges a change that breaks production to fix a path that only runs in a disaster.
The fix is to make each change conditional, so that production plans “No changes” and the failover path still works:
# Inside the azurerm_linux_virtual_machine resource
boot_diagnostics {
# Primary region keeps its dedicated account; null means managed boot diagnostics
storage_account_uri = var.location == var.primary_location ? var.diag_storage_uri : null
}
resource "azurerm_managed_disk" "vm_disk" {
count = var.manage_adopted ? 1 : 0 # true only for the imported production estate
# ...
}
In the azurerm provider, a boot_diagnostics block with a null storage_account_uri uses a managed storage account, which Microsoft recommends anyway. The disk attributes were removed outright, since they never applied to Premium SSD. The standalone disk is gated behind the “adopted resources” flag, and the VM’s OS disk builds its name from the same expression instead of referring to the gated resource. Before each fix merged, every production workspace still planned “No changes”.
Proving the fixes were fixes
Getting past a defect on the day only proves the workaround. So we tore the failover environment down and ran a second, clean failover the next day from merged code. All three workspaces applied on the first attempt, including the application layer. Both environments were then destroyed without errors, and production was untouched throughout.
One of the first day’s four failures was not a real defect: a race between deleting orphaned disks and creating VMs that needed those names, caused by applying fixes repeatedly to a half-built environment. It cannot happen in a clean-slate failover, so we did not count it as a finding.
Restoring a VM that no longer existed
For the data half, we restored a VM that had been decommissioned from production six months earlier, from its replicated backup in the secondary region. The restore took 21 minutes from an application-consistent recovery point, and the VM came up running in Central US. Restoring a system that is completely gone is a stronger test than restoring one that still exists, because nothing is left to fall back on.
What else the exercise surfaced
- Privileged access expired twice. Just-in-time elevation lapsed between the Terraform build and the restore, so the cross-region restore failed a permission check, and it lapsed again before teardown. It also had to be an Azure role at subscription scope: the Entra Global Administrator role does not by itself grant Azure resource permissions. We recommended a standing break-glass DR role, or at least an explicit re-activation step in the runbook.
- Private IPs are not preserved. Allocation was dynamic, so failover addresses differed from production. Anything with a hard-coded peer address, such as firewall route tables, needs checking in a real cutover.
- The failover estate drifts at once. Within minutes, Microsoft Defender deployed an extension onto all five VMs, outside Terraform’s state. Good security behavior, but the failover estate is never exactly what the code describes.
Checks for your own estate
- Find out which production resources were imported rather than created. Their create path has never been tested.
- Search the repository for DR or exercise tags and branches that never merged. Each one is a list of known defects.
- Run the failover build into an empty region, then again from merged code. Make region-specific fixes conditional, and require production to plan “No changes” before they merge.
- Confirm cross-region restore is available: it needs a geo-redundant vault with Cross Region Restore enabled, and newly enabled items can take up to 48 hours to appear in the secondary region.
- Restore something that has been decommissioned, not just something still running.
- Compare how long your just-in-time elevation lasts with how long a recovery actually takes.
How we can help
We run DR exercises against the real estate and the real code, and we write up what failed as well as what worked. If your failover plan has only been tested on paper or against imported resources, get in touch and we will help you find out whether it builds.
