When the one engineer who understood a platform leaves, the risk is rarely the code. It is the operational knowledge and access that lived with that person, and it shows up as outages nobody can fix, launches that stall, and access nobody can account for in an audit. The lesson: inventory secrets, state, credentials and access before you need them, and check every finding against the live cloud.
We worked through this with an AI software company whose lead engineer had been the main author of 15 of its 22 repositories. The engagement produced a 28-item continuity and risk register, and its most valuable finding was almost anticlimactic: a finished product’s launch had stalled, and the immediate blocker was one expired deploy credential. The case study has the outcome. This post is the checklist we now use, written so you can run it on your own estate.
1. Who holds the secrets, and the keys to the secrets
Start with the secrets store itself. A self-hosted vault that must be unsealed after a restart is only as recoverable as its unseal keys and root token. If those were held by one person, the platform is one reboot away from an outage nobody can fix.
- Find the unseal key shares and recovery keys, and confirm at least two people can reach them.
- If they cannot be found while the vault is still running and unsealed, export what you need and stand up a re-keyed instance now, not after an unplanned restart.
- Check cloud key vaults for purge protection and for single-copy keys, such as a license-signing key whose loss would invalidate every license it signed.
2. Infrastructure state that lives on one laptop
Terraform and OpenTofu need their state file to manage existing resources safely. If no repository configures a remote backend, the state was probably local, on the departed engineer’s machine.
Without it, an apply may try to recreate resources that already exist, which can take down DNS, mail or the site. Until state is recovered or rebuilt with imports, treat the live infrastructure as manually managed and do not apply. Then move state to a remote backend with locking, and run plan and apply from CI rather than a workstation.
3. Deploy credentials about to expire
Service-principal secrets and other pipeline credentials have expiry dates, and nobody notices until a deploy fails. That was the credential in our case: the pipeline’s service-principal secret had expired, so CI could no longer authenticate to the cloud and a product that had been built was never deployed. A removed auto-deploy step and a workflow bug also needed fixing, but the expired secret alone was stopping every deploy. From the outside it looked like a product problem, not an access problem.
- List every service principal and app registration your CI uses, with its owner and expiry date.
- Give each one a named owner. An ownerless service principal is one nobody will rotate.
- Where the platform supports it, replace stored secrets with workload identity federation (OIDC), so there is no secret to expire or leak.
4. Committed secrets in history
Search full git history, not just the current tree, across every repository the person touched. Look for API keys, webhook and token-signing secrets, default client secrets in bootstrap code, committed SSH public keys, and build artifacts that were tracked by accident because a .gitignore was broken.
Rotate first, then purge. A committed SSH public key also needs a sweep of every authorized_keys file and cloud-init or user-data template, because removing it from the repository does nothing to the hosts that already trust it.
5. Access revocation and break-glass
Disabling the directory account is the start, not the end. Access that the identity provider does not control survives it:
- source-control organization membership and personal access tokens
- VPN client certificates and SSH keys
- release-signing keys, which may need re-keying of a release chain
- vendor dashboards for payments, email, DNS and registrars
- hard-coded admin or owner identities in bootstrap code, which will provision the departed person as owner on the next fresh deployment
- groups and resources whose only owner was the disabled account
Then check the other direction: make sure no single remaining person holds every admin role, add a second owner to each subscription, and confirm break-glass access works without any one individual.
6. Separate core IP from plumbing and demos
An estate with dozens of repositories and thin documentation is hard to value and hard to protect. Classify each repository by what it is: defensible core IP, supporting plumbing, duplicates and forks, stubs, or demos. Rank by value, maturity and authorship.
This tells you where to spend scarce attention. Core IP gets cross-training, consolidation of divergent copies, and backups of the data it depends on. Demos and stubs get archived so they stop cluttering the picture.
7. Rebuild documentation from the code
When the author is gone, the code is the only reliable record. Write the architecture and a short deep-dive per repository from what the code actually does, not from old design documents, which describe intent. Where they disagree, note it.
Then capture the tribal steps. Have a new engineer perform a witnessed end-to-end deploy of a non-production environment and write down every manual step. That one exercise surfaces most of what lived only in someone’s head.
8. Verify ground truth in the live cloud
Repository contents tell you what was intended. The live cloud tells you what exists. Run a read-only audit with a temporary, least-privilege identity, and remove that identity when the assessment closes.
This is also where we made a mistake of our own. In our first report we stated that there was no production environment, because the subscriptions named for production appeared empty. That was wrong. The audit identity had been granted access to the real production subscription after its last CLI sign-in, and az account list reads the CLI’s locally cached subscription list unless you pass --refresh. The cached list held only older, empty subscriptions. Refreshing it showed production running.
The lesson is broader than one CLI flag. Tools show you a view, and the view can be stale, filtered or scoped to the wrong identity. Before you report that something does not exist:
# fetch the subscription list from Azure rather than the local cache;
# --all also shows subscriptions that are not in the Enabled state
az account list --refresh --all --output table
- Confirm which identity you are signed in as and what it can see.
- Cross-check with a second source: billing, DNS, the CI pipeline’s target, or the people who run it.
- Treat a finding of absence with more suspicion than a finding of presence.
We corrected the report and the risk register. Measured, not assumed, applies to our own findings too.
How we can help
We run continuity assessments that map the estate, build the risk register, and check every finding against the live cloud before it reaches a report. Read the case study, or see our services for how we scope this work.
