Twice on the same SQL Server estate on Azure VMs, the database engine kept running while something around it failed. In the first incident, customers lost access for 71 minutes, most likely because a backup job kept retrying against a storage credential that did not exist. In the second, seven databases went weeks without a full backup, and nothing alerted until their log backups started failing, by which point point-in-time recovery was already being lost. In both cases “is SQL Server running?” was the wrong question.
The client is a regulated financial-services SaaS platform running SQL Server on Azure VMs in Always On availability groups, protected by Azure Backup. The engagement is summarized in our case study.
Incident 1: a 71-minute outage with no crash
Customers reported the database unreachable for 71 minutes early one morning. The usual explanations were ruled out: no service stop or start, no SQL Server restart, no availability-group failover, and the private endpoint to the backup storage account resolved correctly and accepted connections on port 443.
The Windows Application event log told a different story. Every few seconds, a BACKUP LOG ... TO URL issued through the SQL Server IaaS Agent extension’s automated backup failed against an Azure storage account and was retried almost at once. Then we listed the credentials. sys.credentials, and the database-scoped credential views in master and msdb, returned zero rows. Backup to URL needs a credential that holds either a shared access signature (SAS) or, on recent versions, a managed identity. There was neither, which is the most likely reason every attempt failed.
Our working explanation, presented as one, is that each failed attempt briefly tied up the backup machinery, and at that retry rate client connections appeared to hang. The data shows the retry storm; it does not prove how the storm blocked clients, or why the outage ended when it did, since the errors continued after connections recovered. We reported both gaps.
Two details helped. Both nodes showed the same pattern, which pointed to configuration rather than a sick host. And the VM had two backup pipelines: system-database fulls to a local target kept succeeding, while the URL log backups never completed and so never appeared in msdb history. Backup history looked healthy because only the working pipeline wrote to it.
We recommended stopping the retry loop, reissuing the credential, and alerting on backup failures against the storage URL; that night, customers noticed before any monitor did. Managed-identity credentials for backup to URL need SQL Server 2022 CU17 or later on an Azure VM. On earlier versions, use a SAS credential and track its expiry like any other secret.
Incident 2: missing full backups that never failed
Months later, transaction-log backups had stopped for 7 of 30 protected databases, so their point-in-time recovery ended at the last log backup. Four could no longer truncate their logs; the largest log file had reached 141.2 GB, on volumes with 12 to 14 percent free. Three behaviors combined.
1. Azure Backup will not extend an old log chain. When the latest full backup behind a log chain is more than 15 days old, the next log backup fails unless Azure Backup can first take a “remedial” full backup to re-base the chain.
2. The remedial full could only run on the primary. For SQL Server 2022 and earlier, Azure Backup takes full and differential backups of availability-group databases only on the primary replica, whatever the backup preference. Log and copy-only backups follow the preference. These groups preferred secondary replicas, so log backups ran on a secondary, where the remedial full is not allowed, and the error says exactly that. From SQL Server 2025, secondaries can take full backups too, so this trap is version-dependent.
3. Why the fulls were missing. When protection was first configured in bulk, several configure-protection jobs failed because another configure operation was already running on the same item, and they were not retried. Those items still showed as protected and still received log backups, but no daily full was ever scheduled for them. The signature: the vault took exactly 23 full backups a day against 30 protected items.
An unplanned failover masked the problem for a while: it put the log-backup node into the primary role, the remedial fulls succeeded, and the clock reset. Fifteen days later the failures returned.
A second, unrelated change blocked the repair. A dormant default SQL Server instance had been disabled. In our testing, the Azure Backup plugin’s configure-protection path connected to that default instance first and failed when it could not, while discovery and backups carried on normally. We reproduced the failure on a single item, then saw it succeed once the default instance was temporarily started.
The repair was to rediscover the workloads, re-apply the existing policy one item at a time (each waiting for its configure job, since concurrency caused the original failure), and take an on-demand full of each database. The largest, about 1.7 TB, took two and a half hours. No failover, no production instance restart, no policy change; the default instance was disabled again afterwards. Log truncation resumed, and a follow-up shrink of three log files moved the log volumes from 11–13 percent free to 27–28 percent.
The repair could not recover about three days of point-in-time recovery for those seven databases, or a similar earlier window before the failover. We documented both in an explicit data-loss statement.
Why the alerting missed it
The existing alert fired on failed backup jobs. For 15 days this defect produced no failed jobs; it produced no full-backup jobs at all. Once the chain broke, failures arrived quickly, more than 150 failed log backups a day, but by then the recovery gap had already started.
The lesson is to alert on absence. “A protected item with no successful full backup in more than two days” would have caught this the day after the batch failed.
Checks you can run today
On each SQL Server that backs up to URL, confirm a credential exists:
SELECT name, credential_identity FROM sys.credentials;
Check the latest full and log backup for every database. Exclude copy-only fulls, such as those taken by VM-level snapshot backups, because they cannot serve as the base of a log chain. A database whose newest full is approaching 15 days old needs attention:
SELECT d.name,
MAX(CASE WHEN b.type = 'D' AND b.is_copy_only = 0 THEN b.backup_finish_date END) AS last_full,
MAX(CASE WHEN b.type = 'L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases AS d
LEFT JOIN msdb.dbo.backupset AS b ON b.database_name = d.name
GROUP BY d.name;
Run it on every replica, because in an availability group the history is split across nodes. Also find logs waiting on a backup, and check where backups are preferred:
SELECT name, log_reuse_wait_desc FROM sys.databases WHERE log_reuse_wait_desc = 'LOG_BACKUP';
SELECT name, automated_backup_preference_desc FROM sys.availability_groups;
Then, outside SQL Server:
- In the vault, compare full backups per day with the number of items whose policy calls for a daily full. They should match.
- Look for configure-protection jobs that failed and were never retried.
- Alert on repeated
BACKUP LOGfailures against your backup storage URL, not only on failed vault jobs. - Before disabling a default instance that looks unused, check whether a backup agent depends on it.
How we can help
Both incidents were diagnosed from evidence the estate already held: event logs, msdb history and vault job counts. If your SQL estate has backups you assume are working, talk to us and we will measure them.
