The logging feedback loop that more than quadrupled an AWS bill

A client’s monthly AWS bill went from roughly $1.7K to $7.8K in a single month, with no meaningful change in application usage. Three security and logging features, each reasonable on its own, had been connected so that each one’s output became input for the others. Logging has to be designed as a system, because a single console change can turn it into a loop that pays for its own growth.

The three ingredients

The account’s security tooling had been built out over the summer. By early September two of the three pieces were in place:

• GuardDuty Malware Protection for S3 on the CloudTrail log bucket. It scans every object uploaded to a protected bucket and can tag each object with the result. It is billed by objects evaluated and data scanned.
• A CloudTrail trail recording S3 data events for every bucket, with no bucket filter, and streaming those events to CloudWatch Logs as well.

The third piece came in a single console session over about five minutes: S3 server access logging was turned on for 14 buckets. Every one of them was pointed at the CloudTrail log bucket, including the CloudTrail bucket itself.

How the loop worked

There were two loops, and the malware scanner sat in both.

Loop A. GuardDuty scans a new log file, which means reading it and writing a tag with the result. Access logging on the bucket records those requests and delivers a new access-log file into the same bucket. That file is new, so GuardDuty scans it, and the cycle repeats.

Loop B. The trail records the same reads, tag writes and log deliveries as S3 data events, and writes them back into the bucket as more log files. Those get scanned too, and each scan generates more data events.

AWS’s documentation says the destination bucket for access logs should not itself have access logging enabled, and that delivering logs to the source bucket causes an infinite loop of logs. The malware plan turned that loop into a large bill, because every turn of it was a billable scan.

The evidence of a closed loop was clear. In a two-hour sample on the second day, GuardDuty tag writes and access-log deliveries on the CloudTrail bucket ran almost one-for-one, at about 57,000 an hour each. Application traffic doesn’t look like that.

What it cost

The loop ran for about three weeks at roughly $250–290 a day. Smaller costs followed the same curve: S3 request and tagging charges, CloudWatch Logs ingestion of the extra data events, EventBridge events for each new object, and a support plan billed as a percentage of spend. Left running, it would have cost about $8,700 a month.

Why it ran for three weeks

The signals were there from the first full day. Daily scans jumped from about 12,000 to about 790,000 and kept climbing. Based on daily spend, the budget forecast alert should have fired the next day and the actual-spend alerts within a week. Those alerts were configured to go to a single mailbox with no named owner, and no action followed. Cost anomaly detection wasn’t set up until the evening the scanning was stopped.

The first remediation attempt shows how easy it is to treat the symptom. Deleting the malware plan on the log bucket stopped the scanning. A new plan was created on the same bucket 42 seconds later, which restarted the loop until it too was removed. The feedback path is the self-targeted logging plus the unfiltered data-event selector. Remove those first. Otherwise anything per-object attached to the bucket later can start the loop again.

How to check your own account

You don’t need special tooling.

• Cost Explorer, daily, grouped by service and then by usage type. Look for usage types containing MalwareProtectionS3ScanRequest and DataEventsRecorded. A jump from a few dollars a day to hundreds is obvious at daily granularity and easy to miss in a monthly view.
• Malware Protection plans. List them and check for any on buckets that only receive logs.
• Bucket logging targets. List every bucket’s logging configuration. Flag any bucket that logs into itself, or into a bucket that something else scans or triggers on.

for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
  t=$(aws s3api get-bucket-logging --bucket "$b" \
        --query 'LoggingEnabled.TargetBucket' --output text 2>/dev/null)
  case "$t" in
    ""|None) ;;                          # logging off, or no access
    "$b")    echo "SELF-LOGGING: $b" ;;
    *)       echo "$b -> $t" ;;
  esac
done

• Object counts. CloudWatch’s daily NumberOfObjects metric for each log bucket. In this account the CloudTrail bucket went from under a million objects to over 22 million in about two and a half weeks.

Designing logging so it can’t loop

• Use a dedicated access-log bucket with its own access logging off, no malware plan, a per-source prefix, an expiration rule, and a bucket policy that lets only the S3 logging service (logging.s3.amazonaws.com) write to it.
• Never send access logs into the CloudTrail bucket. Keep log types separate so lifecycle rules, queries and permissions can be applied to each.
• Exclude log buckets from CloudTrail data events. Add a resources.ARN condition with NotStartsWith for each log bucket’s ARN to the advanced event selector. Better still, record data events only for buckets that hold application data.
• Put Malware Protection for S3 only on buckets that take uploads from untrusted sources, scoped to the upload prefixes. Compressed JSON log files from AWS services don’t need scanning. In this account, the buckets that actually received user uploads had no malware protection until the cleanup.
• Manage logging, trail selectors and GuardDuty plans as infrastructure-as-code with peer review. Add an AWS Config custom rule or a pre-deployment check that flags any bucket whose logging target is itself or the CloudTrail bucket.
• Route budgets and anomaly alerts to people who will act on them: a named owner and a team channel, a one-business-day response expectation, a low anomaly threshold, and a per-service budget for GuardDuty.
• Check the result the next day. After any logging or security-service change, look at that service’s line in Cost Explorer and at the target bucket’s object count.

A note on evidence

The investigation was read-only. We captured every finding to an evidence pack with a SHA-256 manifest, used CloudTrail’s signed digest files to confirm the sampled log files were unaltered, and wrote a script that recomputes each figure in the report from the pack. That made the findings checkable by the client and gave a sound basis for requesting a billing adjustment from AWS. One footnote: our own reads of the log bucket triggered a GuardDuty anomalous-behavior finding, which we documented as caused by the audit.

How we can help

If your AWS bill has moved and nobody can say why, we can trace it to the configuration change behind it, verify the finding, and put guardrails in place so it doesn’t recur. Our services page describes how we approach platform reliability and FinOps work.