A multi-agent software development platform was spending about $107 a day on model calls before it had any product traffic, and no one could say which agent was spending it. Multiplied across the tenants it planned to run, that was not a viable cost base. The fix was attribution first, then caching, routing and hard caps, each verified against a full day of real traffic rather than a projection.
The platform runs AI agents with distinct roles (tech lead, backend developer, QA). Each wakes on a schedule, checks for work and acts, and every model call passes through a metering proxy that prices it at list rates. Over about three weeks, metered daily spend went from about $107 to about $22, a 79% reduction measured over a full 24-hour window on the platform’s one active tenant. Cost per merged change fell from about $1.78 to about $0.30, assuming 74 merged changes a day, a count not re-taken on the final day.
Measure first, and close the attribution gap
At the worst point, 10.7% of calls, about $15 a day, carried no agent role and were billed to nobody. You cannot tune what you cannot attribute. We made the role a required field at the metering layer and turned “a call with no role” into an alert condition. Unattributed calls fell to 0.4%.
Short samples mislead. An early round of fixes looked like a large cut on a 10-minute sample, yet full-day spend rose to about $70 over the next days, from an attribution leak and one agent’s volume regression. After that we reported only full 24-hour windows.
Three cache layers
Each agent sends a large, stable system prompt (around 30,000 tokens) and often the same short “check for work” message. On the baseline day, tokens written into the prompt cache outnumbered output tokens about 100 to 1, so cache effectiveness was the bill.
| Layer | What it matches | Lifetime | Cost on a hit |
|---|---|---|---|
| Exact match | SHA-256 of model, system prompt, messages and tools | 24 hours | No model call |
| Semantic match | Embedding similarity, cosine 0.97 or higher (0.99 on one model family) | No fixed expiry | No model call |
| Provider-side prompt cache | The stable system and tool prefix | 1 hour | A discounted cached-input rate |
The first two layers run in the proxy. In the third, one setting mattered more than any layer we built: Anthropic’s prompt cache defaults to a 5-minute lifetime, shorter than most agents’ schedules, so nearly every scheduled call rewrote its system prompt into the cache. The proxy now requests 1-hour lifetimes from both providers.
Check current pricing before copying this: cache reads cost about a tenth of the input rate on both providers today, but 1-hour Anthropic cache writes cost more than 5-minute ones, and Gemini explicit caches bill storage by the hour. With all three layers working, 98.8% of calls on the busiest model were served from a cache.
Route by task, not by agent
Most scheduled agent work is observation: read some state, or confirm nothing changed. At current list prices, Gemini 2.5 Pro costs about four times as much per token as Gemini 2.5 Flash, for both input and output.
We moved model choice to the task level: routine sweeps and classifications went to Flash or Flash-Lite, and larger models stayed only where the output needed them. Pro spend fell from about $26 a day, 40% of the total, to $0.36 a day. The biggest single lever was giving the heartbeat, the “do I have work?” check, its own cheap model instead of each agent’s default.
Stop idle agents from thinking
At baseline, one agent spent 12 model round-trips, each paying for the full system prompt, on a scheduled prompt with nothing to do.
The fix was a cheap check before the model is called: if the agent has no new messages, recent actions, open directives or assigned issues, it exits. A skipped firing costs about $0.0001 against about $0.003 for a full heartbeat. The first version never fired because it was scoped to the deployment instead of each role, so count the skips over 24 hours before you believe a short-circuit works.
Caps for runaway loops
Once caching and routing were in place, cost spikes were volume spikes, and nearly every one was a loop. We capped the main loop classes twice, in the prompt and on the server, because prompt rules drift silently.
- Duplicate posts: server-side deduplication of near-identical messages within 2 hours.
- Probe loops: status checks capped at two per heartbeat.
- Fix-push loops: at most three CI-fix attempts per change per agent per hour.
- Stale directives: old instructions expire instead of re-triggering work.
One class needed its own detector. The model called only the agent runtime’s built-in actions and never wrote a final reply, so nothing reached the platform’s audit tables; it burned 79 million tokens in 30 minutes. The detector watches the ratio instead: more than 50 model calls with fewer than 5 tool calls in 15 minutes flags the agent.
Caps decay when someone edits a prompt, so they are part of the build. One CI gate fails the pipeline if a required hard stop or a cap’s marker string disappears from the agent prompts. A cost-regression gate blocks any new scheduled task that pairs an expensive model with a schedule of 120 minutes or less and synchronous inference. The fix is a smaller model, the batch lane, or a slower schedule.
A semantic cache needs identity in the key
During the work, six agents with different system prompts began issuing the same shell command in tight loops, all receiving the same upstream response ID.
The semantic cache grouped requests by a key of model, tools, maximum tokens and sampling parameters, and left the system prompt out on the theory that embedding similarity would tell agents apart. It did not. The embedding captured only about the first kilobyte of the system prompt, and agents with similar shared boilerplate cleared the 0.97 threshold. One agent’s cached response, tool calls included, was served to another, which executed them. Telemetry later ruled out replay by the provider or the agent runtime, which left the cache.
We stopped the loop first and traced it second: all scheduled jobs were paused, the platform ran chat-only for about a day, and jobs came back one at a time. Turning the semantic cache off for one model family stopped the loop, but that workaround hid the bug for four more days.
Identity belongs in the hash, not the embedding. A semantic cache key should include:
- the model, tools and sampling parameters
- a fingerprint of the system prompt (we hashed its opening bytes; hashing all of it is safer)
- the tenant and environment namespace
- the agent’s identity, explicitly
Similarity should only choose between entries that already match on all of those. A standing check also treats any response ID served to more than one agent role as an incident.
How we can help
If your AI platform’s model bill is growing faster than its usage, or you cannot say which agent or tenant is spending it, we start by measuring, then work through attribution, caching and loop caps. For attribution on a different platform, see our Baseline case study.
