Client: Baseline, a market- and geopolitical-intelligence platform that applies the discipline of journalism to the information decision makers rely on. Baseline turns news and public data into intelligence an executive can defend: it breaks each article into atomic, sourced claims it calls Microfacts™, scores each claim for credibility and corroboration, clusters them into signals with lineage back to every source, and writes briefings on top. It runs on Google Cloud, with Gemini models on Vertex AI and a BigQuery lakehouse.
The challenge
Baseline’s pricing works because the cost of tracking a company, person or topic is shared by every customer who watches it. That only works if cost can be traced back to customers. It couldn’t:
- About 100% of real spend was unrecorded. In one month the AI provider billed about $86, while the platform’s own telemetry had logged roughly $1.70 all-time in development, and nothing, ever, in staging or production.
- About 73% of the cloud bill could not be grouped by product. Compute was the largest line and carried no attribution.
- One tenant’s runaway workload could use up the whole daily budget and stop scheduling for every other customer.
- Reliability hazards made it worse: liveness probes killing workers mid-computation, a message-queue deadlock, and infrastructure lock races failing unrelated pipelines.
What we did
Made every dollar attributable
- Moved cost recording to the provider layer, so any call path that forgets to record fails loudly instead of silently logging nothing.
- Fixed attribution at the source. Two live defects were billing spend to the wrong tenant or to no one. Both now use explicit, leak-proof tenant scoping.
- Built a unit-economics view that breaks the bill down per board, per monitored subject and per user, so pricing decisions start from measured cost rather than estimates.
Capped spend at every layer
- Spend caps per collection run, per tenant per day, and org-wide, each sized to contain a different failure, so a runaway tenant can no longer stop the shared scheduler. We wrote up the design in Spend caps at every layer, and partition pruning as a requirement.
- Partition pruning made a requirement on the lakehouse’s large tables, so a query that would scan everything is refused rather than billed.
- Cut idle infrastructure. We stopped writes to a lakehouse tier that billed per active table in environments that had never had a tenant: about 75% of that bill, with no loss of function.
Made the platform dependable
- Restored reliability. A file-based worker heartbeat ended probe kills, a dedicated queue drain fixed the deadlock, lock timeouts stopped false failures, and a corrected CI capacity fallback ended org-wide build outages.
- Keyless delivery. CI deploys through workload identity federation, with no long-lived cloud credentials to leak or rotate.
- A weekly release train. Production ships on a fixed weekly day, verified in the running pods rather than by image tag, with emergency hotfixes as the only exception.
- Guardrails that catch the quiet failures: deletion protection on the production database, a monitor for workloads pinned to container images that no longer exist, and platform alerts routed into a support queue instead of an inbox.
Kept the output honest
Baseline’s product is trust, so its briefings have to show their working. Briefings now cite each development they draw on, say plainly when a citation cannot be traced, and report a quiet day as quiet rather than padding it.
Results
| Outcome | Result |
|---|---|
| LLM cost observability | From ~0% captured to full provider-layer capture with tenant attribution |
| Lakehouse spend | ~75% eliminated |
| Attribution | Cost per board, subject and user; per-tenant margin now computable |
| Spend control | Caps per run, per tenant and org-wide; unpruned scans refused |
| Reliability | Platform-wide outage class closed; probe-kill, deadlock and CI-outage classes fixed |
Baseline gained what its pricing model needed and never had: the ability to see, per customer and per monitored subject, what it costs to serve them, with a lower, cleaner bill along the way.
Baseline is named with its permission. The environment, findings, and results are drawn from a real Hat Boy Software engagement; figures are representative and rounded.