Client: Shovel Solutions, a regional dirt- and aggregate-hauling marketplace. Job creators pay for delivered loads; material pits, independent truckers, and fleets get paid. A job flows from creation through pit selection, trucker claiming, loading, GPS-tracked driving, and drop-off to payment, with ACH money movement through Stripe, Dwolla, Plaid, and QuickBooks.
Hat Boy Software was engaged to assess the live system and design its next generation. This is an assessment-and-architecture engagement: the findings below are from the production system, and the remedies are in the target design and migration plan we delivered.
What we found
The payments layer. Our conclusion was blunt — treat it as breached — and the evidence was concrete:
- Authorization barely existed on the money paths. About 1 of ~80 backend functions verified the caller’s identity. Withdrawal, bank-linking, and payment endpoints were effectively open, and the production database rules resolved to allow read, write for anyone.
- Live production secrets were committed to source control — payment, bank-linking, and email keys, all in git history.
- Multi-factor authentication was a facade. The confirm endpoint always returned “confirmed”; the send-code endpoint sent nothing.
- Payments were not idempotent. No transfer carried an idempotency key, so a network retry could send a duplicate ACH transfer. Meanwhile the “money-out” feature wrote a bookkeeping record and returned a hard-coded balance without ever moving money.
- Bank account and routing numbers were stored and emailed in plain text.
The platform. Around 80 serverless functions, three separate mobile apps, and a document database with no relational integrity. The same data existed in three live generations at once across 88 collections, including test data in production. Repair scripts had become a way of working. Backend test coverage was roughly zero.
The root cause was one design choice: a real-time, denormalized, trigger-driven system with no transactions and no idempotency.
What we designed
Phase 0 — treat it as a live incident. Before any rebuild: lock down the open database rules, run a breach review, rotate every committed secret and purge history, remove test backdoors, gate unauthenticated data exports, and kill the fake MFA.
A money layer that is secure, compliant, and correct:
- One authenticated API boundary in front of every money endpoint.
- Real MFA with step-up re-authentication before payouts and bank-account changes.
- Secrets in a vault, injected at runtime — never in the repository.
- PostgreSQL row-level security, so tenant isolation holds even if the API is bypassed.
- An append-only, double-entry ledger with derived balances, replacing the mutable balance field that drifted.
- Idempotency keys on every external call, de-duplicated webhooks, and a saga with compensating actions for the payout flow.
- Tokenized bank and personal data, with Know-Your-Customer capture restored.
- A reconciliation mart across processor, bank rail, accounting, and ledger.
A platform that stops fighting its own data: a single modular API on Azure Container Apps behind API Management; PostgreSQL with PostGIS as the system of record (transactions end the drift, and row locking ends over-claimed loads); SignalR and Service Bus for real-time location and events, with GPS pings kept off the hot database path; and one React Native client for iOS, Android, and web in place of three.
A migration that de-risks itself: a strangler-fig cutover, with domains moved one at a time behind the gateway using feature-flagged, percentage-based rollout — low-risk domains first, money last.
Advice sized to the client
The client assumed modernizing meant changing clouds. At this platform’s scale — a runtime bill around $16 a month — cloud cost was not the deciding factor, and the fixes were fully achievable on the existing cloud. We separated “modernize the architecture” from “change providers” and gave the client the trade study to decide the second deliberately. The right answer for the client, not the biggest project for us.
Results of the design
The six critical payment risks, and how the target design closes each one:
| Payment risk in the live system | How the target design closes it |
|---|---|
| World-open financial data | Row-level security plus a single authenticated API boundary |
| Committed production keys | Vault-managed secrets injected at runtime |
| Fake MFA on money movement | Real MFA and step-up re-authentication |
| Duplicate ACH on retry | Idempotency keys and event-ID webhook de-duplication |
| “Money-out” that moved no money | Real, idempotent ACH backed by a double-entry ledger |
| Bank data stored and emailed in plain text | Tokenization, with the payment processors as vault of record; KYC restored |
And on the platform side:
| Platform liability | Resolution in the target design |
|---|---|
| Three live generations of the same data | One relational system of record with transactions and constraints |
| Repair scripts as a way of working | Integrity enforced by the database; idempotent workers; nightly reconciliation |
| ~80 functions and three mobile apps | One modular API and worker; one React Native client |
| ~0% tests, no Infrastructure-as-Code | CI test gates, structured observability, Infrastructure-as-Code |
Shovel Solutions is named with its permission. The environment, findings, and results are drawn from a real Hat Boy Software engagement.