Client: an early-stage intelligence-services platform whose users communicate through a confidential channel. The application runs on a hardened, private Kubernetes cluster behind a strict Content Security Policy, and confidentiality is the product’s premise.
The challenge
The pilot needed participants to talk live. Until then, a “call” was a link to an outside meeting service, so the most sensitive part of the work happened somewhere else.
Embedding a commercial meeting SDK was the obvious route, and it was the wrong one. The page’s policy allows network connections to its own origin and nothing else (connect-src 'self'). That one line is what makes “this page can only talk to its own backend” a fact rather than a promise. A meeting SDK would have needed outside connection hosts, a script host, WebAssembly, and, for HD video, cross-origin isolation. Cross-origin isolation would have blanked the embedded e-signature ceremony and the policy documents on the same page.
There were three more constraints:
- Identity. No participant may learn another’s email address, name, or record ID through the call.
- Who may join. Only the people already authorized for that conversation, decided on the server.
- No new exposure. The cluster’s nodes have no public IP addresses, and its pod-security policy forbids host networking. Both rules stay.
What we did
Day one: a peer-to-peer mesh that changed no security policy
WebRTC media sits outside connect-src: RTCPeerConnection carries DTLS-SRTP, which that directive does not govern. So we built the call in-house and left the security headers untouched.
- One server endpoint decides everything. It handles join, signalling, polling, and leave. It works out who may join from records that already exist, so the data model did not change.
- Media is forced through a TURN relay. Participants never exchange their own network addresses. Each one appears to the others under a fresh, random identity per join.
- Signalling fans out over the message bus, because the backend runs two replicas. With in-memory signalling alone, half of every handshake would go missing.
Measuring before fixing
The first real three-way call worked, but one participant’s audio was choppy, and only for the two people listening. Earlier in the day, guessing had cost us time: three plausible causes had each been true of the layer below and none of them was the bug. So we built a measurement into the call page instead of guessing again.
The page reduces the browser’s getStats() to three numbers, each a delta rather than a lifetime total. The key one is concealment: the share of audio the browser had to invent to cover gaps. It is not a proxy for choppiness; it is choppiness, counted.
The numbers were decisive. Both listeners saw 8.5% concealment, 35–42 ms of jitter, and 0.0% packet loss on that participant’s audio, while the participant’s own incoming audio was clean. Jitter with zero loss means packets were being queued, not dropped: a saturated uplink.
That is the structural cost of a mesh. Each participant uploads one copy of their media per peer, and each copy runs its own congestion controller, so the copies compete for the same link without knowing about each other. We capped video at 350 kbps per peer and gave audio priority. That eased the symptom, but the cost still grew with the size of the room.
Day two: a self-hosted SFU, without weakening the cluster
We wrote up the decision record and moved the call to a selective forwarding unit (SFU). With an SFU, each participant uploads one copy, whatever the room size. That also makes simulcast and server-side recording possible.
Self-hosted, not managed. Two managed SFU services were cheaper and easier to run. We rejected them for one stated reason. A TURN relay forwards encrypted packets it cannot read; any SFU decrypts and re-encrypts. For a product built on a confidential channel, that ability should sit inside the client’s own cluster, not with a new third party. The decision record says plainly how narrow this argument is: the edge provider already terminates TLS for the rest of the application. The aim was not to extend its reach to the one kind of data it could not yet see.
LiveKit behind STUNner, not LiveKit on host networking. We checked LiveKit’s standard deployment against the live cluster and found two hard blockers. The nodes have no external IP addresses, and the pod-security policy refuses host networking. STUNner runs a media server in ordinary pods on a private cluster, so we put LiveKit behind it rather than open a privileged namespace and a 10,000-port UDP range.
The security policy still did not move. We proxied the SFU’s signalling WebSocket through the application’s own origin, the same way the sign-in endpoints already were. connect-src 'self' held exactly as written, which was the main bet in the decision record.
Making it observable, then switching it on
- A probe every five minutes. It performs a real TURN allocation against each relay address, using the same credentials the backend issues, and asks the SFU to list its rooms. If the backend’s secret and the relay’s secret ever differ, the probe reports it before a participant does.
- A TCP/443 fallback for networks that block UDP, which are common on institutional networks.
- A synthetic three-browser call runs on a schedule. Three real browsers sign in as dedicated test accounts, pass every gate a person would, and hold a call through the relay and the SFU. Every assertion reads
getStats(): audio arriving, a relayed path, video present only while cameras are open. Each transport is pinned and proven separately, because a browser falls back silently, and a pass with nothing pinned proves only one path. Against the test environment, the result was 19 of 19 checks passing over UDP, over TCP/443, and unpinned. - Mid-call credential renewal, so media survives past the first hour. The relay checks expiry on every refresh, not only when a call first connects.
- Production behind a switch. Production uses a per-environment switch that turns on only together with the SFU and the relay. A test fails the build if one is enabled without the others.
Results
| Outcome | Result |
|---|---|
| Calling | Built from nothing to three-way audio and video in production in two days |
| Uplink | One upload per participant, whatever the room size, instead of one per peer |
| Security policy | No change to the Content Security Policy, the pod-security policy, or node exposure |
| Media confidentiality | No third party in the media path; the SFU runs inside the client’s own cluster |
| Participant privacy | A random identity per join; no email, name, or record ID crosses the call |
| Restrictive networks | TURN over UDP, with a TCP/443 fallback, each proven separately in the test environment |
| Detection | Five-minute relay and SFU probe, plus a scheduled synthetic three-browser call |
| Engineering discipline | Fixes shipped with tests, and key protections were mutation-checked, so a test that could not fail was caught |
The honest footnote
Four failures in the rollout all looked healthy until we checked a specific link. A TCP listener was accepted and served nothing. A resource quota, not capacity, blocked the deployment. The half of the backend that issues call tokens was missing its keys while the SFU reported Ready. And with one setting left at its default, the backend issued credentials for the wrong relay: every credential was valid, and no call could connect. None of these turned anything red.
The first real test also found that no handshake had ever crossed. The browser code and the server each had passing tests, but nothing tested the contract between them. We now test that seam with a transport that enforces the server’s rules, not one that just records calls. The broader lesson is that for real-time media, “healthy” is not evidence. That is why the probe and the synthetic call exist.
The client's identity is anonymized. The environment, findings, and results are drawn from a real Hat Boy Software engagement.