Hosting WebRTC on GKE without opening up the cluster

WebRTC media wants things a hardened Kubernetes cluster refuses on principle: public addresses on the nodes, host networking, and a wide range of UDP ports open to anyone. This month we put a self-hosted SFU (selective forwarding unit) into production on a private Google Kubernetes Engine (GKE) cluster for a client, and we didn’t give up any of those protections. Here is the shape that worked, and the five failures we hit on the way. Every one of them looked healthy.

The engagement itself, including why we self-hosted the media server instead of renting one, is written up in the confidential in-product calls case study. This post covers the Kubernetes and Google Cloud side.

Why the usual SFU install doesn’t fit

LiveKit is a good open-source SFU, and its Kubernetes guide is honest about its assumptions. LiveKit pods need “direct access to the network with host networking”, which limits you to one LiveKit pod per node. The guide also says LiveKit “does not support deployment to serverless and/or private clusters”, because the extra layers of NAT make them unsuitable for WebRTC traffic.

We checked both assumptions against the live cluster, and both were hard blockers:

• The nodes have no external IP addresses. That’s a deliberate choice for this cluster, and a common one on GKE.
• The namespace enforces the Pod Security Standards baseline profile (pod-security.kubernetes.io/enforce: baseline). Baseline forbids host namespaces, so a pod asking for hostNetwork: true is refused at admission.

The ways around that were all bad. We could have added a node pool with external IPs and a privileged namespace, plus a firewall rule opening thousands of UDP ports. Or we could have run the media server on a hand-built VM outside the cluster’s deployment model. Both mean deliberately weakening a posture someone built for a reason.

The shape that worked: STUNner in front of the SFU

STUNner is a Kubernetes-native TURN server built for exactly this. Its dataplane runs in ordinary pods with no host networking. Each STUNner Gateway is exposed through its own Kubernetes LoadBalancer Service, and the relay forwards media to the SFU’s internal ClusterIP. The SFU never needs a public address.

The pieces:

• A reserved static IP for each media address, so the TURN URLs handed to browsers never change when a Service is recreated.
• TURN over UDP 3478, plus TCP 443 as a fallback for networks that drop UDP. Institutional and corporate networks often do. At the time of writing the TCP fallback runs in the test environment; production is UDP-only until it is proven there.
• Short-lived TURN credentials issued by the application backend, from a secret it shares with STUNner. Browsers never hold a long-lived credential.
• Signaling through the application’s own origin. The web tier proxies the SFU’s WebSocket, so the page’s Content Security Policy keeps connect-src 'self'. WebRTC media itself isn’t governed by connect-src.

STUNner is configured through the Kubernetes Gateway API, and the objects don’t all live in one place. That matters if your platform is split into tiers with different permissions.

In our setup, no deploy identity could create any of the three at first. The application pipeline could only read Gateways, and the product tier had no rights on the Gateway API group at all. The product tier could have granted itself the rights from inside Terraform, which is exactly what the separation exists to stop. So we closed the gap with a reviewed change to the deploy role, applied by a person. If you separate platform, product and application permissions, plan that step before you start.

Five silent failures between the internet and the SFU

This is the part worth reading. Each failure below left every dashboard green and every status condition True. We found each one by checking a single link in the path, from the side a real caller is on.

1. Six of seven nodes were a black hole

Calls failed at media even with a valid SFU token and a valid relay credential. A STUN binding request from the public internet to our relay got no reply, while the same request to a public STUN server on the same port did. (Our first control test used a public STUN server on a different port. That proved UDP egress on the wrong port, and it nearly sent us the wrong way. Test on the port you’re debugging.)

From inside the cluster, a STUN request to the Service’s node port on each node answered on exactly one node: the one running the relay pod. Every node was still in the load balancer’s pool, and every node was healthy.

The Service declared externalTrafficPolicy: Cluster. For a Cluster service, Google Cloud’s health check probes kube-proxy’s health port (10256), and kube-proxy answers for the node regardless of whether the Service has a pod there. So the health check passed everywhere, and the load balancer spread callers across all the nodes. Six of the seven forwarded nothing. And because the load balancer hashes on the caller’s address, a given caller failed every time. It looked like a total outage, not a one-in-seven fault.

STUNner sets this through its own annotation on the Gateway:

metadata:
  annotations:
    stunner.l7mp.io/external-traffic-policy: local

With Local, GKE probes the Service’s own health-check node port, which only answers where a ready relay pod runs, so only that node stays in the pool. As a bonus, the relay sees the caller’s real source address. The cost: with one relay replica there is one serving node. When the pod moves, the health check follows it, with a short gap.

2. The network policy dropped media at the pod

That was a real defect, but it wasn’t the reason calls failed. Packets now reached the right node, kube-proxy translated them, and they were still dropped at the pod.

The namespace had a default-deny ingress policy (podSelector: {}). It admitted only the namespace itself, kube-system, and Google’s load-balancer health-check ranges. A TURN server’s whole job is to accept traffic from clients whose addresses nobody can know in advance, so the relay was denied by design from the moment it existed. Health checks passed, because their source ranges were allowed.

The test that misled us was “it works from inside the cluster”. Probing the public address from a pod in the same namespace succeeded, because that source was allowed. The fix is an additive, narrow policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-call-media-ingress
spec:
  podSelector:
    matchLabels:
      app: stunner        # the relay dataplane only
  policyTypes: [Ingress]
  ingress:
    - ports:
        - protocol: UDP
          port: 3478      # no "from": callers' addresses can't be known
        - protocol: TCP
          port: 443       # added later, with the TCP fallback

This isn’t an open door into the namespace. STUNner forwards only to backends named by a UDPRoute, and every allocation needs a short-lived credential. Run reachability tests from the side the real caller is on. A test from inside proves only that the path works for sources the policy already allows.

3. The TCP fallback was accepted and served nothing

We added a TURN-over-TCP 443 listener to the same Gateway. Its status read Accepted=True, Conflicted=False and ResolvedRefs=True. The operator’s log, at a verbose level, said otherwise: it couldn’t find a matching service port for the listener.

Unless mixed-protocol support is switched on, STUNner takes the LoadBalancer Service’s protocol from the first listener and skips any listener with a different protocol. It was worse than inert: the unserved listener held the whole Gateway at Programmed=False, while browsers were being told the TCP path existed.

We removed it the same day rather than leave a half-working path. The next day we shipped it properly, in the test environment first: a second Gateway with its own Service and address, attached to the same UDPRoute. The relay-to-SFU leg is UDP either way. (STUNner can put both protocols on one Service with its stunner.l7mp.io/enable-mixed-protocol-lb annotation, if your load balancer supports mixed protocols. We gave TCP its own Gateway and left the working UDP relay untouched.)

4. Valid credentials for the wrong relay, then expired ones

Two credential failures, both silent:

• The wrong relay. The backend had TURN credentials for an older managed relay as well as STUNner, and its auto-detection preferred the managed one whenever those keys existed. Every credential it issued was valid, and none of them could reach the SFU’s internal address. We now name the provider explicitly, and a test fails the build if calls are enabled without it.
• Expiry mid-call. STUNner checks a credential’s expiry on every TURN Refresh, not only when the allocation is first made, and a client must refresh an allocation within ten minutes. A credential issued once at join therefore cut off a participant’s media around the time it expired, about an hour in. The SFU client’s own reconnect reused the expired credential. The fix renews the relay credential ten minutes before expiry and restarts ICE through the new one.

5. A quota, not capacity, blocked the rollout

The relay dataplane and the SFU both shipped with their charts’ default limits of 2 CPU each, into a namespace whose CPU-limit quota was already almost full. The rollout failed as a cascade: unrelated web and worker updates blocked, a five-minute CronJob stopped scheduling, and Helm waited ten minutes and rolled back.

We raised the quota to get unblocked, then measured. Over 14 hours in the test environment, including a real three-person video call, the relay peaked at 70 millicores and the SFU at 98. We sized them at 100m requests with 500m and 1 CPU limits, and lowered the quota to match. Size media components from a measured call, not chart defaults. At small scale they are much lighter than the defaults assume.

Make the path prove itself

Since none of the failures above showed up on their own, we added two checks and are wiring in a third.

• A probe every five minutes. It performs a real TURN allocation against every advertised relay address, using credentials made the same way the backend makes them. It checks the 401 challenge, the authenticated allocation and the release, then asks the SFU to list its rooms. A secret that differs between backend and relay fails here, not on a call.
• A scheduled synthetic call. Three real browsers join one call, and every assertion reads getStats(). Each transport is pinned and tested on its own, because a browser falls back silently, and an unpinned pass proves only one path.
• Relay metrics in Google Cloud Managed Service for Prometheus (in progress). STUNner’s dataplane can serve allocations, sessions and bytes relayed. Its pods declare a metrics port, but the endpoint is switched off by default, so nothing answers on it. And the default-deny policy would also block the managed collector in gmp-system. Either way the panel reads “no data”, which looks exactly like a relay with nothing to report. The collector gets its own network policy. We deliberately didn’t add the metrics port to the media policy: that one has no from block, so it would expose metrics to the internet. Bytes relayed will also be the cleanest way to attribute this product’s network egress cost, which the billing export couldn’t break down by product.

The checklist

If you are putting WebRTC media on GKE:

1. Don’t loosen the cluster for the SFU. Put a TURN relay that runs in ordinary pods in front of it, and keep the SFU on a ClusterIP.
2. Reserve the media address, and give UDP and TCP their own Services unless you have verified mixed-protocol load balancing end to end.
3. Set the relay’s external traffic policy to Local.
4. Write a narrow network policy for the relay ports, and a separate one for your metrics collector.
5. Issue short-lived TURN credentials, renew them mid-call, and name the relay provider explicitly.
6. Size from a measured call, and check the namespace quota before the first rollout.
7. Test from outside the cluster, pin each transport, and probe the whole path on a schedule.

Most of this isn’t specific to Google Cloud. The default-deny policy, the credential handling and the quota would bite on any managed Kubernetes, and the health-check behavior under externalTrafficPolicy: Cluster is worth checking on yours. GKE is simply where we measured them. If you are planning real-time media on Kubernetes and want a second pair of eyes, talk to us.