TLDR Tech / Dev
How OpenAI delivers Low Latency Voice AI for 900M Users
OpenAI serves WebRTC voice AI to 900M weekly users by splitting the stack into a stateless edge relay for packet routing and a stateful transceiver that owns the heavy WebRTC state, sidestepping the port-exhaustion and state-stickiness problems WebRTC hits on Kubernetes. The key method is encoding routing metadata into the ICE ufrag, a field already exchanged during setup, so the relay routes the first packet by reading the initial STUN request rather than doing a database lookup.
Read the full story