How to achieve truly serverless GPUs (20 minute read)

2026-05-13

Inference workloads are more variable and less predictable than training workloads. This makes them a natural fit for serverless computing. However, serverless computing only works if new replicas can be spun up as fast as demand changes. This article looks at how Modal took AI inference server scaling from multiple kiloseconds to just tens of seconds.