TLDR Tech / DevOps
DigitalOcean Gradient™ AI GPU Droplets Optimized for Inference: Increasing Throughput at Lower the Cost
DigitalOcean launched its Inference Optimized Image that achieved 143% higher throughput (2,000 vs. 823 tokens/second) and 75% lower costs ($1.47 vs. $5.80 per million tokens) while running Meta's Llama 3.3 70B on just 2 H100 GPUs instead of the typical 4, by combining speculative decoding, FP8 quantization, FlashAttention-3, and other optimizations into a pre-configured deployable package.
Read the full story