TLDR Tech / AI
Two different tricks for fast LLM inference
Anthropic and OpenAI have unveiled new 'fast modes' for LLM inference, offering differing performance enhancements. OpenAI's model speeds exceed 1,000 tokens per second using Cerebras chips, but rely on a less capable model, while Anthropic's fast mode supports real models with up to 2.5x speed through low-batch-size inference. Despite OpenAI's technical achievement, the utility of faster but less capable inference remains debatable, focusing more on OpenAI's exploration of Cerebras' potential.
Read the full story