TLDR Tech / AI
Can Kafka Support Elastic Partitioning?
Kafka could support elastic partitioning, dynamically splitting and merging key ranges to scale ingestion and processing while balancing load. The design could preserve per-key ordering and Kafka's batch-oriented efficiency, but it would complicate consumer state locality, repartitioning, replay tracking, and recovery. Eliminating partitions entirely would sacrifice important batching and storage benefits.
Read the full story