TLDR Tech / Practical
Operating Data Processing Systems
This chapter from the Google SRE book addresses the operational difficulties of managing data processing workflows, distinguishing between batch and continuous processing models. It highlights typical issues with batch pipelines, including brittleness, uneven workload distribution, stuck data segments, and the thundering herd effect. To handle these problems at scale, Google employs its Workflow system, which uses a leader-follower architecture and system prevalence for dependable, continuous processing with strong accuracy.
Read the full story