TLDR Tech / Dev
From SSH to REST: Modernizing Slack's EMR Data Pipelines with Security in Mind
Slack has modernized its data infrastructure by migrating over 700 data pipeline jobs from a legacy SSH-based submission model to a REST-based architecture. The previous reliance on direct SSH access to AWS EMR clusters created security vulnerabilities, operational inefficiencies like 'zombie' processes, and technical debt that hindered infrastructure upgrades. By utilizing a custom gateway called Quarry and the YARN Distributed Shell feature, the engineering team successfully transitioned workloads into managed containers with server-side lifecycle management.
Read the full story