TLDR Tech / DevOps
Introducing Apache Spark 4.2
Apache Spark 4.2, now available in Databricks Runtime 19 Beta, introduces governed metric views as a native semantic layer, vector similarity search primitives for AI retrieval, and first-class change data capture (CDC) support through Auto CDC in Spark Declarative Pipelines. The release also brings Arrow-optimized Python UDFs enabled by default, Real-Time Mode streaming to PySpark with millisecond latency, and expanded Data Source V2 capabilities, including transaction APIs and row-level operations, all built from over 1,900 commits by 260+ contributors.
Read the full story