TLDR Tech / Data
Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft
AI is already strong enough to handle much of data engineering, especially with declarative workflows, and strong quality gates. Use “plan mode,” fresh context resets, and external tests to manage LLM non-determinism. Substrait-like format may be more appropriate than SQL for agents to express transformations, as they convey physical operations. Data engineering may blur into a broader “data” role as agent ergonomics start to matter more than human ergonomics.
Read the full story