From plumber to platform builder
The first generation of data engineers were essentially ETL developers: extract data from here, transform it, load it over there.
The job was largely reactive:
- Business stakeholders asked for a report; engineers built a pipeline to feed it.
- Repeat indefinitely, until someone senior asked why the data team was always the bottleneck.
What changed in the early 2020s was the emergence of the data platform concept.
Rather than building one-off pipelines for every request, data engineers started building infrastructure that other teams, analytics, data science, and product, could use themselves.
The job became less about moving data and more about building the system that lets everyone else move data safely, reliably, and at scale.
That is a very different job. And it requires a very different kind of hire…
The modern stack reshaped the role
The rise of cloud-native data warehouses, Snowflake, BigQuery, Redshift, combined with tools like dbt, Airflow, and Fivetran, fundamentally changed what data engineers spend their time on.
A lot of the old ETL grunt work was abstracted away. This created space, and expectation, for data engineers to think more like software engineers.
Today, a strong data engineer:
- Writes modular, tested, version-controlled transformation code
- Applies CI/CD and code review practices to data systems
- Manages infrastructure as code rather than a collection of manually configured services
- Treats data pipelines with the same engineering rigor as production software
For tech leaders, this means the hiring bar has moved. A data engineer who cannot work within a modern software engineering workflow is increasingly a liability, not an asset.
AI is the biggest forcing function yet
The most significant shift currently underway is the collision of data engineering with AI and ML infrastructure. Building and operating LLM-powered products turns out to require exactly the kind of work data engineers do, but applied to new primitives.
Retrieval-augmented generation (RAG) pipelines, for instance, require clean, chunked, embedded documents stored in vector databases with fast retrieval. Evaluation and observability for AI models require tracking inputs, outputs, and model behavior over time, which is fundamentally a data problem.
Real-time is no longer a nice-to-have
There is a structural shift away from batch processing toward streaming architectures. Products that personalize in real time, detect fraud as it happens, or update dashboards instantly all require data pipelines that run continuously rather than on a schedule.
Tools like Kafka, Flink, and cloud-native streaming services have matured to the point where streaming-first design is increasingly the default for new systems, not a specialist add-on.
This raises the bar significantly. Debugging a failed batch job at 3am is unpleasant. Debugging a streaming pipeline where subtle schema drift is silently corrupting downstream AI models in real time is a genuinely different category of problem.
Data engineers working in this space have had to develop much stronger operational instincts, and for tech leaders, that skillset is worth paying close attention to when hiring.
Data contracts and trust
One underappreciated shift is the growing emphasis on data contracts: formal agreements between the teams producing data and the teams consuming it. This emerged out of a familiar pain point.
A producer team changes a field name or removes a column, and three downstream pipelines silently break, often discovered only when someone notices the revenue numbers look wrong in a board deck.
Data engineers are increasingly responsible for:
- Designing and enforcing data contracts across teams
- Building data quality checks directly into pipelines
- Implementing lineage tooling so that when something breaks, the blast radius is understood immediately
This is partly a cultural shift, treating data as a product with consumers who have expectations, and partly a technical one. For tech leaders, it is worth asking whether your current data engineering function has the mandate and tooling to do this work properly.
Where the role is heading
The trajectory points toward data engineers becoming infrastructure owners for AI systems as much as for analytics. The skills that matter most in this new phase include:
- Understanding how large language models consume data and depend on data quality
- Building and maintaining feature pipelines that feed inference endpoints at scale
- Versioning, storing, and refreshing embeddings on a schedule that matches model update cycles
- Monitoring and evaluating AI system behavior continuously in production
It also means a continued push toward self-serve infrastructure, building internal platforms that reduce the bottleneck of data engineers being in the critical path of every analysis or experiment.
Want to go deeper? Join us at Agentic AI Summit New York on June 4
Join 500+ engineering peers shaping the agentic AI landscape, from foundational models to the application layer. NY Tech Week’s largest assembly of applied builders.
Unlock the following:
- A clear view of what’s working now: agent workflows that are transparent and interpretable, built for smarter debugging and more reliable systems
- Benchmarks against live architectures: see what is actually working across inference, evaluation, and continuous fine-tuning, from the people running it in production
- Connections that accelerate progress: peers, partners, and innovators building industry-ready applied AI, all in one room for one day
No slides dressed up as insights. Just the people solving the hardest parts of this problem, talking honestly about how they do it.
