Cribl sponsored this post.
Data engineering started as a support role for data scientists in the Big Data era. While data scientists are skilled at statistical analysis, they aren’t computer scientists. Manipulating and refining large datasets and dealing with complex infrastructure is outside the skill set of data scientists. So, the data engineering role developed to fill this emerging need.
The popularity of the role has since exploded. According to LinkedIn, demand for data engineers grew 33% in 2020. This exceeded the annual growth rate for cybersecurity specialists, back end engineers, and cloud engineers.
Over time, the responsibilities of data engineers have expanded beyond supporting data scientists. Now, data engineers are expected to build and support data pipelines across the enterprise. Those pipelines may consist of batch or streaming data infrastructure, as well as data quality and integration tooling — but they aren’t limited to that.
The Evolving Role and Tools of the Data Engineer
Data engineers can be tasked with a variety of data-centric tasks. These may include data infrastructure configuration and maintenance, curating data sets for downstream consumption, collaborating with data scientists or researchers on their data needs, and improving data quality.
They use several tools over the course of their day. Python and Spark are common, as is SQL and a variety of relational and nonrelational databases. Increasingly, serverless and container technologies will find use, speeding deployment of new data pipelines. However, data engineering tools are maturing beyond their artisanal roots. There are multiple vendors offering tools for data engineers. Some go as far as fully integrated data engineering environments. Others offer lighter weight managed environments for things like Apache Spark, leaving the data engineer to develop the solution as they see fit.
In many ways, data engineers are morphing into a type of glue role, like systems administrators. It’s common to see overworked data engineers responsible for everything from data warehousing, to cloud migration, to supporting data scientists. This makes data engineers hard to find and harder to keep. Companies must refine their concept of the data engineer’s role, supporting those staffers with appropriate management and funding. Much like data scientists frustrated in roles where they’re stuck creating canned reports, data engineers unable to break out of routine tasks will also be unsatisfied.
DataOps and Emerging “Ops” Drive Data and Analytics Productivity
Data engineering is a key facet of DataOps, but it’s not the only element. Like DevOps, DataOps represents a cultural shift for how organizations use and organize around data. The objective of DataOps is more effective collaboration between data producers and consumers. Better collaboration drives higher quality data delivered faster, for more use cases. More roles will develop as part of DataOps — including data product managers, who will be responsible for the successful delivery of data products in the enterprise.
Coping with data challenges has spurred other “ops” concepts and practices. In addition to DataOps, companies aiming to improve success rates with machine learning and artificial intelligence are exploring things like MLOps and AIPlatformOps. The intention of each of these practices is to accelerate the delivery of data models for various uses. There isn’t much structure behind any of these concepts and practices yet, but that doesn’t mean you can’t use them — even if it’s only to get your organization thinking differently about how it uses and collaborates around data.
Regardless of how the variety of emerging “ops” trends develop, it looks certain that data engineers will be at the center of each of them.
We’re also seeing this in the observability space at Cribl. There’s a massive gap between data teams, application developers, and infrastructure and operations teams. We believe the concept of ObservabilityOps will emerge as a reaction to collaboration challenges between application developers and infrastructure and operations teams.
Data Engineers Must Take a Customer-First Mindset
Despite their humble beginnings as data janitors for data scientists, data engineers are emerging as an essential role in data and analytics programs. Data engineers will further evolve into data pipeline engineers, responsible for end to end data delivery — whether we’re talking about machine learning uses, self-service analytics, or operational analytics in the context of a single application. Data scientists will be another customer that data engineers, or data pipeline engineers, serve.
This highlights where data engineering is going as a practice and as a role. As more data consumers emerge throughout organizations, it’s data engineering that will enable those consumers. People in this role need robust technical skills, but they’ll also need a strong customer-first mindset. That’s the only way data engineers, and the companies they work for, will be successful.
Feature image via Pixabay.
