Data engineering as a profession continues to grow in importance, and as more companies push towards revamping legacy systems and overhauling their data strategy, we’ve seen the need for this talent increase dramatically over the past decade. In the past, certain job responsibilities like data cleaning or architecture, among many others, may have been assigned to data scientists. However, we’ve seen that as teams grow, there is a growing importance placed on hiring more data engineers, since these professionals are experts at building data infrastructure and pipelines, which is not typically a data scientist’s primary area of expertise. 

How Do You Identify Data Engineering Talent?

In our recent 2021 Data Engineer Salary Report, we found that data engineers typically hold a Bachelor’s or Master’s degree in Computer Science, Information Systems, or Computer Engineering, with the most common degree being a Master’s degree (62% of the sample), followed by Bachelor’s degrees (32%), and PhDs were rare (5%). By comparison, PhDs are a lot more common among data scientists, who will also have more math and statistics education in their background to focus more on data modeling and analysis.

Data engineers often have a wider range of technical skills and will often work alongside data scientists to prepare data for analysis and put data products into production. This can include everything from building data pipelines and ETL or ELT, having experience with complex distributed computing, the deployment of data science models, hybrid cloud architecture, and more. Because of the wide variety of their job responsibilities, and the plethora of technical tools that a data engineer may need to employ to accomplish them, it’s not uncommon to see a very extensive tool section on a data engineer resume or job description. 

There is no singular tool that makes someone a data engineer, and so we find that most data engineers will have a very broad set of experience with many tools. Their skillset may include programming languages, cloud computing tools, relational and NoSQL databases, and many other Big Data technologies.

The Rise of Hybrid Roles in Data Engineering

As the needs of data teams continue to shift with team growth and new technologies, we’ve seen an increasing prevalence of hybrid or blended-type roles, that may mix skillsets from data engineering, cloud engineering, data science, analytics, and other areas (like DevOps). This may include professionals like Machine Learning Engineers, who will have significant expertise applying machine learning concepts, as well as deploying, scaling, monitoring, and continually improving algorithms. 

We’ve also seen some teams with Data Science Engineers who work to build scalable, user-friendly systems for internal customers, as well as Analytics Engineers, who will typically have more analytical modeling than a typical data engineer, and a greater focus on how data insights impact business goals (as opposed to focusing mostly on data infrastructure). As the needs of data teams continue to evolve, we’ll likely continue to see more shifting with regards to hybrid-type roles.

Hiring the Right Data Engineering Talent for Your Team’s Needs

Finding the right talent in a field like data engineering, where there are so many tools and technologies that may be required for a role on your team, can be very challenging. From what we’ve seen during the talent search and hiring process for many employers, there are a few things the hiring team can do that can streamline a search and make it more effective:

1. Consider which tools are required for success day one vs. what can be trained

2. Prioritize hard requirements vs. “nice to haves”

3. Be wary of using keyword-only talent search approaches

4. Evaluate requirements and choose a salary range 

5. Be open to junior hires that can grow into a role if salary is a concern

6. Provide training if the required skill set is more niche and decreases your potential talent pool

7. Identify strong benchmark candidates to guide the recruiting team

8. Give access to the hiring manager so that their specific needs can shape the search

9. Provide timely feedback on surfaced candidates to zero in on the ideal talent profile

10. Consider location: are remote data engineers a possibility?

While the conversation around remote working is still evolving, we do know from our own research that the vast majority of data engineers and data scientists prefer some sort of hybrid work arrangement. From our survey sample, when asked about the ideal number of days in the office, 34%, would prefer to be in the office two days a week, followed by 24% of the sample who preferred 3 days. Only a very small percentage of our sample said they’d prefer to be in the office four or five days a week (6% and 2% respectively).

These work-from-home (WFH) preferences can vary based on a variety of factors, including what stage someone is at in their career and whether they find distanced collaboration methods to be effective. We’ve also seen some employers adjust their data engineering searches to allow for 100% remote staff, either in an effort to access a wider talent pool or, in some cases, to better compete with their competitors’ recruiting strategy.

Reporting Structure & Organization of Data Engineering Teams Varies

During our discussions with leaders on data engineering and data science teams, we’re often consulted about typical reporting structures and team organization for these types of roles. What we’ve found is that this can vary widely from organization to organization, but we wanted to share a few of our observations from working with many different teams and employers in this space. 

In data engineering, we’ve observed that middle or lower-level management roles are not as common as they might be in data science and analytics. We often speak with principal data engineers without any direct reports, which means they are very experienced (and often highly paid) individual contributors who may be directing some strategy but not necessarily managing their own team. Regarding reporting structure, a team might have a Data Architect reporting to a VP of Analytics or Data Science, but we’ve also seen some teams where data engineering is classified under IT. In many cases, the ideal reporting structure may depend on job responsibilities as well as the structure of the rest of the organization, since we’ve also found there are teams where some data engineers are under both the data science/analytics and IT teams. 

How are Companies Approaching Building Their Data Engineering Teams?

So how are data teams approaching their hiring needs for data engineers? When working with companies who are interested in building out their teams, where data processes are not built out or well-developed yet, this often requires a senior data engineering hire first. A data engineering leader can help to establish strategy and develop a roadmap, as well assist in hiring the right team to support their strategy. We’ve been working with a number of companies recently who are hiring multiple data engineers simultaneously, as they’re building out their pipelines and or migrating to the cloud.

Typically, data scientist hires would come after the data engineering team has been established to support their work, and a rough guideline for team size that we’ve often heard is that there are three data engineers needed for every data scientist. Even in cases where there is an offshore data engineering team, we’d recommend having some onshore talent who can act as the liaison or lead. 

This summary has shared a variety of our insights on building a data engineering team, but our full industry reports offer a plethora of data and insights that are crucial to managing talent strategy, including WFH, industry trends, salaries examined by various factors, attrition insights, and more. We’re able to provide this data to the community because of our unique position between both employers and data professionals, which gives us an exceptional vantage point on market trends. To access even more information to aid your battle in the talent war, you can download the full reports here.

Natalie Rubenstein, Executive Recruiter at Burtch Works, has significant experience in the technical recruiting sector with a strong background in business development and long-term relationship building. She has significant experience identifying contract, contract-to-permanent, and permanent positions in the areas of IT, Business Applications, Telecommunications and Engineering, and is a leader on our data engineering and technology recruiting team.