Director of Data Engineering (Allconnect)

Role Overview

Join Red Ventures as the Director of Data Engineering to lead and scale the team responsible for the company’s data infrastructure. You’ll set the multi-year technical direction, partner closely with business and technical stakeholders, and manage data engineering leaders and managers.

Responsibilities

• Lead the data engineering team building and scaling Red Ventures’ data infrastructure across the full data lifecycle
• Define and communicate multi-year technical strategy for Data Engineering to both technical and business audiences
• Serve as the primary technical liaison to business leadership
• Manage and develop data engineering managers; raise the bar for coaching, hiring, and team execution
• Ensure delivery of high-quality, scalable solutions and drive operational excellence in data pipelines

Required Qualifications

• Strong programming background in Python and/or Scala, plus SQL fluency on large-scale datasets
• Proven ability to communicate technical strategy and tradeoffs clearly to non-technical executives
• Deep hands-on experience building large-scale Spark applications for batch and streaming workloads
• Experience with ANSI SQL relational databases (Oracle, SQL Server, PostgreSQL, or MySQL)
• Experience with lakehouse platforms (e.g., Databricks) alongside traditional Spark pipelines
• Strong grounding in distributed systems and distributed data processing
• Hands-on experience with AWS data services (e.g., Kinesis, S3, Lambda, DynamoDB)
• Experience leading engineering teams in an Agile environment, coaching both engineers and managers
• Hands-on experience with Spark Structured Streaming and SparkSQL (including broadcast joins)
• Deep curiosity about end-to-end data production and the ability to troubleshoot pipeline issues
• 8–10 years of data/software engineering experience, including 5+ years managing engineering teams and at least 1 year managing managers

Nice-to-Haves

• Expertise with Spark RDD and DataFrame/Dataset APIs, with emphasis on DataFrames for querying and manipulation