About the role

Role Overview

Databricks Data Engineer on Guidehouse’s AI & Data team, supporting client engagements focused on scalable data pipelines, data transformation, platform modernization, and advanced analytics/AI solution delivery.

Responsibilities

Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake
Develop and support batch and streaming workflows (ingestion, transformation, validation, publishing of curated data products)
Write and optimize Python/PySpark/SQL for data processing, orchestration, and performance tuning
Work with large-scale datasets using Databricks/Spark/Delta Lake, Unity Catalog, and cloud storage
Troubleshoot pipeline failures, performance issues, data quality problems, and bottlenecks
Translate business/technical requirements into data engineering designs and orchestration workflows
Perform data validation, quality checks, code reviews, and issue resolution to ensure reliable data products
Collaborate with solution architects, data scientists, analysts, DevOps engineers, and client stakeholders
Communicate delivery impacts and engineering considerations to technical and non-technical audiences
Document pipelines, datasets, models, workflows, and architectural decisions
Follow data governance, security, lineage, and compliance standards within the Databricks ecosystem

Requirements

Bachelor’s degree in computer science, engineering, mathematics, statistics, or a related field
3–8 years of experience in data engineering, data architecture, or cloud data platform implementation
Strong experience with Python, PySpark, and SQL for transformation and pipeline development
Experience with Databricks, Spark, Delta Lake, Unity Catalog (or similar cloud-native platforms)
Experience building batch/streaming ETL/ELT pipelines and reusable data assets
Knowledge of data modeling, data warehousing, data quality validation, and large-scale processing
Ability to troubleshoot issues, communicate recommendations clearly, and deliver in team-based environments
Experience applying governance, access control, lineage, and security in cloud data platforms

Nice to Have

2+ years hands-on experience with the Databricks platform
Additional Unity Catalog experience for governance, schema design, modeling standards, access controls, lineage, and secure enterprise data management

About Guidehouse

Guidehouse is a consulting firm that delivers data, AI, and technology solutions for clients. The role sits within their AI & Data team, supporting client projects that include large-scale data pipelines, analytics, and platform modernization.