Databricks Data Engineer

Role Overview

Databricks Data Engineer to join Guidehouse’s AI & Data team. You will design, build, and maintain scalable, secure data pipelines and curated data products for client projects, using Databricks and related cloud-native data technologies.

What You Will Do

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, Delta Lake, and cloud-native tooling.
• Develop and support batch and streaming workflows (ingestion → transformation → validation → publishing of data products).
• Write, optimize, and maintain Python/PySpark/SQL for data processing, orchestration, and performance tuning.
• Work with large-scale datasets using Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage services.
• Troubleshoot pipeline failures, performance issues, data quality problems, and workflow bottlenecks.
• Translate business/technical requirements into data engineering designs, processing scripts, and orchestration workflows.
• Perform data validation, quality checks, code reviews, and resolve issues to ensure reliable data products.
• Collaborate with solution architects, data scientists/analysts, DevOps engineers, and client stakeholders.
• Communicate technical concepts and delivery impacts to both technical and non-technical audiences.
• Document pipelines, datasets, data models, workflows, and engineering decisions.
• Follow Databricks governance, security, lineage, and compliance standards.

What You Will Need

• Bachelor’s degree in CS/engineering/math/stats or related field.
• 3–8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation.
• Strong experience with Python, PySpark, and SQL for transformations and pipeline development.
• Experience with Databricks, Spark, Delta Lake, Unity Catalog, or similar cloud-native platforms.
• Experience building batch/streaming ETL/ELT and reusable data assets.
• Knowledge of data modeling, warehousing, data quality validation, and large-scale processing concepts.
• Ability to troubleshoot, clearly communicate recommendations, and work in team-based delivery.
• Experience with data governance, access control, lineage, and security in cloud data platforms.

Nice to Have

• 2+ years hands-on Databricks experience.
• Experience with Unity Catalog for governance, schema design, data modeling standards, access controls, lineage, and secure enterprise data management.