Data Engineer

Role Overview

Join Hamilton Lane as a Data Engineer (ETL Data Engineer) to scale and strengthen the company’s data engineering capabilities. You’ll design, develop, and maintain ETL integration processes using Python (PySpark) and Azure Synapse Analytics, delivering accurate and reliable data for analytics and decision-making.

Responsibilities

• ETL Data Engineering: Build and maintain ETL processes using Python (PySpark) in Azure Synapse Analytics Notebooks and/or Azure Synapse Analytics Pipelines.
• Data Warehousing: Design and build data storage structures using MPP (Massively Parallel Processing) SQL pool concepts, including star schemas, facts, and dimensions.
• Data Source Expertise: Extract data from REST APIs, database tables (SWL), and CSV files.
• Azure Synapse Optimization: Optimize notebooks/pipelines for scalability and performance.
• Data Fabric Concepts: Contribute to data management patterns such as data lakes, lakehouses, delta lakes, and data cataloging.
• Data Modeling: Partner with data architects to create models and schemas aligned to business requirements.
• Data Quality: Implement validation checks to ensure accuracy and consistency.
• Performance Tuning: Diagnose bottlenecks and optimize pipelines to meet SLAs.
• Monitoring & Troubleshooting: Monitor ETL jobs, troubleshoot issues, and ensure pipeline reliability.
• Documentation: Maintain clear documentation for data engineering processes, flows, and transformations.
• Collaboration & Security: Work with cross-functional teams; support security, governance, and privacy requirements.

Requirements

• Bachelor’s degree in Computer Science, Information Technology, or related field (or equivalent experience).
• Proven ETL experience with strong Python (PySpark) capability.
• Demonstrated use of Azure Synapse Analytics for ETL development.

Nice to Have

• Certifications related to data engineering/data science (e.g., Azure Data Engineer).