Principal Data Engineer

About the role

Join Octus as a Principal Data Engineer, where you will design, develop, and lead scalable data ingestion and transformation pipelines. This hands-on role includes management duties such as mentoring and providing technical leadership. You will play a key role in maintaining robust data infrastructure that powers data platforms, products, and automation initiatives across the firm. Key missions: Lead the design and development of scalable data ingestion and transformation pipelines, ensuring efficiency and reliability across diverse data sources.. Serve as a foundational technical leader and mentor for senior engineers within the data platform team, guiding architecture, design, and implementation decisions.. Architect and manage data pipelines and orchestration workflows using AWS services, implementing and maintaining infrastructure as code (IaC) using Terraform. Profile: - Familiarity with containerization (Docker) and workflow orchestration best practices - 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions - Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including those used in AI/ML pipelines - Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks, etc.) - Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices - Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end - Excellent communication, collaboration, and problem-solving skills - Advanced SQL skills and experience optimizing complex queries and data models - Expert proficiency in Python for data pipeline and automation development - Experience implementing and managing Terraform or similar IaC frameworks - Strong foundation in software engineering principles, including SOLID design, modularity, and scalability - Proven experience designing and maintaining cloud-native data pipelines on AWS (e.g., MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift, etc.) - Exposure to streaming data technologies (Kafka, Kinesis, Flink) - Experience integrating data quality and observability tools (Great Expectations, Monte Carlo, etc.) - Familiarity with Scrapy, BeautifulSoup, or other data extraction frameworks for ingestion pipelines