Senior Data Engineer (AWS to GCP)

Role overview

Senior Data Engineer responsible for building and evolving large-scale data products on a major cloud transformation.

• Join a high-performing data engineering team in a media/streaming environment.
• Migrate key data systems from AWS to Google Cloud Platform (GCP), then focus on optimization, performance tuning, and new data capabilities for business decision-making.

Responsibilities

• Design, build, and maintain scalable data pipelines and data products on GCP with performance, reliability, and scalability.
• Support migration of existing AWS-based data workloads to GCP with minimal disruption.
• Develop and optimize ETL/ELT workflows using:
  • Python
  • SQL
  • PySpark (large-scale processing)
• Build/maintain/enhance data transformations using:
  • BigQuery
  • Dataproc
• Create and manage orchestration workflows using Apache Airflow (Cloud Composer).
• Implement/improve data quality frameworks to ensure trusted outputs.
• Collaborate with analysts, product owners, and engineers to translate business needs into data solutions.
• Participate in code reviews, technical discussions, and continuous improvement.
• Contribute to platform reliability, monitoring, documentation, and operational excellence.

Requirements

• Strong commercial experience as a Data Engineer with cloud-based data platforms.
• Hands-on expertise with GCP, especially BigQuery.
• Proven experience building/maintaining Airflow pipelines (prefer Cloud Composer).
• Strong Python skills and solid SQL expertise.
• Experience with PySpark and distributed/large-scale data processing.
• Understanding of data modelling and modern ETL/ELT design patterns.
• Experience with Git-based workflows in collaborative environments.
• Strong communication skills with technical and non-technical stakeholders.
• Experience delivering in Agile environments.

Nice to have

• Dataproc, Dataplex, Great Expectations (or similar) for data quality.
• AWS experience (e.g., S3, MWAA, Databricks).
• Experience in cloud migration projects.
• Exposure to media/streaming/large-scale consumer platforms.