Data Engineer - GCP

Role Overview

As a Google Cloud Data Engineer, you will design, develop, and manage scalable data pipelines and data infrastructure on GCP to ensure data availability, accuracy, and performance for analytics and ML models.

Responsibilities

Data Pipeline Development and Management

• Design, build, and maintain scalable data pipelines using Cloud Dataflow, Cloud Pub/Sub, and Cloud Composer
• Create ETL/ELT processes for structured and unstructured data at scale
• Optimize performance, scalability, reliability, and meet ingestion/processing SLAs
• Monitor workflows and resolve issues to prevent recurrence

Data Storage and Management

• Implement storage solutions with BigQuery, Cloud Storage, and Firestore
• Optimize data structures, partitioning, and storage for cost and performance
• Ensure security, integrity, and availability
• Manage lifecycle policies and archiving

Data Transformation and Processing

• Build transformation workflows using BigQuery, Apache Beam, and Cloud Functions
• Implement data quality checks, validation rules, and monitoring
• Support both real-time and batch processing

Data Integration and Automation

• Integrate data from APIs, databases, and third-party applications
• Automate ingestion/transformation/export using Cloud Composer and Cloud Functions
• Ensure data consistency across environments and systems

Collaboration and Stakeholder Engagement

• Partner with data scientists and analysts to understand data needs and business goals
• Provide technical guidance to data engineering and business teams
• Work with security/compliance teams to meet data governance standards

Performance Monitoring and Troubleshooting

• Monitor pipeline health and troubleshoot in real time
• Analyze failures and implement fixes
• Set up logging/monitoring using Stackdriver and Cloud Monitoring

Requirements

• Bachelor’s degree in Computer Science/Data Engineering (Master’s preferred)
• 3+ years data engineering experience; 2+ years on Google Cloud Platform
• Google Professional Data Engineer certification required
• Strong skills with BigQuery, Cloud Dataflow, Cloud Composer, Cloud Pub/Sub, Firestore, Cloud Functions
• Hands-on experience with Apache Beam and big data frameworks (Hadoop/Spark/Flink)
• Hands-on experience with dbt
• Proficiency in Python, Java, or Scala
• Strong SQL, data modeling, and query optimization
• Experience with CI/CD and version control (e.g., Git, Cloud Build)
• Knowledge of data governance, security, and compliance
• Ability to manage large-scale and real-time pipelines
• Excellent communication, analytical, and problem-solving skills
• Must be a U.S. Citizen or Green Card holder

Preferred Skills

• Experience with ML pipelines and AI/ML model deployment
• Familiarity with Terraform and Infrastructure as Code (IaC)
• Experience with NoSQL/key-value stores on GCP
• Knowledge of containerization and orchestration using GKE