Data Platform Engineer

Role Overview

Bright Vision Technologies is seeking a Data Platform Engineer to design, build, and operate large-scale data processing pipelines and analytics platforms using Hadoop and related big-data technologies. This is a 100% remote (U.S.), full-time role.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines across relational, file-based, streaming, and API-driven sources.
• Build robust ETL/ELT workflows with Apache Spark, Hive, Pig, and Sqoop, with focus on data quality, idempotency, error handling, and recoverability.
• Create high-throughput streaming pipelines using Kafka, Spark Streaming, or Flink; integrate with downstream systems.
• Optimize Spark and MapReduce jobs (partitioning, memory, serialization, skew) to meet SLAs at minimal cost.
• Design and maintain data models and storage on HDFS, Hive, HBase, and lakehouse formats such as Parquet/ORC and Delta, Iceberg, Hudi.
• Implement data governance, lineage, and quality controls in collaboration with governance/security teams.
• Set up monitoring, alerting, and logging for pipeline reliability and proactive failure detection.
• Partner with data scientists and analysts to deliver curated, reliable, well-documented datasets.
• Automate pipeline orchestration using Airflow, Oozie, or similar workflow engines.
• Continuously evaluate and adopt new big-data/cloud technologies (e.g., EMR, Databricks, Snowflake, BigQuery) when they provide meaningful improvements.
• Lead performance reviews and architecture audits, proposing refactoring and optimization.
• Document architectures, schemas, pipeline behavior, and runbooks; support scalability.
• Mentor junior engineers and contribute to engineering standards and best practices.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical field.
• 5+ years experience designing and operating big-data pipelines on Hadoop.
• Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands-on streaming experience with Kafka, Spark Streaming, or Flink.
• Strong SQL skills; experience with both relational and NoSQL data stores.
• Experience with Airflow or Oozie for orchestration.
• Strong distributed systems understanding (partitioning, replication, fault tolerance).

Nice-to-Haves / Additional Signals

• Experience integrating modern lakehouse platforms and formats (e.g., Delta/Iceberg/Hudi).
• Experience with cloud big-data ecosystems such as EMR or Databricks.
• Leadership/mentoring experience for junior engineers.