Job Description

Data Engineer – GCP / PySpark

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Google Cloud Platform (GCP), Apache Kafka, Apache Airflow, PySpark, and Hive. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, processing large volumes of data, and supporting batch and real-time data processing requirements.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using GCP services and Apache Spark.

Build and optimize data processing workflows using Python, PySpark, and Spark.

Develop and maintain real-time data streaming pipelines using Apache Kafka.

Create, schedule, and monitor data workflows using Apache Airflow and DAGs.

Work with Hive for querying, transforming, and managing large datasets.

Implement efficient ETL/ELT processes for data ingestion, transformation, and integration.

Optimize data pipelines for performance, scalability, reliability, and cost efficiency.

Troubleshoot data processing issues and ensure data quality and consistency.

Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.

Required Skills

Strong hands-on experience with Google Cloud Platform (GCP).

Experience with Apache Kafka and real-time data streaming.

Proficiency in Apache Airflow for workflow orchestration and scheduling.

Strong programming skills in Python and experience with PySpark or Apache Spark.

Experience working with Apache Hive and large-scale data processing.

Strong understanding of distributed computing, data engineering concepts, and ETL/ELT pipelines.

Nice-to-Have Skills

Experience with Scala programming.

Familiarity with additional GCP data services such as BigQuery, Google Cloud Storage, and Dataproc.

Experience with data pipeline performance tuning and optimization.

Preferred Candidate Profile

The ideal candidate will have practical experience building and supporting data pipelines in a cloud environment, processing large datasets using Spark, implementing streaming solutions with Kafka, and orchestrating workflows through Airflow. Strong GCP, PySpark/Spark, Kafka, Airflow, and Hive skills are essential. Scala is an added advantage.