About this role
The team is seeking an experienced PySpark Engineer to join their team. This role focuses on designing and developing enterprise-grade Spark applications and scalable ETL data pipelines. The ideal candidate will have strong expertise in Apache Spark, PySpark, Spark SQL, Python, and Spark Streaming.
Key Responsibilities:
- Design, develop, and maintain Spark applications to support data processing needs.
- Build and optimize scalable ETL data pipelines using PySpark.
- Implement and manage Spark cluster configurations for optimal performance.
- Work with Delta Lake and cloud-based Big Data platforms to ensure data integrity and availability.
- Collaborate with cross-functional teams to gather requirements and deliver data solutions.
- Troubleshoot and resolve issues related to Spark applications and data processing workflows.
Required Skills & Qualifications:
- Proficient in Apache Spark, PySpark, Spark SQL, and Python.
- Experience with Spark Streaming and building real-time data processing applications.
- Hands-on experience with Delta Lake and cloud-based Big Data platforms.
- Strong understanding of data modeling and ETL processes.
- Excellent problem-solving skills and ability to work in a fast-paced environment.
Experience:
- 5-7 years of relevant experience in data engineering or similar roles.
What we offer:
- Opportunity to work on cutting-edge technologies in a dynamic environment.
- Collaborative team culture that encourages innovation and professional growth.
- Flexible work arrangements and a focus on work-life balance.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.