About this role
The team is seeking a Lead Data Engineer with strong experience in building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure. This role requires a hands-on expert who can leverage their knowledge in various data engineering techniques and technologies to drive data solutions.
Key Responsibilities:
- Design and build batch, near-real-time, and streaming data pipelines on Azure.
- Develop Bronze, Silver, and Gold layers in a Medallion Architecture framework.
- Implement and optimize data processing workflows using Python and PySpark.
- Utilize advanced SQL for data manipulation and analysis.
- Deploy and manage Apache Spark workloads on Kubernetes or Azure Kubernetes Service (AKS).
- Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
- Ensure data quality, integrity, and security throughout the data lifecycle.
Required Skills & Qualifications:
- Strong experience with Microsoft Azure data services.
- Proficiency in Python and PySpark for data processing.
- Advanced SQL skills for querying and managing large datasets.
- Familiarity with Medallion Architecture principles.
- Experience deploying applications on Kubernetes or Azure Kubernetes Service (AKS).
- Excellent problem-solving skills and ability to work in a fast-paced environment.
Experience:
- Minimum of 5-8 years of experience in data engineering or related fields.
What we offer:
- Opportunity to work on cutting-edge data technologies.
- Collaborative and innovative work environment.
- Professional development and growth opportunities.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.