About this role
The team is seeking a Senior Data Engineer to lead the design and delivery of data pipelines and ETL/ELT processes that support Client-Facing Analytics (ACIA) reporting and analytical capabilities. In this role, you will architect, develop, and optimize data models, transformations, and pipeline solutions that can scale to multi-terabyte datasets in Databricks.
Key Responsibilities:
- Lead the design and implementation of robust data pipelines and ETL/ELT processes.
- Architect and optimize data models to ensure efficient data processing and storage.
- Develop solutions using Python, Spark, and SQL to manage large datasets.
- Tune Spark performance, focusing on memory management, caching strategies, and cluster resource optimization.
- Manage Databricks cluster configurations to ensure optimal performance and reliability.
- Collaborate with cross-functional teams to understand data requirements and deliver high-quality solutions.
Required Skills & Qualifications:
- Proven experience with Databricks, Python, Spark, and SQL.
- Strong understanding of data engineering principles and best practices.
- Experience in optimizing data pipelines for performance and scalability.
- Familiarity with cloud platforms and data storage solutions.
- Excellent problem-solving skills and attention to detail.
Experience:
- 5 to 8 years of experience in data engineering or related fields.
What we offer:
The team provides a dynamic work environment, opportunities for professional growth, and the chance to work on innovative projects that make a significant impact.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.