About this role
The team is seeking an experienced Data Engineer to join their dynamic team. The ideal candidate will be responsible for designing, building, and maintaining robust ETL/ELT data pipelines using Apache Spark on Databricks. This role involves implementing a data Lakehouse architecture using Delta Lake for cost-effective data storage and analytics.
Key Responsibilities:
- Design, build, and maintain ETL/ELT data pipelines using Apache Spark on Databricks.
- Implement data Lakehouse architecture utilizing Delta Lake.
- Use Databricks Workflows for orchestrating both batch and streaming pipelines.
- Develop and maintain CI/CD pipelines for data applications using tools such as AWS, GitHub Actions, or Databricks Repos.
- Monitor pipeline performance and troubleshoot data issues in real-time and batch environments.
Required Skills & Qualifications:
- Proficiency in Apache Spark and Databricks.
- Experience with Delta Lake and data Lakehouse architecture.
- Familiarity with CI/CD practices and tools, particularly in a data context.
- Strong problem-solving skills and ability to troubleshoot complex data issues.
- Knowledge of AWS services and GitHub Actions.
Experience:
- 5-8 years of experience in data engineering or a related field.
What we offer:
The team provides a collaborative work environment, opportunities for professional growth, and the chance to work on innovative data solutions that drive business success.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.