Senior Data Engineer – Data Engineering

We are looking for experienced Senior Data Engineers to join our data engineering team. The ideal candidates will have strong hands-on experience working with large and complex datasets, developing data transformation pipelines, and leveraging Python, PySpark, Apache Spark, AWS, and Azure.

Key Responsibilities:

• Handle large, complex, multi-dimensional datasets, including structured, unstructured, and real-time data.
• Develop complex data transformation and ETL pipelines using Python, PySpark, and Apache Spark.
• Design, develop, and maintain scalable and reliable data pipelines.
• Work extensively with AWS and Azure cloud data services.
• Utilize cloud data engineering technologies such as AWS Glue, EMR, Azure Data Factory, and related services.
• Work with on-premises and cloud-based data warehouse databases and understand their architectures.
• Understand and apply data modeling concepts to create, maintain, and update data models for business requirements.
• Independently lead and deliver data engineering projects while ensuring high-quality outcomes.
• Ensure scalability, reliability, performance, and efficiency of data pipelines and processes.
• Troubleshoot and resolve data engineering, performance, and scalability issues.
• Stay updated with emerging data engineering tools, technologies, and industry practices.
• Collaborate with data scientists, analysts, business stakeholders, and other cross-functional teams.
• Translate business and technical requirements into effective data engineering solutions.
• Prepare and maintain technical specifications, design documents, and best-practice documentation.
• Communicate complex technical and design concepts effectively to stakeholders.
• Collaborate with team members to achieve project objectives and deliverables.

Required Skills & Qualifications:

• 4+ years of experience in data engineering or a related field.
• Strong programming skills in Python.
• Hands-on experience with PySpark and Apache Spark.
• Strong understanding of ETL/data transformation processes.
• Experience with both AWS and Azure cloud ecosystems.
• Knowledge of AWS data services such as AWS Glue and EMR.
• Knowledge of Azure data services such as Azure Data Factory.
• Good understanding of on-premises and cloud data warehouses.
• Strong knowledge of data modeling concepts.
• Experience building scalable, reliable, and high-performance data pipelines.
• Strong analytical and troubleshooting skills.
• Excellent communication skills and the ability to work independently with stakeholders.
• Bachelor's degree in Computer Science, a related discipline, or equivalent practical experience.

Good to Have:

• Experience with Snowflake.
• Experience with Dataiku.
• Experience with Alteryx or similar data technologies.
• Exposure to real-time data processing and streaming technologies.
• Experience working with data science and analytics teams.

What We Offer:

The team provides a dynamic work environment with opportunities for professional growth and development. You will be part of a collaborative team that values innovation and excellence.