About this role
The team is seeking a skilled Data Engineer to join their dynamic team. In this role, you will be responsible for designing, developing, and maintaining ELT/ETL pipelines on Google Cloud Platform (GCP). You will work with various tools and technologies to ensure efficient data processing and management.
Key Responsibilities:
- Design and implement ELT/ETL pipelines using Dataflow/Beam, Dataproc/Spark, and Airflow/Composer.
- Model and optimize datasets in BigQuery, utilizing partitioning, clustering, materialized views, and user-defined functions (UDFs).
- Build streaming and near-real-time data ingestion processes using Pub/Sub, Dataflow, and Change Data Capture (CDC) where applicable.
- Implement data quality checks and validation frameworks, ensuring adherence to service level agreements (SLAs).
- Monitor and optimize pipeline performance and costs across Google Cloud Storage (GCS) and other services.
Required Skills & Qualifications:
- Proficiency in GCP services, particularly Dataflow, Dataproc, BigQuery, and Pub/Sub.
- Strong experience with ETL/ELT processes and data modeling techniques.
- Familiarity with data quality frameworks and monitoring tools such as Cloud Monitoring and Logging.
- Knowledge of programming languages such as Python or Java.
- Excellent problem-solving skills and the ability to work collaboratively in a team environment.
Experience:
- 5-8 years of relevant experience in data engineering or a similar role.
What we offer:
- Opportunity to work with cutting-edge technologies in a collaborative environment.
- Professional development and growth opportunities.
- A supportive team culture that values innovation and initiative.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.