As a Data Engineer, you will develop efficient data pipelines, collaborate with the Data Architect, conduct peer code reviews, and ensure quality through testing and deployment processes. You will work with GCP and Terraform to manage infrastructure and apply industry best practices in data architecture.
Job Description
We are seeking a skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will play a key role in understanding business and technological challenges, developing efficient data pipelines, and ensuring the smooth deployment of solutions. You will apply industry best practices while helping evolve the data architecture and infrastructure of our company.We are seeking a skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will play a key role in understanding business and technological challenges, developing efficient data pipelines, and ensuring the smooth deployment of solutions. You will apply industry best practices while helping evolve the data architecture and infrastructure of our company.
Key Responsibilities:
-
Project Understanding and Communication:
- Understand business challenges from the user’s perspective and clearly communicate to ensure a deep understanding of the issues.
- Collaborate with the Data Architect to fully comprehend the provided architecture and ensure alignment with your work.
- Communicate technical solutions with your peers and provide regular updates to the Project Manager overseeing the project.
-
Development:
- Write clear interface contracts for new or updated features and ensure they are communicated effectively.
- Develop data pipelines based on the defined architecture.
- Apply best practices to ensure quality code and maintainable solutions.
- Deploy infrastructure as requested, with a particular focus on using Terraform.
- Conduct peer code reviews and actively engage in reviewing peers’ code when merging new versions.
-
Testing:
- Collaborate with the Project Manager to define tests based on functional and technical requirements.
- Perform tests and regularly communicate the results to the team.
- Document and summarize test results for ongoing monitoring and improvements.
-
Deployments:
- Present completed work to the Data Architect and Lead DataOps during Deployment Reviews.
- Track and communicate any potential issues during the active monitoring phase after deployment.
- Diligently apply the deployment process, including logging and monitoring strategies.
Key Responsibilities:
-
Project Understanding and Communication:
- Understand business challenges from the user’s perspective and clearly communicate to ensure a deep understanding of the issues.
- Collaborate with the Data Architect to fully comprehend the provided architecture and ensure alignment with your work.
- Communicate technical solutions with your peers and provide regular updates to the Project Manager overseeing the project.
-
Development:
- Write clear interface contracts for new or updated features and ensure they are communicated effectively.
- Develop data pipelines based on the defined architecture.
- Apply best practices to ensure quality code and maintainable solutions.
- Deploy infrastructure as requested, with a particular focus on using Terraform.
- Conduct peer code reviews and actively engage in reviewing peers’ code when merging new versions.
-
Testing:
- Collaborate with the Project Manager to define tests based on functional and technical requirements.
- Perform tests and regularly communicate the results to the team.
- Document and summarize test results for ongoing monitoring and improvements.
-
Deployments:
- Present completed work to the Data Architect and Lead DataOps during Deployment Reviews.
- Track and communicate any potential issues during the active monitoring phase after deployment.
- Diligently apply the deployment process, including logging and monitoring strategies.
Qualifications
Requested Hard Skills:
- Google Cloud Platform (GCP): Strong knowledge of GCP and at least one year of hands-on experience with its services.
- Azure Cloud Platform: General knowledge of Azure is a plus.
- Apache Airflow: Minimum of two years of experience with Airflow orchestration; experience with Google Composer is an advantage.
- Terraform: Experience deploying infrastructure, especially using Terraform.
Desired Soft Skills:
- Strong communication skills for effective collaboration with cross-functional teams.
- Ability to tackle problems from a user perspective and ensure clear understanding across technical teams.
- Strong problem-solving abilities and a proactive approach to resolving issues.
- Detail-oriented, results-driven, and capable of working independently and as part of a team.
Additional Information
If you have experience with data pipelines, cloud technologies, and orchestrators like Apache Airflow, and you're looking for a new challenge, we’d love to hear from you!
Top Skills
Apache Airflow
Azure Cloud Platform
Google Cloud Platform
Terraform
Similar Jobs
Be an Early Applicant
The ITSM Analyst will monitor and troubleshoot ServiceNow environments, manage user access and permissions, perform updates and configurations, and provide guidance on platform capabilities. They will document system configurations and enhancements while collaborating with stakeholders to ensure system stability and compliance.
Be an Early Applicant
As a Data Scientist at Sword Health, you will lead and contribute to various AI/ML projects aimed at enhancing the company's health platform. Responsibilities include designing, testing, and implementing data models, conducting statistical analyses, collaborating with multiple departments, and deriving insights to improve patient recovery and business solutions.
Be an Early Applicant
As a Payments Optimization Analyst, you will analyze payment processing workflows to enhance efficiency, lower costs, and improve approval rates. Your responsibilities include working with large datasets, creating reports and dashboards, and conducting A/B testing on payment methods to provide actionable insights for management.
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
