Design, build, and maintain data pipelines using AWS and Python. Collaborate with teams for data solutions and ensure quality and governance.
Job Overview:
We are seeking a highly skilled Data Engineer with strong expertise in AWS cloud services and Python programming. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines, ensuring data availability, quality, and performance across enterprise systems. You will collaborate closely with data analysts, data scientists, and business stakeholders to deliver reliable, high-quality data solutions.
Key Responsibilities
- Design, develop, and maintain ETL/ELT data pipelines using Python and AWS native services (Glue, Lambda, EMR, Step Functions, etc.)
- Build and manage data lakes and data warehouses using Amazon S3, Redshift, Athena, and Lake Formation
- Implement data ingestion from diverse sources (RDBMS, APIs, streaming data, on-premise systems)
- Optimize data workflows for performance, cost, and reliability using AWS tools like Glue Jobs, Athena, and Redshift Spectrum
- Develop reusable, modular Python-based frameworks for data ingestion, transformation, and validation
- Work with stakeholders to understand data requirements, model data structures, and ensure data consistency and governance
- Deploy and manage data infrastructure using Infrastructure as Code (IaC) tools such as Terraform or AWS CloudFormation
- Implement data quality, monitoring, and alerting using CloudWatch, Glue Data Catalog, or third-party tools
- Support data security and compliance (IAM roles, encryption, data masking, GDPR policies, etc.)
- Collaborate with DevOps and ML teams to integrate data pipelines into analytics and AI workflows
Required Skills & Qualifications
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
- Minimum 5 to 8 years of experience as a Data Engineer or similar role.
- Strong programming experience in Python (pandas, boto3, PySpark, SQLAlchemy, etc.)
- Deep hands-on experience with AWS services, including:
- AWS Glue, Lambda, EMR, Redshift, Athena, S3, Step Functions
- IAM, CloudWatch, Kinesis (for streaming), and ECS/EKS (for containerized workloads)
- Experience with SQL and NoSQL databases (e.g., PostgreSQL, DynamoDB, MongoDB)
- Strong knowledge of data modeling, schema design, and ETL orchestration.
- Familiarity with version control (Git) and CI/CD pipelines for data projects.
- Understanding of data governance, lineage, and cataloging principles.
- Excellent problem-solving, debugging, and performance-tuning skills.
Preferred Skills
- Experience with Apache Spark or PySpark on AWS EMR.
- Exposure to Airflow, dbt, or similar workflow orchestration tools.
- Knowledge of containerization (Docker, Kubernetes) and DevOps practices.
- Experience with machine learning data pipelines or real-time streaming (Kafka, Kinesis).
- Familiarity with AWS Glue Studio, AWS DataBrew, or AWS Lake Formation.
Soft Skills
- Strong analytical and communication skills.
- Ability to work independently and in cross-functional teams.
- Passion for automation and continuous improvement.
- Adaptability in fast-paced, evolving cloud environments.
Top Skills
Athena
AWS
CloudFormation
DynamoDB
Emr
Git
Glue
Lambda
MongoDB
Postgres
Python
Redshift
S3
SQL
Step Functions
Terraform
Similar Jobs
Artificial Intelligence • Big Data • Cloud • Information Technology • Machine Learning
Design, build, and optimize data platforms using AWS and Python. Responsibilities include architecting data pipelines, ensuring data quality, and leading technical teams.
Top Skills:
Amazon S3AthenaAws GlueBoto3Ci/CdDockerEmrGitKubernetesLambdaPandasPysparkPythonRedshiftSQLSqlalchemyStep Functions
Big Data • Marketing Tech • Analytics
As a Senior Data/Cloud Platform Engineer, you will design and implement secure cloud solutions, lead infrastructure strategy, and enhance DevSecOps practices within an agile team.
Top Skills:
AWSAws Lake FormationCi/CdDevsecopsEc2GitGlueGroovyIamPythonShell ScriptingTerraform
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
The role involves supporting sales operations, ensuring forecast accuracy, analyzing customer data for revenue opportunities, and collaborating with various departments to enhance sales efficiency.
Top Skills:
Ai-Powered ToolsCrm SystemExcelMicrosoft PowerpointMicrosoft Word
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
