Design, build, and optimize data platforms using AWS and Python. Responsibilities include architecting data pipelines, ensuring data quality, and leading technical teams.
Job Overview:
We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable, high-performance data platforms using AWS cloud services and Python. The ideal candidate will play a key role in architecting end-to-end data pipelines, driving automation, ensuring data quality, and enabling analytics and AI workloads across the organization.
This role requires deep technical expertise in AWS data services, modern data architecture, and a passion for delivering reliable, high-quality data solutions at scale.
Key Responsibilities
- Architect and implement scalable, fault-tolerant data pipelines using AWS Glue, Lambda, EMR, Step Functions, and Redshift
- Build and optimize data lakes and data warehouses on Amazon S3, Redshift, and Athena
- Develop Python-based ETL/ELT frameworks and reusable data transformation modules
- Integrate multiple data sources (RDBMS, APIs, Kafka/Kinesis, SaaS systems) into unified data models
- Lead efforts in data modeling, schema design, and partitioning strategies for performance and cost optimization
- Drive data quality, observability, and lineage using AWS Data Catalog, Glue Data Quality, or third-party tools
- Define and enforce data governance, security, and compliance best practices (IAM policies, encryption, access control)
- Collaborate with cross-functional teams (Data Science, Analytics, Product, DevOps) to support analytical and ML workloads
- Implement CI/CD pipelines for data workflows using AWS CodePipeline, GitHub Actions, or Cloud Build
- Provide technical leadership, code reviews, and mentoring to junior engineers
- Monitor data infrastructure performance, troubleshoot issues, and lead capacity planning
Required Skills & Qualifications
- Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field
- 5–10 years of hands-on experience in data engineering or data platform development
- Expert-level proficiency in Python (pandas, PySpark, boto3, SQLAlchemy)
- AWS Glue, Lambda, EMR, Step Functions, DynamoDB, EDW Redshift, Athena, S3, Kinesis, Amazon Quicksight.
- IAM, CloudWatch, CloudFormation / Terraform (for infrastructure automation)
- Strong experience in SQL, data modeling, and performance tuning
- Proven ability to design and deploy data lakes, data warehouses, and streaming solutions
- Solid understanding of ETL best practices, partitioning, error handling, and data validation
- Hands-on experience in version control (Git) and CI/CD for data pipelines
- Knowledge of containerization (Docker/Kubernetes) and DevOps concepts
- Excellent analytical, debugging, and communication skills
Advanced experience with AWS Data Services, including:
Preferred Skills
- Experience with Apache Spark or PySpark on AWS EMR or Glue
- Familiarity with Airflow, dbt, or Dagster for workflow orchestration
- Exposure to real-time data streaming (Kafka, Kinesis Data Streams, or Firehose)
- Knowledge of Lake Formation, Glue Studio, or DataBrew
- Experience integrating with machine learning and analytics platforms (SageMaker, QuickSight)
- Certification: AWS Certified Data Analytics – Specialty or AWS Certified Solutions Architect
Soft Skills
- Strong ownership mindset with focus on reliability and automation
- Ability to mentor and guide data engineering teams
- Effective communication with both technical and non-technical stakeholders
Top Skills
Amazon S3
Athena
Aws Glue
Boto3
Ci/Cd
Docker
Emr
Git
Kubernetes
Lambda
Pandas
Pyspark
Python
Redshift
SQL
Sqlalchemy
Step Functions
Similar Jobs
Big Data • Marketing Tech • Analytics
As a Senior Data/Cloud Platform Engineer, you will design and implement secure cloud solutions, lead infrastructure strategy, and enhance DevSecOps practices within an agile team.
Top Skills:
AWSAws Lake FormationCi/CdDevsecopsEc2GitGlueGroovyIamPythonShell ScriptingTerraform
Information Technology • Insurance • Software
We seek a Senior Software Engineer skilled in Full Stack Development with .NET and SQL Server, responsible for designing, developing, and maintaining web applications, collaborating with teams, optimizing databases, and ensuring code quality.
Top Skills:
.NetAngularAsp.NetC#CSSDevOpsDockerHTMLJavaScriptJqueryKubernetesMvcReactSQLSQL ServerSsisWeb Api
Fintech • Financial Services
The Operations Processor will perform operational duties, data collection, report preparation, policy guidance, and project coordination in a banking environment.
Top Skills:
ChipsFed WireIso20022Pega SystemSwiftTelex
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
