As a Senior Data Engineer joining our new Marine Insurance initiative, you will play a central role in shaping the systems that power Kpler’s global market intelligence platform. You will lead the end-to-end design and delivery of complex data pipelines and cloud infrastructure, ingesting, processing, and serving large-scale data to our global customers. Acting as a technical leader, you will set data architecture directions and mentor peer engineers, ensuring our platform reliably scales as our datasets and product offerings expand.
Key Responsibilities
Design and own end-to-end data pipelines: Build, deploy, and operate production-grade data pipelines and infrastructure from initial architecture through deployment, monitoring, and maintenance.
Deploy containerized AWS services: Develop and containerize applications within AWS using Python, Docker, Kubernetes, Airflow, and Infrastructure as Code (Terraform).
Ensure data integrity and event streaming: Architect robust PostgreSQL database structures and event-streaming flows (Kafka/Flink) to guarantee seamless data flow and integrity.
Set technical direction and standards: Lead complex pipeline design, peer reviews, and logging, monitoring, and alerting frameworks to maintain pipeline resilience.
Partner cross-functionally and mentor peers: Collaborate closely with engineers, product managers, and data scientists while mentoring engineers to elevate team craft and technical development.
Evaluate emerging data tools: Research and implement modern data infrastructure improvements and frameworks to continuously elevate Kpler's platform capabilities.
Experience & Background
- What you'll need (Must-haves)
Data engineering experience at scale: 5+ years of experience designing, operating, and owning production-grade data pipelines and infrastructure.
Core tech stack depth: Strong Python skills with hands-on experience in Docker, Kubernetes, AWS, Airflow, and relational databases like PostgreSQL.
Streaming, IaC, and observability: Practical experience with event streaming (Kafka/Flink), Infrastructure as Code (Terraform), CI/CD practices, and production logging/monitoring systems.
Technical leadership & AI coding tools: Demonstrated ability to lead architectural discussions, mentor engineers, and leverage AI-assisted coding tools (Copilot, Cursor, Claude) with strong technical judgment.
Methodologies & communication: Experience working in Agile environments with strong collaboration skills and fluency in written and spoken English.
Nice-to-havesLarge-scale analytics platforms: Experience building large-scale data processing systems or analytics platforms.
Cross-team initiatives: Track record of leading technical initiatives across multiple teams.
Domain knowledge: Familiarity with commodities, shipping, energy, or marine insurance market data.
Kpler London, England Office
London, United Kingdom
