.png)
Build and operate Spotify’s lakehouse and BigQuery platforms at scale. Develop backend services supporting Apache Iceberg, migrate pipelines, manage BigQuery capacity, create monitoring and tooling, and troubleshoot production issues across storage, Kubernetes, IAM, and processing systems. Participate in support and on-call rotations while improving reliability, cost efficiency, and user enablement across global teams.
LilEngine is leading the Lakehouse, a company-strategic effort to make Apache Iceberg a first-class citizen in Spotify's data ecosystem. We're building it in close collaboration with Google, on open standards, so that different kinds of data are queryable the same way and teams can bring the right engine, from BigQuery to Dataflow to Trino, to the job.
The work has two parts. Hecate enables Iceberg for the Parquet datasets that already exist across Spotify, so teams get the benefits without rewriting their data. Alongside it, we're building the foundations of a new Lakehouse architecture that lets workflows write Iceberg tables natively. We're shipping the MVP this cycle, and you'll help shape what comes next.
We also own BigQuery for all of Spotify. Thousands of engineers, data scientists and pipelines depend on it every day, which puts us on the critical path for storing, finding and querying data. We're judged on reliability, cost efficiency and how quickly we unblock users when something breaks. This backend role sits where open table formats, cloud data warehouses and processing engines meet.
What You'll Do
- Build and evolve the lakehouse: Help build the foundations that let data workflows read and write Iceberg tables directly, from this cycle's MVP to a production-grade platform.
- Drive adoption: Help migrate data pipelines from across Spotify onto the new stack, making the move as painless as possible through tooling, standards and hands-on support.
- Own BigQuery at Spotify scale: Manage reservations and capacity, build monitoring and anti-pattern detection, and develop tooling like the BigQuery MCP server so teams can use BigQuery well.
- Keep the platform healthy: Take part in a weekly support rotation and shared on-call. You'll debug production issues that span storage formats, catalog APIs, Kubernetes reconciliation, GCP IAM and BigQuery internals, and you'll automate away recurring toil.
Who You Are
- You have solid experience building and operating backend services in production with Java or Scala.
- You're comfortable on GCP and Kubernetes. You can read kubectl output, reason about IAM and quotas, and work with CRDs and operators. You don't need deep expertise in all of these on day one.
- You've worked with Apache Iceberg or JVM-based processing frameworks like Spark, Beam or Flink, or with a cloud data warehouse such as BigQuery, Snowflake or Databricks.
- You think in systems and can connect the dots when a problem crosses several layers of the stack.
- You like owning things in production, and you're motivated by seeing your systems run reliably for thousands of users.
- You communicate clearly in writing and work well asynchronously across US and EU time zones.
- You own the quality of what you ship, including code written with AI assistance.
Where You'll Be
This role is based in London or Stockholm.
We offer the flexibility to work where you work best! There will be some in person meeetings, but still allows for flexibility to work from home.
Spotify is an equal opportunity employer. You are welcome at Spotify for who you are, no matter where you come from, what you look like, or what’s playing in your headphones. Our platform is for everyone, and so is our workplace. The more voices we have represented and amplified in our business, the more we will all thrive, contribute, and be forward-thinking! So bring us your personal experience, your perspectives, and your background. It’s in our differences that we will find the power to keep revolutionizing the way the world listens.
At Spotify, we are passionate about inclusivity and making sure our entire recruitment process is accessible to everyone. We have ways to request reasonable accommodations during the interview process and help assist in what you need. If you need accommodations at any stage of the application or interview process, please let us know - we’re here to support you in any way we can.
Spotify London, England Office
London, United Kingdom
Similar Jobs
Music
Build and operate backend services, APIs, data catalogs, lineage systems, and publishing platforms for Spotify’s data ecosystem. Develop batch and streaming pipelines, manage metadata and schemas, ensure data quality and reliability, and own projects from technical design through production rollout. Participate in on-call support, observability, automation, and cross-team collaboration.
Top Skills:
Apache BeamApache FlinkApache IcebergSparkBigQueryDatabricksJavaOpenlineageSnowflakeSQL
Artificial Intelligence • Consumer Web • Digital Media • Information Technology • Social Impact • Software
Lead the AI Quality engineering team while remaining hands-on building evaluation infrastructure, observability tooling, diagnostic systems, datasets, annotation workflows, and CI/CD pipelines for production AI agents. Run experiments across prompts, models, routing, caching, and agent harnesses to improve quality, latency, and cost. Partner with AI Core engineering and manage technical direction and team priorities.
Top Skills:
Ai AgentsBraintrustCi/CdLangsmithLlmsPythonRuby On Rails
Artificial Intelligence • Big Data • Enterprise Web • Fintech • Software • Financial Services
Supports Nordic business growth through prospect research, lead generation, outreach, opportunity qualification, account planning, CRM maintenance, and pipeline reporting. Collaborates with Sales, Marketing, Consulting, and Client Success teams while engaging financial institutions and developing expertise in B2B sales, ESG, and sustainable finance.
Top Skills:
CRM
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
