Lead the design, development, and optimization of Azure-based data platforms, pipelines, enterprise data models, and streaming solutions. Build AI-ready data assets using Databricks, PySpark, SQL, and Delta Lake. Ensure data quality, governance, security, reliability, and cost efficiency while implementing CI/CD and integrations. Mentor engineers, collaborate with stakeholders, and align technical priorities with commercial objectives.
We are looking for a Lead Data Engineer to join our eCommerce Data team. Based from NEXT Head Office in Enderby, Leicestershire!
The Role:
As a Lead Data Engineer, you will play a pivotal role in designing, building and evolving our cloud based data platform on Azure. You will be responsible for developing scalable, reliable and high performance data solutions that power analytics, reporting and emerging AI capabilities across the business.
Working primarily with Azure Databricks, PySpark, SQL and Azure Data Factory, you will lead the development of robust data pipelines, champion engineering best practices and help shape the technical direction of our data platform. You will also drive the design of a modern, AI ready data model that enables faster adoption of machine learning, generative AI and advanced analytics.
This is a hands-on technical leadership role where you'll mentor engineers, collaborate closely with business stakeholders and balance technical excellence with commercial priorities to ensure the team delivers maximum business value.
What You’ll Take On:
- Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure services such as Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.
- Design and evolve a modern enterprise data model that provides trusted, reusable data assets and accelerates AI, machine learning, and advanced analytics initiatives.
- Ensure data quality, integrity, security, and governance across the data platform by implementing engineering best practices and robust monitoring.
- Collaborate with data scientists, analysts, product teams, and business stakeholders to understand requirements and deliver scalable data solutions.
- Optimise data platforms and pipelines for performance, scalability, reliability, and cost efficiency.
- Lead the implementation of streaming data solutions and Delta Live Tables using PySpark and Databricks.
- Build and maintain CI/CD pipelines for data engineering solutions using Azure DevOps, Git, and infrastructure-as-code where appropriate.
- Integrate with third-party systems and APIs to support secure and reliable data ingestion.
- Mentor and support junior engineers, promoting high engineering standards, knowledge sharing, and continuous improvement.
- Troubleshoot and resolve complex data engineering challenges while ensuring high availability of critical data services.
- Develop a strong understanding of business priorities and commercial objectives, helping to prioritise engineering work that delivers the greatest value and business impact.
What You’ll Bring:
- Strong Python, PySpark and SQL skills with extensive experience designing, developing and optimising large scale data pipelines.
- Hands-on experience with Azure data technologies, including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Delta Lake.
- Experience working with relational and NoSQL databases such as Cosmos DB.
- Experience building streaming data pipelines and Delta Live Tables using Databricks.
- Strong understanding of modern data modelling techniques, including designing data models that support analytics and AI workloads.
- Experience implementing CI/CD pipelines using Azure DevOps, Git, and related tooling.
- Experience integrating with third party APIs and external data sources.
- Strong understanding of data governance, data quality, security and operational best practices.
- Excellent problem solving, communication and stakeholder management skills.
- Strong commercial awareness, with the ability to balance technical priorities against business value and make pragmatic engineering decisions.
Desirable Skills:
- Knowledge of machine learning concepts, MLOps and deploying data products for AI use cases.
- Experience with modern AI and Generative AI platforms, including preparing data for Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) solutions.
- Experience with other cloud platforms such as AWS or GCP.
- Experience with enterprise data governance frameworks and tooling.
- 25% off most NEXT, MADE*, Lipsy*, Gap* and Victoria's Secret* products (*when purchased through NEXT)
- Company performance based bonus
- Sharesave scheme
- On-site Nursery available; OFSTED outstanding in all areas
- 10% off most partner brands & up to 15% off Branded Beauty
- Early VIP access to sale stock
- Access to fantastic discounts at our Staff Shops
- Restaurants with great food at amazing prices
- Access a digital GP and other free health and wellbeing services
- Free on-site parking
- Financial Wellbeing - Save, track and enhance your financial wellbeing
- Apprenticeship - Grow and develop on the job whilst gaining a qualification
- Direct to Work - Discount online and instore, collect your items the next day for free from your place of work or local store
- Support Networks - Access to Network Groups to empower and celebrate each other
- Wellhub - Discounted flexible monthly gym memberships, with apps, PT sessions and more
Conditions apply to all benefits. These benefits are discretionary and subject to change.
We aim to support all candidates during the application process and are happy to provide workplace adjustments when necessary. Should you need support with your application due to a disability or long-term condition, feel free to get in touch with us by email [email protected] (please include 'Workplace Adjustments' in the subject line), or call us on 0116 284 2486 and leave a voicemail.
Similar Jobs
Financial Services
Lead development of production AI agents and machine learning workflows for B2B agentic commerce and payments. Responsibilities include agent orchestration, negotiation, onboarding, model training and serving on Kubernetes, evaluation, observability, secure software development, model-risk documentation, and technical leadership. The role also guides AI-assisted engineering practices and evaluates architectures with vendors and internal teams.
Top Skills:
A2AAg-UiAmazon EksApache IcebergSparkCi/CdDatabricksDelta LakeGoogle AdkJavaKubernetesLanggraphLlmsMcpMicro-Vm IsolationMlflowOpentelemetryPythonSidecar PatternsTypescript
Financial Services
Lead architecture and engineering for B2B agentic commerce and machine-learning systems in global payments. Define multi-agent architectures, authorization, information barriers, MLOps pipelines, evaluation frameworks, safety guardrails, and model-risk evidence. Develop and review secure production code, establish AI-assisted engineering standards, integrate external agents, improve operational resilience, and guide technical teams and vendors. The role requires advanced Python, production LLM agent experience, secure distributed-systems expertise, and financial-services technology knowledge.
Top Skills:
A2AAg-UiCi/CdDatabricksDistributed SystemsGoogle AdkGpu InferenceGraphragIdentity And AuthorizationKnowledge GraphsKubernetesLanggraphLlm AgentsMcpMicro-Vm IsolationMl PlatformsMlflowMlopsMulti-Agent SystemsOpa/RegoOpenfgaPythonService Mesh
Financial Services
Leads development of secure, scalable data platforms supporting batch, streaming, ML, and LLM workloads. Designs distributed data systems using Spark, Ray, Kafka, Iceberg, and OpenSearch; productionizes model training, deployment, monitoring, and lifecycle management; and builds governed LLM solutions. Provides technical leadership, code review, incident response, vendor evaluation, responsible AI guidance, automation, and mentorship across engineering teams.
Top Skills:
Ci/CdCloud ComputingContainersDatabasesDistributed SystemsIcebergInfrastructure As CodeJavaKafkaKotlinKubernetesLarge Language ModelsMachine LearningMcpObservabilityOpensearchPythonRagRayScalaSparkVector Search
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
