As an AI Engineer, you'll develop the LLM evaluation framework, conduct A/B testing, define metrics, and ensure reliable model releases.
About Mistral
At Mistral AI, we believe in the power of AI to simplify tasks, save time, and enhance learning and creativity. Our technology is designed to integrate seamlessly into daily working life.
We democratize AI through high-performance, optimized, open-source and cutting-edge models, products and solutions. Our comprehensive AI platform is designed to meet enterprise needs, whether on-premises or in cloud environments. Our offerings include le Chat, the AI assistant for life and work.
We are a dynamic, collaborative team passionate about AI and its potential to transform society.
Our diverse workforce thrives in competitive environments and is committed to driving innovation. Our teams are distributed between France, USA, UK, Germany and Singapore. We are creative, low-ego and team-spirited.
Join us to be part of a pioneering company shaping the future of AI. Together, we can make a meaningful impact. See more about our culture on https://mistral.ai/careers.
Role summary
Embedded directly in the product Team, you will build the evaluation and A/B testing framework, add end-to-end observability, and run a reliable model release process. Work with Science to ship measurable improvements to quality, latency, safety, and reliability
What you will do
• Build and maintain an LLM evaluation framework (reference tests, heuristics, model-graded checks).
• Define and track metrics: task success, helpfulness, hallucination proxies, safety flags, latency/cost.
• Run A/B tests for prompts, models, and system prompts, analyze results, recommend rollout or rollback.
• Set up observability for LLM calls: structured logging, tracing, dashboards, alerts.
• Operate the model release: canary and shadow traffic, sign-offs, SLO-based rollback criteria, regression detection.
• Improve core behaviors: memory write/retrieve policies and evals, intent classification, follow-ups, routing, tool-call reliability.
• Create templates and docs so other teams can author evals and ship safely.
• Partner with Science, diagnose regressions and lead post-mortems.
About you
• Strong TypeScript or Python skills
• Production LLM experience: prompts, tool/function calling, and system prompts.
• Hands-on with evals and A/B testing, you can design metrics and make rollout decisions from data.
• Observability: logging, tracing, dashboards, alerting
• Product mindset: form hypotheses, run experiments, interpret results, iterate.
• Clear written and spoken communication, autonomous; and product-oriented.
Now it would be ideal if you have experience with
• Safety systems: moderation, PII handling/redaction, guardrails.
• Release operations: canary/shadowing, automated rollbacks, experiment platforms.
Hiring Process
• Introduction call - 30 min
• Hiring Manager interview - 30 min
• Technical Rounds
- Live-coding Interview - 45 min
- AI Engineering Interview - 45 min
• Culture-fit discussion - 30 min
• References
Location & Remote
This role is primarily based at one of our European offices (Paris, France and London, UK). We will prioritize candidates who either reside in Paris or are open to relocating. We strongly believe in the value of in-person collaboration to foster strong relationships and seamless communication within our team.
In certain specific situations, we will also consider remote candidates based in one of the countries listed in this job posting — currently France, UK, Germany, Belgium, Netherlands, Spain and Italy. In that case, we ask all new hires to visit our Paris office:
• for the first week of their onboarding (accommodation and travelling covered)
• then at least 3 days per month
What we offer
💰 Competitive salary and equity
🧑⚕️ Health insurance
🚴 Transportation allowance
🥎 Sport allowance
🥕 Meal vouchers
💰 Private pension plan
🍼 Parental : Generous parental leave policy
🌎 Visa sponsorship
Top Skills
Python
Typescript
Similar Jobs
Fintech • Software • Financial Services
As a Senior Machine Learning Engineer, you will develop and scale customer-facing AI models, ensuring quality integration with business solutions, and mentoring peers.
Top Skills:
AirflowArgocdAWSFastapiGitKafkaKibanaPostgresPrometheusPythonSnowflake
Fintech • Software • Financial Services
As a Senior MLOps, you'll build and maintain data pipelines, design data storage solutions, and deploy model-serving systems, collaborating with ML Engineers for performance and quality.
Top Skills:
AirflowDbtKafkaPythonSnowflakeSQL
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
The role focuses on supporting service operations, ensuring adherence to service level agreements, and leading process improvements.
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
