Easy Apply
Easy Apply
The Senior Product Manager will lead the DPAI Arena project's evaluation standards, working with stakeholders, curating datasets, and defining metrics for success.
About us
At JetBrains, code is our passion. Ever since we started, we have strived to make the strongest, most effective developer tools on earth. By automating routine checks and corrections, our tools speed up production, freeing developers to grow, discover, and create.
We believe in building tools developers love – and we see the next frontier in AI-powered developer tools. The DPAI Arena project aims to define and maintain an open, community-driven benchmark for evaluating AI agents and IDE-embedded AI features at scale. We believe that the benchmark industry is only in its early stages, and we’re planning to take part in the evolution of the field.
We’re looking for a Senior Product Manager to own the direction of DPAI Arena and help define what “good evaluation” looks like and how we measure it.
- Refine our answer to the question of what makes an evaluation task a representative data point for software engineers and IDE AI features, using usage data, feedback loops, and community input.
- Define and iterate the evaluation methodology, designing tasks, specifying metrics and success criteria, and covering real developer workflows from both product and UX perspectives.
- Build and maintain relationships with external vendors, platform partners, outsourcing providers, and internal stakeholders to expand participation in DPAI Arena and drive both business and ecosystem development.
- Curate and manage datasets, coordinating contributions from JetBrains teams, external partners, and outsourcing vendors (including crowdsourced workers when needed) to keep evaluation tasks high-quality, representative, and diverse.
- Own the evaluation standard and roadmap, representing DPAI Arena internally and externally while aligning plans with the broader JetBrains AI and IDE strategy.
- Proven experience in product management (developer tools or data-driven products is a plus).
- Experience working on GenAI features for developer tooling.
- Strong analytical skills, with the ability to define meaningful metrics, interpret results, and turn them into product design.
- Experience working with multiple stakeholders: internal teams, external partners, and vendors (including outsourcing providers).
- Understanding of developer workflows, IDEs, and AI-assisted development, and the ability to translate real-world coding tasks into evaluation workloads.
- Strong communication skills, both written and spoken.
- Ownership mindset and proactive approach.
- Engineering background (particularly SWE or QA for tech products).
- Experience maintaining or curating open-source products.
- ML experience.
- Familiarity with outsourcing or crowdsourcing platforms, data collection pipelines, or community-driven ecosystems.
#LI-HYBRID
#LI-AA1
We process the data provided in your job application in accordance with the Recruitment Privacy Policy.
Top Skills
AI
Data-Driven Products
Developer Tools
Genai
Ides
Machine Learning
Quality Assurance
Software Engineering
Similar Jobs
Artificial Intelligence • Cloud • Information Technology • Sales • Security • Software • Cybersecurity
As an Associate Detection & Response Analyst, you will investigate security events and work with the Incident Response team to analyze incidents, write reports, and enhance threat detection services.
Top Skills:
LinuxScriptingSecurity ToolsWindows
Artificial Intelligence • Cloud • Information Technology • Sales • Security • Software • Cybersecurity
The Revenue Operations Analyst will enhance operational efficiency, analyze metrics, improve processes, and support sales initiatives to drive revenue growth.
Top Skills:
ClariGoogle SheetsPower BISalesforceTableau
Artificial Intelligence • Cloud • Information Technology • Sales • Security • Software • Cybersecurity
As a Detection & Response Analyst, you will investigate security events, conduct threat hunting, and collaborate with the Incident Response team to identify and document security incidents.
Top Skills:
CodingLinuxScriptingThreat IntelligenceUser Behavior AnalyticsWindows
What you need to know about the London Tech Scene
London isn't just a hub for established businesses; it's also a nursery for innovation. Boasting one of the most recognized fintech ecosystems in Europe, attracting billions in investments each year, London's success has made it a go-to destination for startups looking to make their mark. Top U.K. companies like Hoptin, Moneybox and Marshmallow have already made the city their base — yet fintech is just the beginning. From healthtech to renewable energy to cybersecurity and beyond, the city's startups are breaking new ground across a range of industries.
