Senior Production Engineer - Senior Vice President

The Production Engineer is a pivotal role within Citi's Technology organisation, responsible for designing, building, and operating the intelligent systems that underpin our global production environment. This is an engineering-first position at the intersection of software craftsmanship, AI-native development, and large-scale distributed systems. 

As part of a multi-year transformation journey, the successful candidate will help define what production engineering looks like in an era of autonomous agents, generative AI, and self-healing infrastructure. You will be expected to write production-grade code daily, design agentic workflows, and contribute meaningfully to the evolution of our AI engineering practices across Citi's technology hub. 

The role requires a comprehensive understanding of multiple areas within a function and how they interact to achieve the objectives of the function. Applies in-depth understanding of the business impact of technical contributions. Accountable for delivery of a full range of end-to-end projects. 

Excellent communication skills required to negotiate internally. Involved in short- to medium-term planning of actions and resources for own area. 

Responsibilities: 

• Designs, develops, and maintains production-grade software systems with a strong emphasis on reliability, scalability, and operational excellence across Citi's global technology estate. 

• Architects and implements agentic AI workflows — building autonomous systems that can reason, plan, and act across production environments with minimal human intervention. 

• Applies advanced prompt engineering techniques to integrate large language models (LLMs) into operational tooling, incident response pipelines, and developer productivity platforms. 

• Leads the development of AI-native observability solutions — leveraging intelligent agents to detect anomalies, predict failures, and automate remediation before issues impact end users. 

• Writes clean, well-tested, and well-documented code across the full stack; champions engineering best practices including code review, pair programming, and test-driven development. 

• Drives Continuous Delivery and Automation efforts across supported applications by means of Root Cause Analysis reviews, knowledge management, performance tuning, and user training. 

• Operates and evolves CI/CD pipelines, Infrastructure-as-Code tooling, and GitOps workflows to support rapid, safe delivery of software at scale. 

• Collaborates with platform, data, and product engineering teams to embed AI capabilities into the production lifecycle — from deployment to decommission. 

• Implements the Agile Framework through one of its implementations (SCRUM or Kanban) and ensures it integrates with overall organisation processes. 

• Operates within a highly regulated financial environment, maintaining in-depth understanding of compliance requirements and their implications for system design and data handling. 

• Coaches and mentors team members on AI engineering practices, prompt design patterns, and agentic system architecture — fostering a culture of continuous learning and technical excellence. 

• Avidly communicates progress and project status across the organisation and ensures that stakeholders are managed appropriately throughout the execution period. 

• Fosters a culture that promotes transparency and innovation for increased team productivity. 

Qualifications:

• Demonstrable experience in a critical software engineering or production engineering role with high business impact and a strong programming foundation (Java, Python, Go, or equivalent). 

• Hands-on experience with AI/ML engineering — including working with LLM APIs (OpenAI, Anthropic, Gemini, or open-source equivalents), embedding models, and vector databases. 

• Proven expertise in prompt engineering: designing, iterating, and evaluating prompts for production use cases including classification, summarisation, code generation, and autonomous decision-making. 

• Experience designing and deploying agentic systems using frameworks such as LangChain, LangGraph, AutoGen, CrewAI, or equivalent — including multi-agent orchestration and tool-use patterns. 

• Excellent engineering skills and strong understanding of Software Development Lifecycle, GitOps, and modern DevSecOps practices. 

• Excellent working knowledge of key computer science concepts (networking, operating systems, virtualisation, containerisation, etc.). 

• Polyglot full-stack developer mentality and ability to pick up new languages and skills. 

• Excellent debugging and analytical skills: ability to isolate root cause across networking/infrastructure, application, and database stacks. 

• Operational experience of deploying and running services at scale on top of Docker/Kubernetes stack and a service mesh (Istio or equivalent) is highly desirable. 

• Operational experience with orchestration tools for CI/CD and Infrastructure-as-Code tooling (Terraform, CloudFormation, Pulumi, etc.) is highly desirable. 

• Experience of delivering software using Agile delivery methodologies is a must (SCRUM/Kanban). 

• Operational experience of using middleware technologies (MQ, Apache Kafka, etc.) to run services at scale is desirable. 

• Strong experience with end-to-end observability stacks (Datadog, AppDynamics, Dynatrace, etc.) is desirable. 

• Degree in Computer Science, Mathematics, Physics, or a related technical subject is desirable. 

• Experience of senior stakeholder management. 

• Consistently demonstrates clear and concise written and verbal communication skills. 

• Ability to operate in a global environment with on-/near-/off-shore matrix reporting structures. 

Human Qualities & Soft Skills:

Beyond technical capability, the Production Engineer who will thrive in this role brings a distinct set of human qualities that amplify their engineering impact and elevate those around them. 

• Learnability — Rapidly acquires new skills, frameworks, and paradigms. In a field evolving as fast as AI engineering, the ability to learn is the most durable skill of all. 

• Teachability — Receives feedback with openness and intellectual humility. Actively seeks mentorship and applies guidance to accelerate growth. 

• Flexibility & Adaptability — Thrives in ambiguity. Pivots gracefully when requirements shift, technology evolves, or priorities change — without losing momentum or quality. 

• Engineering Mindset — Approaches every problem systematically: decomposing complexity, forming hypotheses, and validating solutions with rigour and precision. 

• Product-Minded Thinking — Understands that code serves users and business outcomes. Balances technical elegance with pragmatic delivery and user impact. 

• Collaborative Spirit — Builds trust across disciplines — engineering, product, operations, and leadership. Elevates the team's collective output through generosity and clear communication. 

• Intellectual Curiosity — Asks "why" before "how". Explores the edges of what's possible with AI and production systems, driven by genuine fascination rather than obligation. 

• Ownership & Accountability — Takes end-to-end responsibility for what they build. Does not hand off problems — follows through from design to deployment to post-incident review.