AI Principal Software Engineer (Remote)

Position Summary: 

The AI CoE Principal Software Engineer will report to the Sr. Director, AI Center of Excellence, in the Technology Department. This role is the most senior individual contributor on the team and the technical owner of the enterprise AI engineering foundation, including the semantic router, model and harness hosting, the enterprise AI gateway, and the AI operations layer used across the business. The role is responsible for setting AI engineering architecture and standards and then building them, carrying a multi-year capability roadmap from discovery through prototype, validation, and production scale. Primary duties include designing and building the shared AI platform and semantic routing layer; product inference and personalization, partnering with AI-Ops on monitoring, evaluation, and governance guardrails; building secure, compliant AI integration patterns for a regulated financial services and health and benefits environments; leading agentic pipeline and multi-agent orchestration work; delivering production AI services for CoE-enabled business functions; developing reusable reference architectures for other engineering teams; managing AI unit economics; prototyping emerging AI capabilities; mentoring senior and lead engineers; and partnering with security, risk, compliance, and data governance. This role does not have direct reports. 

Duties & Responsibilities: 

• Architect, build, and operate the shared enterprise AI platform, including core semantic data, model, harness, and hosting capability used across developer and in-product personalization use cases. 

• Design and maintain intelligent request routing using semantic analysis and deterministic logic to direct requests to the best-fit LLM, internal tool, or agentic pool, using cost, latency, and privacy as routing criteria. 

• Partner with AI-Ops to enable metric-based reporting, quality, performance, and drift monitoring across live systems, build model evaluation harnesses, and establish governance guardrails for safe AI use at scale. 

• Design and implement secure, compliant AI integration patterns appropriate for a regulated financial services environment. 

• Build and scale agentic pipelines and multi-agent orchestration across the software development lifecycle, from pilot through measured engineering capacity across product teams. 

• Deliver production AI services supporting document intelligence and claims adjudication, conversational and voice self-service, call intelligence, fraud and anomaly detection, and regulatory monitoring. 

• Develop reusable reference architectures and standard patterns so other engineering teams and embedded consultants can build AI solutions without rebuilding the foundation. 

• Evaluate unit economics for AI workloads, including cost per transaction, token and inference spend, latency budgets, and build-versus-buy and self-host-versus-vendor decisions. 

• Identify and prototype emerging AI capabilities relevant to the business and technology roadmap, including fast, focused feasibility spikes with clear recommendations. 

• Lead hands-on training sessions and pair directly with engineering teams on new development methods and prototype or pattern adoption. 

• Monitor the AI landscape and translate findings into roadmap recommendations on what to pilot, what to watch, and what to decline. 

• Set technical direction and standards for embedded AI consultants and the coaching network to scale AI delivery capacity beyond the CoE. 

• Mentor senior and lead engineers across business units, review AI designs, and serve as the escalation point for complex agentic and integration issues. 

• Partner with security, risk, compliance, and data governance teams to support safe and compliant AI delivery. 

• Perform other duties as assigned. 

Education & Experience: 

Minimum required: 

• Bachelor of Science Degree in Computer Science. 

• AI Practitioner Certifications are a plus  

• 12+ years building and operating production software, including 4+ years delivering production AI/ML systems and 2+ years of LLM or agentic systems in production, with a track record as the most senior technical owner of an enterprise-scale platform. 

Skills & Abilities: 

• Architecting and operating production agentic workflows as state machines, including branching logic, retry and recovery paths, and precise control over multi-step process flow. 

• Deep fluency with AI-assisted development environments such as GitHub Copilot, Claude Code, or ONA, with a track record of driving adoption and measured impact across engineering teams. 

• Semantic routing and model orchestration, including embedding-based intent analysis, deterministic fallback logic, and policy-based model selection across multiple providers on cost, latency, and privacy criteria. 

• Observability and AI Ops for non-deterministic systems, including distributed tracing, behavioral analysis, usage and quality metrics, drift detection, and cost and latency telemetry for executive reporting. 

• Security-aware engineering with demonstrated experience handling PII and sensitive data, including encryption, secrets management, least-privilege access, and audit trails in a regulated environment. 

• Model hosting and serving experience, including deploying and operating commercial and open-weight models behind a gateway or proxy, with sound self-host-versus-vendor judgment. 

• Evaluation and reliability engineering for probabilistic systems, including eval harnesses, golden datasets, offline and online scoring, guardrails, red-teaming, hallucination mitigation, and human-in-the-loop review design. 

• Technical leadership without formal authority, including architecture decision records, design reviews, mentoring senior and staff engineers, and communicating trade-offs to executive stakeholders. 

• Ability to own a multi-year capability roadmap, moving work from discovery through prototype, production validation, and scale against defined business metrics. 

• Ability to move fluidly between abstract system thinking and tactical, incremental delivery while managing stakeholders and delivering against a critical path. 

• Financial services, healthcare benefits, or other regulated industry experience preferred, with HSA/FSA, claims, custody, or payments domain exposure especially valuable. 

• Knowledge of AI governance and compliance frameworks preferred, such as NIST AI RMF, ISO/IEC 42001, model risk management, SOC 2, or HIPAA control environments. 

• Experience with rapid prototyping and feasibility assessment in an R&D or innovation capacity preferred. 

• Experience facilitating technical learning sessions or pairing with engineers at varying skill levels preferred. 

• Multi-agent system design experience preferred, including role assignment and coordination, agent-to-agent conversation patterns, and code execution loops for iterative problem solving. 

• API and integration experience across REST, event-driven, and MCP patterns preferred, including integrating AI into systems of record and workflow platforms. 

Other Requirements: 

• Ability to occasionally work outside normal business hours to meet project deadlines as needed. 

• Prolonged periods of sitting at a desk and working on a computer.