Senior AI/ML Engineer: LLM & Agent Stack

Every production AI system, whether it's powering customer support, writing code, analyzing financial data, or diagnosing medical conditions, needs the same foundational infrastructure.

A way to route between models. A way to manage tools and integrate them securely. A way to orchestrate agents and enforce governance. A unified compute layer to run it all.

We're TrueFoundry, and we're building it. We're looking for a Senior AI/ML Engineer: LLM & Agent Stack to join the team.

Companies are moving beyond simple chatbots to production agentic systems. These systems route between OpenAI, Anthropic, Google, and self-hosted models. They integrate dozens of tools via protocols like MCP. They orchestrate multi-agent workflows where agents coordinate with other agents.

The infrastructure to support this doesn't exist yet. You can't just duct-tape together a few API calls and call it production-ready.

You need a control plane that handles:

• Intelligent routing with observability, cost policies, and fallback logic
• Centralized tool and MCP server management with security and lifecycle controls
• Agent orchestration with governance and guardrails
• A unified compute layer to run self-hosted models, custom tools, and agents

We've built two products to solve this:

You’ll design and own core components that enable enterprise customers to run production agentic AI safely and efficiently on TrueFoundry. This includes building robust orchestration for multi-step agents (graph/stateful workflows), model/routing logic, observability and policy enforcement (cost, data residency, rate limiting), and integrating upstream tooling like LangGraph, LangChain, vector stores, and specialized LLM runtimes.

• →

  Architect and implement scalable agent orchestration patterns (graph-based executors, state management, multi-agent coordination) for production workloads.

• →

  Own critical integrations: model adapters, LLM gateway hooks, vector DBs, tools & external APIs, and the platform’s LLMops flows.

• →

  Build and improve tracing, benchmarking and observability for LLMs and agents, token/cost accounting, latency p95, throughput, and correctness checks.

• →

  Drive design for safety/guardrails: moderation hooks, human-in-the-loop checkpoints, replayable audit trails and policy enforcement.

• →

  Mentor junior engineers, run design reviews, and improve engineering practices (testing, CI/CD, chaos testing for agents).

• →

  Work directly with strategic customers to prototype complex agentic solutions and translate them into product features.

• 4–10 years of software engineering with substantial experience building distributed systems, infra, or ML platforms.

• Deep practical experience integrating and deploying LLMs in production (RAG, retrieval, embeddings pipelines).

• Hands-on experience with agent orchestration frameworks (LangGraph / LangChain or custom agent runtimes) and stateful workflow design.

• Proven track record building observability, cost controls, and policy enforcement for production services.

• Experience building or contributing to open-source LLM orchestration tools (LangGraph, LangChain, or similar).

• Familiarity with enterprise constraints: on-prem/cloud hybrid deployments, data residency, compliance requirements.

• Background in security, privacy, or model governance for LLMs.

• Demonstrated leadership in cross-functional projects and direct customer engagement.

• Open-source contributions, architecture blogs, or public talks on agentic LLMs or LLMops.

• Examples of productizing research or shipping complex infra features.