AI Engineer – Trust & Explainability (AI Platform)

United StatesRemoteFull-timemid

Machine Learning EngineerData

3 views0 saves0 applied

Apply Now

Quick Summary

Overview

This position is listed on behalf of a partner company, who manages all applications and next steps.

Technical Tools

Machine Learning EngineerData

This is a hands-on AI engineering role focused on making multi-agent systems observable, explainable, testable, and trustworthy.
You will help build shared platform capabilities for tracing agent workflows, evaluating non-deterministic outputs, and surfacing meaningful explanations.
The role sits at the intersection of LLM engineering, distributed systems, observability, application security, and responsible AI.
You will work closely with product engineers, architects, security teams, and other AI engineers to build reusable platform primitives.
Your work will help engineers understand why agents behave as they do and enable customers to make more informed decisions about AI-generated outcomes.
You will have the opportunity to evaluate and extend open-source frameworks while building new capabilities where existing tooling falls short.
The environment is highly technical and collaborative, with an emphasis on continuous learning, AI-assisted development, rigorous testing, and practical engineering.

Build end-to-end tracing across AI platform components, including gateways, orchestration, memory, tools, model calls, agent handoffs, parallel branches, and retries.
Develop correlation capabilities that connect actions across multiple agents into coherent, readable workflow traces.
Build developer-facing trace and debugging experiences that allow engineers to understand complete agent interactions.
Develop explanation capabilities that transform raw trace information into human-readable accounts of what an agent did, what information it relied on, and why it followed a particular path.
Create platform primitives for customer-facing trust, including explanation records, confidence and provenance metadata, and summaries of information used by agents.
Partner with product engineering teams to integrate trust and explainability capabilities into production AI applications and iterate based on feedback.
Evaluate and integrate open-source observability, tracing, and evaluation frameworks, extending them when existing capabilities do not meet platform requirements.
Build missing trust and explainability capabilities and contribute useful fixes or extensions to open-source projects where appropriate.
Develop and maintain evaluation tooling covering golden datasets, test runners, scoring pipelines, regression reporting, and model or prompt comparisons.
Create processes for generating and versioning evaluation datasets using de-identified traffic and synthetic scenarios.
Build automated quality checks for model, prompt, and tool changes so regressions are identified before production release.
Develop adversarial and red-team testing for prompt injection, jailbreaks, tool misuse, and data-exfiltration risks.
Build automated tenant-isolation tests to ensure agents cannot access another customer's data through memory, retrieval, tools, or model context.
Collaborate with Security Operations to incorporate threat models and emerging attack patterns into platform testing.
Participate in design discussions and code reviews, support onboarding of junior AI engineers, and contribute documentation that reduces tribal knowledge.

Requirements

~2 min read

3+ years of professional software engineering experience delivering production features independently.
Bachelor's degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience.
Strong foundations in algorithms, data structures, software design, and modern engineering practices.
Production-level proficiency with Python; TypeScript experience is a plus.
Hands-on experience building applications that integrate LLMs, such as LLM APIs, agent frameworks, RAG pipelines, or comparable systems, either professionally or through substantial personal or open-source work.
Active daily use of AI-assisted software development tools and an interest in applying them effectively to engineering workflows.
Experience with Git, Docker, automated testing, and modern scripting or development tooling.
Hands-on experience in at least one area such as LLM observability and tracing, LLM evaluation and testing, agent frameworks and multi-agent orchestration, or application security testing, with an interest in developing broader expertise.
Experience with distributed tracing or observability tooling in a production environment.
Strong automated testing practices, particularly for systems where outputs can vary between runs.
Experience running workloads on AWS or Azure, including foundational knowledge of identity and access management, networking, and secrets management.
Strong analytical and problem-solving skills, with the ability to work independently while knowing when to seek guidance on complex system design.
Clear written and verbal communication skills and the ability to collaborate effectively across engineering, product, architecture, and security teams.
Preferred experience with OpenTelemetry, GenAI semantic conventions, OpenLLMetry, or comparable tracing standards.
Familiarity with LLM observability and evaluation platforms such as Langfuse, Arize Phoenix, LangSmith, Braintrust, promptfoo, DeepEval, or equivalent tools.
Experience contributing to open-source AI observability, evaluation, or agent-framework projects is a plus.
Experience with AWS Bedrock, Azure OpenAI, or other cloud-managed model services is beneficial.
Experience building multi-tenant SaaS platforms where tenant isolation is a critical security requirement is preferred.
Experience in financial services, fintech, or another regulated environment where explainability influenced technical decisions is advantageous.
Experience building developer-facing debugging, monitoring, or visualization tools is a plus.