Senior AI SDET

At Reltio®, we believe data should fuel business success. Reltio’s AI-powered data unification and management capabilities—encompassing entity resolution, multi-domain master data management (MDM), and data products—transform siloed data from disparate sources into unified, trusted, and interoperable data. Reltio Data Cloud™ delivers interoperable data where and when it's needed, empowering data and analytics leaders with unparalleled business responsiveness. Leading enterprise brands—across multiple industries around the globe—rely on our award-winning data unification and cloud-native MDM capabilities to improve efficiency, manage risk and drive growth.

At Reltio, our values guide everything we do. With an unyielding commitment to prioritizing our “Customer First”, we strive to ensure their success. We embrace our differences and are “Better Together” as One Reltio. We are always looking to “Simplify and Share” our knowledge when we collaborate to remove obstacles for each other. We hold ourselves accountable for our actions and outcomes and strive for excellence. We “Own It”. Every day, we innovate and evolve, so that today is “Always Better Than Yesterday”. If you share and embody these values, we invite you to join our team at Reltio and contribute to our mission of excellence.

Reltio has earned numerous awards and top rankings for our technology, our culture and our people. Reltio was founded on a distributed workforce and offers flexible work arrangements to help our people manage their personal and professional lives. If you’re ready to work on unrivaled technology where your desire to be part of a collaborative team is met with a laser-focused mission to enable digital transformation with connected data, let’s talk!

Reltio's AI-powered data unification and management platform—spanning entity resolution, multi-domain master data management (MDM), and data products—transforms siloed data from disparate sources into unified, trusted, and interoperable data. The Reltio Connected Data Platform™ delivers data where and when it's needed, empowering data and analytics leaders with unparalleled business responsiveness. Leading enterprise brands across industries rely on our award-winning, cloud-native MDM capabilities to improve efficiency, manage risk, and drive growth.

At Reltio, our values guide everything we do: Customer First, Better Together, Simplify and Share, Own It, and Always Better Than Yesterday. If you share these values, we invite you to join us.

We are looking for a powerhouse engineer who bridges AI development and automated validation. This is a builder-tester role — you will own the full lifecycle of AI features: from writing core Python logic for agentic workflows and tuning prompts, to architecting robust evaluation frameworks that ensure those systems are enterprise-ready.

1. → AI Development & Engineering

• →Agentic Implementation: Design and implement autonomous agents using frameworks such as LangGraph, CrewAI, or AutoGen.
• →Prompt Engineering: Own the prompt lifecycle — design, version, and tune system prompts to minimize hallucinations and maximize intent recognition.
• →RAG Pipeline Development: Build and optimize Retrieval-Augmented Generation (RAG) components, including document ingestion, chunking strategies, and vector database indexing.
• →Feature Prototyping: Rapidly prototype AI-driven features in Python to validate feasibility before full-scale integration.
• →Data Curation: Build golden datasets and synthetic data generation scripts to train and evaluate models.

1. → Advanced Validation & Quality Architecture

• →Automated Evaluation (Eval-as-Code): Build automated pipelines to measure LLM performance across metrics such as faithfulness, relevancy, and toxicity.
• →Non-Deterministic Testing: Develop strategies to test "fuzzy" outputs using LLM-assisted evaluation (using one LLM to grade another).
• →Hybrid Framework Development: Design and maintain a dual-stack automation framework covering both backend/infrastructure and AI/ML validation.
• →End-to-End Orchestration: Integrate AI tests into the MLOps pipeline so that every model deployment or prompt change triggers a full regression of the agent's reasoning capabilities.
• →Mentorship & Standards: Serve as the team's subject matter expert, defining standards for writing inherently testable AI code.

• Python: Strong proficiency with Pandas, NumPy, LangChain, and LlamaIndex.
• LLMs: Hands-on experience with OpenAI, Anthropic, or open-source models (Mistral, Llama); solid understanding of temperature, top-p sampling, and tokenization.
• Vector Databases: Experience with Pinecone, Milvus, or Weaviate.
• AI Evals & Observability: Practical experience building LLM-judge evaluation pipelines.

• Production-Grade Code: Ability to write and review complex automation suites for distributed systems.
• Architecture: Familiarity with microservices, async processing (Kafka/RabbitMQ), and RESTful APIs.
• Performance Testing: Experience with JMeter or Locust to benchmark LLM inference latency and streaming responses.

• 5+ years of experience as an SDET, QA Automation Engineer, or AI Engineer.
• 2+ years of hands-on experience testing AI/ML products.
• Experience integrating AI tools and agents into quality lifecycle management.
• Proven ability to write and review production-quality code.
• Solid background in test case design, test planning, and issue lifecycle management in a continuous deployment environment.
• Experience working in agile environments and testing highly scalable SaaS applications.