Artificial Intelligence, Technical Referent - AI Lab

Technology Scouting & Evaluation

• Run short, instrumented spikes and benchmarking on new models, tools and frameworks: LLMs, vector databases, orchestration frameworks, copilots, assistants and more.

• Compare vendor and open-source options, documenting trade-offs across quality, cost, latency, security and integration complexity.

• Deliver concise decision memos with clear recommendations: adopt, watch, or avoid.

Evaluation Harnesses & Sandboxes

• Design and maintain evaluation environments (i.e. datasets, prompts, scenarios, telemetry) to test models under realistic constraints.

• Build automation and tooling to measure quality, robustness, latency and cost, including regression tracking over time.

• Ensure every evaluated technology has benchmark coverage and a documented risk and limitations view.

Readiness Playbooks & Hand-offs

• For promising technologies, produce readiness playbooks describing recommended patterns, guardrails and integration guidelines.

• Coordinate with platformization teams to turn validated technologies into platformized capabilities.

• Track which validated items progress to platformization or pilots, and capture learnings to sharpen future bets.

Governance, Risk & Standards

• Work with Security, Legal, Compliance and other AI teams to document risk assessments, mitigations and governance recommendations for each evaluated technology.

• Maintain checklists, decision templates and lightweight standards reusable across evaluations and by partner teams.

• Incorporate learnings from third-party AI tooling already in use (e.g., external copilots, AWS AI suite) into adoption guidelines.

Collaboration, Mentoring & Community

• • Partner with other AI teams and domain teams to ensure clear boundaries and smooth collaboration.

  • Participate in hiring as a technical evaluator and culture champion.

  • Mentor engineers in the Lab and adjacent teams on evaluation methods, benchmarking and experimental design.

  • Share knowledge through internal write-ups, tech talks and occasional external meetups and conferences.

Technical depth

• 8+ years in software engineering; 3+ years working hands-on with LLMs and AI tooling.

• Strong experience with distributed systems and event-driven architectures, both synchronous and asynchronous.

• Proficiency with LangChain, LangGraph or similar orchestration frameworks, including custom tools and multi-step workflows.

• Solid knowledge of AWS infrastructure and how to run evaluation workloads in a secure, cost-aware way.

• Track record designing and running benchmarks comparing AI models and tools under real constraints.

Evaluation & decision-making

• Able to turn ambiguous "we should try this new thing" ideas into well-scoped evaluation plans with clear hypotheses and metrics.

• Comfortable making trade-off calls — quality vs. latency vs. cost vs. vendor lock-in — and documenting them clearly.

• Experience writing short, opinionated decision memos that help others move fast.

Collaboration & communication

• Can explain technical results to non-specialists in concrete, concise terms.

• Experience working with platform, product and operations teams to align evaluations with real use cases.

• Able to influence without authority, aligning teams around shared standards and guardrails.

Mindset

• Curious and biased toward experimentation, combined with disciplined measurement and risk awareness.

• Comfortable in a small, high-leverage team without embedded PMs. You structure your own work and keep stakeholders informed.

• Builder attitude: you prefer reusable tools, templates and playbooks over one-off work.