Member of Technical Staff - Evaluations

Eval design is where we need you to be unusually strong: you can take a high-level description of a capability and turn it into an evaluation that is accurate, general-purpose, separates good models from bad ones, and is hard to game.

Evals you have built before that we can recognize or check, and that hold up on coverage, reproducibility, contamination handling, and the signal they were meant to give. At least one built from scratch: a benchmark, an eval paper, or real contributions to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.

Fluency with the fundamentals: contamination, saturation, judge bias, and variance.

Hands-on experience running and managing evals in a team, whether on an eval team, a training team, or in an engineering/DevOps role.

The habit of debugging unexpected scores by reading individual generated answers before believing the number.

Software engineering strong enough to own eval infrastructure and debug the inference and infra failures that surface when running evals at scale, without hand-holding.

Model training experience.

Publications or research background (industry backgrounds are equally welcome).

Deep systems expertise.

Familiarity with what small edge models do and don't do well.

An eval-specific title is not needed: training-team or DevOps profiles who ran a lot of evals are a good fit.

Internal evals cover every main target capability and drive post-training decisions, so that model decisions in post-training cite an internal eval you built or hardened rather than an external benchmark.

The eval suite runs reliably at scale and is co-owned, with other teams coming to you for evals for their own models.

At least one open-source eval released with a paper, plus a failure-mode report on a popular evaluation, with Liquid's edge-model evals becoming an external reference.