Evaluations Operations Analyst (Contract)

Our mission is to architect AI that learns from and interacts with the world like humans do.

We're pioneering the model architectures that will make this possible. Our founding team met as PhDs at the Stanford AI Lab, where we invented State Space Models or SSMs, a new primitive for training efficient, large-scale foundation models. Our team combines deep expertise in model innovation and systems engineering paired with a design-minded product engineering team to build and ship cutting edge models and experiences.

We're funded by leading investors at Index Ventures and Lightspeed Venture Partners, along with Factory, Conviction, A Star, General Catalyst, SV Angel, Databricks and others. We're fortunate to have the support of many amazing advisors, and 90+ angels across many industries, including the world's foremost experts in AI.

We are a research lab looking for an Evaluations Operations Analyst to support our evaluation operations as we scale our audio model development. You will work alongside our Evaluations Manager within the Product and Research Operations team, helping run the day-to-day operations for our globally distributed workforce.

• Serve evaluation tasks to our annotator workforce and manage assignment distribution across projects and locales.

• Track and verify completion of assignments, flagging delays or issues before they become blockers.

• Act as a first line of support for evaluators, fielding operational questions about project setup, platform access, and task instructions.

• Help maintain project trackers, assignment records, and operational documentation.

• Support calibration workflows as needed to ensure evaluation data meets our quality bar.

• Experience managing or coordinating annotator workforces, including task assignment, scheduling, and follow-up.

• Hands-on experience with data annotation platforms.

• Strong written communication skills, particularly in coordinating with globally distributed contractor workforces in different time zones and language groups.

• Ability to work independently and manage your own time effectively to meet tight project deadlines.

• Comfort working in spreadsheets, task management tools (e.g., Asana), and messaging platforms (e.g., Slack).

• Experience with linguistics or in language-related fields.

• Proficiency in languages beyond English to assist with multilingual evaluation workflows.

• Familiarity with human evaluation methodologies (e.g., scale-based ratings).

• Proficiency in using LLM assistants to move quickly without sacrificing quality.