Head of Central Quality and Project Enablement

Real-world data is the competitive edge in AI.

HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery.

• Turn customer spec documents into project-ready enablement packages: annotator guidelines, decision trees, edge-case libraries with worked examples, and onboarding walkthroughs.
• Build qualification tests and gold-standard sets that confirm annotators understand the spec before they touch production work.
• Run pilot and calibration rounds at project kickoff to surface spec ambiguities early. Resolve those ambiguities with the customer and delivery lead before scaling up.
• Maintain versioned guidelines throughout the project. Roll out updates as new edge cases appear, and confirm the workforce has absorbed the changes.

• Design the quality plan for every project. Choose the review structure (gold tasks, overlap/consensus, multi-stage review, expert adjudication) based on the task type, risk, and budget.
• Set the sampling methodology. Size samples to hit target confidence levels, stratify by class and difficulty, and adjust review rates for each annotator based on performance.
• Select the right metrics for each task, such as accuracy against gold, agreement measures like κ or α, or per-class error rates. Set acceptance thresholds that fit how subjective the task is.
• Configure these workflows in our labeling platform, and turn what works into reusable quality playbooks by task type.

• Own final quality sign-off. No delivery ships without meeting its agreed acceptance criteria.
• Monitor quality throughout each project. Catch drift early, run root-cause analysis on defects, and drive corrective action for individuals and for guidelines.
• Produce a clear quality report for every delivery that shows the methodology, the results, and any known limitations.

• 6+ years in quality assurance or quality operations for data labeling, human data, or ML training/evaluation data, including experience leading a quality function or team
• A proven ability to translate complex or ambiguous specs into guidelines and training that produce consistent results
• Strong applied statistics for QA: sampling design, confidence intervals, and agreement metrics, plus the ability to explain your choices to a customer
• Hands-on experience configuring review and QA workflows in an annotation platform
• Proficiency in Claude Code for quality analysis
• Crisp written communication. Your guidelines and quality reports are the product.

Nice to have: Experience with LLM, RLHF, or preference-data projects; experience with expert or domain-specialist workforces; a background in instructional design; familiarity with Label Studio; experience with model-assisted QA.

• 90 days: A standard enablement package and quality plan template is in use on every new project. You own sign-off on all active deliveries.
• 6 months: Annotators ramp to target accuracy faster, mid-project guideline churn drops, and rework and customer-reported defects are measurably down.
• 12 months: Quality reports and methodology are a selling point for Data Services, and the playbooks are in place so the function can scale beyond you.