Senior SWE - Python / TypeScript

Key Responsibilities

• Rubric Development: Design, refine, and iterate on rubrics, evaluation frameworks, and quality criteria for preference data.
• Data Review and Quality Assurance: Review, assess, and label data while identifying subtle inconsistencies, quality issues, and opportunities for improvement.
• Engineering Infrastructure: Build, maintain, and optimize pipelines and infrastructure supporting data generation, collection, processing, and evaluation workflows.
• Technical Documentation: Synthesize findings from data analysis and evaluation activities into concise reports, updates, recommendations, and actionable summaries.
• Cross-Functional Collaboration: Work closely with researchers and engineers to translate qualitative judgments into structured, repeatable, and scalable processes.
• Codebase Navigation: Understand, contribute to, and maintain complex software repositories while ensuring code quality, reliability, and performance.
• Continuous Improvement: Identify workflow inefficiencies and implement practical improvements to data quality, evaluation consistency, and engineering processes.

Required Skills and Qualifications

• At least 5 years of professional software engineering experience with Python, particularly in building and maintaining pipelines and infrastructure, or with TypeScript, JavaScript, or Go.
• Hands-on experience with preference data collection, rubric creation, or developing evaluation criteria.
• Strong understanding of software engineering principles, code quality, debugging, and maintainable system design.
• Excellent written and verbal communication skills, with the ability to explain complex technical concepts clearly and concisely.
• Strong analytical skills and the ability to make consistent, well-reasoned quality judgments.
• Demonstrated ability to explain why a particular solution, example, or implementation meets a high quality standard.
• Comfort working independently in ambiguous environments while prioritizing execution and timely delivery.

Preferred Qualifications

• Experience contributing to or navigating large-scale, complex open-source codebases, such as VS Code.
• Experience building data pipelines, developer infrastructure, or automated evaluation systems.
• Background in AI model evaluation, preference data workflows, data annotation, or quality assessment.
• Experience collaborating with AI researchers, machine learning engineers, or multidisciplinary technical teams.
• Education from a leading university or professional experience at a highly regarded technology company.

Evaluation and Selection Process

Candidates will undergo the following evaluation stages:

1. Application and Screening: Submit an application and complete the required screening questions.
2. AI Interview: Complete an AI-based interview.
3. Coding Challenge: Demonstrate technical proficiency through a coding assessment.
4. Technical and Cultural Interview: Participate in an approximately 30-minute discussion covering technical expertise, communication, judgment, and collaboration.
5. Final Review: Complete the final candidate evaluation and selection process.

Additional Requirements

• A LinkedIn profile is mandatory and must be included in the candidate's resume.
• Candidates from leading technology companies and top-tier universities are strongly preferred.
• Candidates must not currently be employed by Microsoft.
• Applicants should be able to commit to the required weekly hours and PST overlap for the full contract duration.