Principal Researcher, Benchmarks, Evaluation, and Standards

The Partnership on AI (PAI) is the leading forum addressing the most important and difficult decisions on the future of AI.

• Develop PAI's point of view on the evaluation and benchmarking ecosystem, including assessing the infrastructure required for PAI to successfully develop and run its own evaluations and directing PAI’s strategy for integrating into the evaluation ecosystem, in collaboration with supervisor and executive team
• Advance the adoption of shared industry guidance through development of and integration into benchmarks, evaluations, and technical standards, in collaboration and partnership with relevant organizations in the auditing and certification space
• Offer technical leadership and shape the strategic direction of PAI’s focus areas (AI, Human Connection, and Well-Being, AI and Shared Prosperity, and Trustworthy AI Value Chains) in collaboration with those leading those focus areas and with the Chief Programs and Insights Officer.
• Support technical efforts from other PAI teams, including Policy and International
• Work with PAI’s Partnerships team to structure work to support fundraising efforts
• Consider which questions across PAI can be addressed most effectively and urgently with sociotechnical solutions that PAI is well placed to develop, and design and build implementation pathways.

• Work closely across PAI projects in order to ensure that guidance is technically feasible and implementable, e.g., as evaluations or standards, and supporting testing and sandboxing exercises related to PAI guidance where relevant
• Design approach for maintaining PAI’s benchmarking, evaluation, and standards work over time, including quality control for established partnerships
• Author/co-author peer-reviewed research papers, policy briefs, and accessible reports that translate PAI’s insights into actionable recommendations for industry, policymakers, and civil society, with an emphasis on bridging technical and social considerations. 

• Initiate and lead collaborative research partnerships with academic institutions, civil society organizations, and industry Partners to advance programmatic work.
• Represent PAI at select conferences (e.g., NeurIPS) and other venues, presenting findings and methodological innovations.
• Run point on key relationships with benchmarking, technical standards, and evaluation-oriented organizations and settings, including AISIs and assurance bodies
• Collaborate with Partnerships team to test approaches to earned-revenue models, including but not limited to licensing or content contributions to credentialing and training programs

• Translate rigorous findings into clear, compelling narratives for diverse audiences, including industry and civil society leaders, policymakers, and affected communities.
• Support strategic communications efforts to amplify and increase the adoption of PAI's guidance

• Represent work to standards bodies, policymakers, media, funders, and the public, and connect open research questions to the resourcing that can answer them.

• Eight or more years of relevant technical experience across some combination of AI, human-computer interaction, trust and safety, or another relevant interdisciplinary field.
• Fluency in both technical and sociotechnical domains, with priority given to those with advanced degrees in a technical discipline like computer science.
• Strong, accessible writing.  A large share of this role's output is documents that other organizations will adopt or reject on the strength of how they are written. Your ability to write in a compelling way for different audiences is key to the success of our goal. 
• Experience translating guidance into evaluations, benchmarks, standards, audit criteria, or informing certifications and professional training
• Demonstrated ability to interface with and convene stakeholders with genuinely conflicting interests and varied disciplinary training, and to produce something from that convening that the cohort can get behind. 
• Experience representing and speaking on behalf of an organization at events featuring executives and leaders from across domains and experiences.
• Demonstrated capacity to set your own objectives and drive multistakeholder strategy with a high degree of autonomy

• PhD in computer science, AI, or a comparably technical discipline. 
• Experience actively participating in and working with implementation partners on standards development and implementation
• Professional emphasis on the social impact of technology and experience working to apply their technical skillset to social questions. 
• Familiarity and experience with how AI systems are actually built and evaluated, including model behavior specification, red-teaming, and automated evaluation.
• Experience working on questions related to psychological well-being and/or content authenticity.
• Comfort with domestic and international travel, at times substantial.
• Comfort working in a small, evolving cross-disciplinary organization, including contributing to priorities beyond your immediate focus.

PAI's compensation philosophy reflects our commitment to fairness and equity across our distributed team. Compensation is based on role level, experience, skills, geographic location, and internal equity.

Applications will be reviewed on a rolling basis until the position is filled. To apply, please submit a cover letter and resume.

Only complete applications will be reviewed.

In your cover letter, we would welcome your view on one question the field has not resolved: how can normative guidance be better integrated with technical approaches to support impact and adoption by AI developers and deployers? Please keep your cover letter/expression to no more than 750 words. We seek to understand your orientation towards the issue as well as how you express your thinking, in your own voice.

• Justice: Notices and addresses inequities, includes diverse perspectives, and acts fairly in decisions and collaboration.
• Trust: Communicates openly, follows through on commitments, and demonstrates reliability and integrity.
• Respect: Listens fully, values others’ time and contributions, and collaborates thoughtfully.
• Compassion: Shows empathy, patience, and care, especially when supporting colleagues or navigating challenges.