Lead Assessment Developer

Outsmart is a mission-driven, early-stage startup reimagining higher education for the age of AI by creating an entirely new collegiate experience that expands access and opportunity. Backed by $38M from top venture firms including Khosla Ventures, Forerunner, DST Global Partners, and Lightspeed, the company is led by former Duolingo executives and a founding team that also includes Emmy Award winners. With a strong culture centered around high-impact, detail-oriented team members, Outsmart is already seeing significant traction, generating 1–5 million weekly organic views on social media.

You'll manage the full assessment development lifecycle at Outsmart — from item authoring through field testing, standard setting, scoring, and reporting. You'll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments. As an AI-forward organization, you'll also help shape how AI tools are responsibly woven into our development and QA workflows.

• Master's degree (Ph.D. preferred) in educational measurement, assessment , or a related field

• 3+ years managing operational, high-stakes assessment lifecycles end-to-end

• Proven experience leading standard-setting studies and managing diverse SME panels

• Experience managing external vendors for field tests and pilot studies

• Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)

• Familiarity with cognitive complexity frameworks (DOK, Bloom's)

• Strong communicator who can translate psychometric concepts for non-technical audiences

• Bonus: knowledge of CTT, IRT, CDM, DIF,, AI-assisted item generation, or accreditation reviews

• End-to-end assessment development operations for assigned assessments

• SME network — recruitment, training, calibration, and quality

• Standard-setting studies — method selection, panel facilitation, and final reporting

• Vendor and external partner relationships, SLAs, and security protocols

• Field testing and pilot operations

• Scoring operations for constructed-response and performance-based assessments

• Project timelines, risk identification, and stakeholder communication

• Core assessment documentation — specs, blueprints, training materials, technical reports

• You thrive in ambiguity and are energized by building from scratch

• You balance psychometric rigor with startup pragmatism — you know when to ship and iterate

• You communicate with unusual clarity across technical and non-technical audiences

• You take ownership without being asked and surface risks early

• You're excited about AI's role in assessment and know where validation is non-negotiable

• You care deeply about fairness and the learners behind every score