Research Associate - AI Evaluations

At Karya, we’re building AI evaluation systems that measure and understand whether AI systems actually work for the communities they’re meant to serve. We’re looking for a Research Associate to help design and run community-facing research that shapes how AI systems are evaluated. You’ll contribute to research workstreams end to end: shaping the research design, building toolkits, running fieldwork, and turning what you learn into evaluation frameworks and datasets. You’ll contribute to a range of evaluation projects across the team. These could include building a gender-bias benchmark for AI chatbots in domains like agriculture; conducting grounded research into what people in India actually use AI for, in which languages and in what contexts, to build evaluation categories from real use; or contributing to conceptual and methods work on what “community” and “participatory” should mean in AI evaluation, pressure-tested against fieldwork with Karya’s worker community. 

The role suits someone with roots in social science who has worked where technology, impact and AI meet, and who cares about gender equity and doing research with communities.

Concretely, this means:

• Contribute to research design, including research questions, methods, site selection and sampling, working closely with project leads
• Build and adapt research toolkits: discussion guides, oral consent scripts in local languages, training materials and feedback tools
• Plan and conduct focus group discussions, interviews and pilots with communities, including participant recruitment and field logistics
• Coordinate with field teams during data collection, keeping research consent-first and respectful of participants’ time and context
• Clean, cull and analyse qualitative and quantitative data through a gender and intersectional lens
• Synthesise insights into findings that strengthen evaluation frameworks, datasets and toolkits
• Contribute to desk research, reports and external outputs
• Support project management by tracking timelines, deliverables and decisions across internal teams (Ops, Tech) and external partners

• At least 2–3 years of research experience, ideally at a tech, impact or AI-focused organisation
• A degree in a social science discipline (e.g., sociology, development studies, gender studies, anthropology), or equivalent experience
• Grounding in gender and intersectionality (caste, class, region) through study, research or practice
• Experience contributing to research design and delivering research projects
• Hands-on qualitative and field research experience, including running FGDs
• Working data skills: SQL, quantitative analysis, and some coding (e.g., Python or R)
• Fluency in spoken and written Hindi
• Willingness to travel for fieldwork

• Applied AI/ML literacy: enough to engage meaningfully with model evaluation, human-in-the-loop systems and language technologies
• Socio-technical systems thinking, meaning you understand technology and society as shaping each other rather than as separate concerns
• Strong written and verbal communication in Hindi and English
• Comfort working in ambiguous, early-stage research settings

• Proficiency in Indian languages beyond Hindi
• Familiarity with agriculture or rural livelihoods
• Exposure to participatory methods 

People matter at Karya and these are some of the perks and benefits we created for our team:
• Flexible vacation and leave policy
• Flexible work options
• Insurance as per industry standards
• Access to industry stalwarts and networking opportunities

Karya is an equal opportunity and affirmative action employer.

Qualified applicants will receive consideration without regard to their race, colour, religion, sex, sexual orientation, gender identity and disability.

Karya invites all qualified interested applicants to apply for career opportunities. If you are a person with a disability and need a reasonable accommodation to use our search to apply for a career opportunity, reach out at hr@karya.in.