Senior Manager – GenAI Performance & Analytics

Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities.

Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction.

Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau.

Design frameworks to evaluate the response quality of LLM responses across dimensions such as:

• factuality and groundedness

• Relevance to user intent

• Instruction Adherence

• tone and customer experience

• safety and compliance

• Build scalable processes for LLM-based automated evaluation and human review workflows.

• Identify systematic failure patterns in AI responses and recommend improvements.

Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities.

Design and analyze experiments to measure the impact of model improvements.

Generate insights that inform model training data, prompts, and product features.

Identify trends in customer interactions to uncover:

• new AI automation opportunities

• gaps in intent coverage

• common customer pain points

• Translate analytics insights into actionable recommendations for product and AI teams.

Lead and mentor a team of data analysts focused on AI performance and conversational analytics.

Establish best practices for analytics, experimentation, and AI evaluation.

Collaborate closely with product, AI research, engineering, and customer success teams.

7–10 years of experience in data analytics, data science, or AI analytics.

Experience managing or mentoring analytics teams.

Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.

Strong SQL and data analysis skills.

Experience with BI tools such as Tableau or similar platforms.

Experience using Python for data analysis or experimentation.

Familiarity with machine learning or large language models (LLMs).

Experience evaluating AI or machine learning model performance.

Experience working with conversational AI platforms or chatbots.

Experience with experimentation frameworks and A/B testing.