Senior Product Manager

ItalyRemoteFull-timesenior

Product ManagementSenior Product Manager

0 views0 saves0 applied

Apply Now

Quick Summary

Requirements Summary

Demonstrated experience owning a quality, evaluation, experimentation, or ML measurement product with its own roadmap, users, and measurable outcomes.

Technical Tools

Product ManagementSenior Product Manager

Lead the product strategy for evaluating and improving the quality of AI-generated advertising creative at scale.
You’ll own the full Evals & Quality product area, from defining what “good” means to building reliable systems that measure it.
The role combines product discovery, measurement methodology, human and automated evaluation, and customer-facing commercial enablement.
You’ll work closely with AI and agent teams to make quality metrics an integral part of how new capabilities are developed and launched.
A key focus will be creating scalable evaluation loops, representative datasets, and judging frameworks for a highly subjective creative domain.
You’ll also translate quality evidence into customer-facing assets, helping demonstrate product capabilities through RFPs, enablement materials, and public evaluations.
This is a remote opportunity for an experienced product leader who enjoys ambiguity, measurable outcomes, fast iteration, and AI-powered products.

Own the overall charter for Evals & Quality, defining the long-term problem, success metrics, priorities, and clear scope boundaries.
Define and improve core quality metrics, including first-pass-right performance, evaluation coverage, and regression detection.
Own the methodology for evaluating creative quality, including quality rubrics and calibration processes that maintain alignment between automated judges and human reviewers.
Design and manage scalable human quality-control processes, determining where human review adds value and how those judgments can improve automated evaluation systems.
Own the golden dataset, including its curation, representativeness across formats and client contexts, versioning, and ongoing maintenance.
Build reusable evaluation loops that enable product and engineering teams to evaluate new capabilities independently.
Manage the public evaluation experience as a product, maintaining a high standard of usability, credibility, and design quality.
Provide quality evidence for commercial activities, including client RFP responses and creative enablement materials.
Partner with Agent Architects to establish client-specific quality standards through configurable rubrics and methodologies rather than bespoke evaluation code.
Lead the complete product lifecycle, from discovery and hypothesis definition through launch and post-launch review.
Establish a consistent practice of documenting hypotheses before releases and reviewing outcomes approximately 14 days after launch.
Use customer evidence and product data to influence decisions beyond the immediate product team and continuously improve evaluation practices.

Requirements

~2 min read

Demonstrated experience owning a quality, evaluation, experimentation, or ML measurement product with its own roadmap, users, and measurable outcomes.
Proven experience defining and measuring quality in a subjective or complex domain where there is no single objectively correct answer.
Hands-on understanding of golden datasets, evaluation rubrics, judge calibration, human review processes, and inter-rater reliability.
Ability to identify weaknesses in measurement systems and develop practical approaches to improve the reliability and scalability of evaluation signals.
Strong product discovery skills, including the ability to understand customer needs, learn unfamiliar domains, test hypotheses, and make evidence-based product decisions before committing to solutions.
Experience owning metrics that have influenced decisions outside your immediate team or function.
Comfortable working directly with clients, including presenting methodologies, explaining evidence, and responding to questions from buyers and stakeholders.
Strong product lifecycle management skills, from discovery and prioritization through launch, measurement, and post-launch learning.
Ability to work effectively with technical, AI, data, design, and customer-facing teams in a fast-moving environment.
Strong ownership mindset, with a focus on measurable outcomes rather than simply delivering features or outputs.
Comfortable making decisions with incomplete information, iterating quickly, and simplifying complex problems into reusable product patterns.
Empathetic approach to users and colleagues, with a strong focus on usability, workflow quality, and attention to detail.
Genuine interest in AI-powered products, creative technology, measurement, and the evolving challenges of evaluating generative AI outputs.