Data Scientist

Arena is the platform for evaluating how AI models perform in the real world. Founded by researchers from UC Berkeley's SkyLab, we're on a mission to measure and advance the frontier of AI for real-world use, and to build the foundation for everyone to understand, shape, and benefit from it.

• Design, implement, and analyze A/B tests, multi-armed bandits, and quasi-experimental methods to measure the impact of product changes.

• Apply causal inference techniques (e.g., difference-in-differences, propensity score matching, synthetic control, regression discontinuity) to estimate treatment effects in non-randomized settings.

• Collaborate with product, engineering, and marketing teams to define hypotheses, success metrics, and statistical power requirements.

• Ensure rigorous statistical validity (e.g., controlling for biases, multiple testing corrections, confidence intervals).

• Develop and refine retention measurement frameworks (e.g., cohort analysis, survival analysis, churn prediction).

• Define and track core engagement metrics (DAU, WAU, MAU, rolling retention, N-day retention) and diagnose trends.

• Identify key drivers of retention through segmentation, funnel analysis, and predictive modeling.

• Work with growth teams to optimize onboarding, engagement loops, and monetization strategies.

• Build and maintain scalable data pipelines (using PySpark, SQL, or big data tools) to process and analyze large datasets.

• Develop automated dashboards and reports (e.g., Tableau, Looker, Metabase) to monitor experiment performance and retention trends.

• Ensure data quality and consistency in metric definitions across teams.

• Optimize queries and computations for performance and cost efficiency in distributed systems (e.g., Databricks, AWS EMR, GCP BigQuery).

• Partner with product managers, engineers, and marketers to translate business questions into data-driven analyses.

• Present findings and recommendations to executive stakeholders in clear, actionable formats.

• Mentor junior data scientists and analysts on best practices in experimentation and retention analytics.

• 3+ years of experience in data science, analytics, or experimentation (or equivalent in academic research).

• Strong background in statistics and causal inference (hypothesis testing, Bayesian methods, experimental design).

• Hands-on experience with SQL and Python (Pandas, NumPy, SciPy, StatsModels, Scikit-learn).

• Proficiency in experimentation tools (e.g., Optimizely, Statsig, Eppo, or custom in-house systems).

• Experience defining and analyzing retention metrics (DAU/WAU/MAU, cohort retention, churn).

• Familiarity with big data tools (PySpark, Hadoop, or similar distributed computing frameworks).

• Expertise in PySpark for large-scale data processing and analytics.

• Experience with time-series forecasting, survival analysis, or uplift modeling.

• Knowledge of ML for retention (e.g., propensity models, clustering, recommendation systems).

• Experience with data visualization tools (Tableau, Looker, Plotly, Matplotlib/Seaborn).

• Background in growth analytics, product analytics, or marketing analytics.

• Advanced degree (MS/PhD) in Statistics, Economics, Computer Science, or a quantitative field.

• Experience with reinforcement learning or bandit algorithms for dynamic experimentation.

• Knowledge of MLOps or productionizing models (e.g., MLflow, Airflow, Docker).