Data & Analytics #2intermediate

Experiments & Decision Systems

Stop guessing, start testing. Master A/B testing, feature flags, progressive rollouts, statistical significance, multi-armed bandits, and decision frameworks like RICE and ICE. Build a culture of experimentation that turns data into better products.

What You Build

An experiment system with A/B tests, feature flags, statistical analysis, adaptive experiments, and a decision framework for your backlog

Sections

Lessons

Estimated

Concepts

Start Course

Take These First

beginnerData & Analytics

SQL & Metrics for AI Builders

Experiments & Decision Systems

You are here

experimentsab-testingfeature-flagsstatisticsdecision-frameworksricebandits

Dr. Maya Chen

lecture · formal

First 5 lessons are free (83% of the course). No login required.

Experiment Design

A/B testing, feature flags, and statistical significance.

A/B Testing Fundamentals

30 min Ship Build in Public

Ship: A/B test plan with hypothesis, sample size calculation, and success criteria

Tweet: Tweet: 'Stopped guessing, started testing. First A/B test showed our original headline was 40% worse than the variant.'

Feature Flags and Progressive Rollouts

30 min Ship

Ship: Feature flag system with percentage-based rollout and kill switch

Statistical Significance: When to Call It

30 min Ship

Ship: Statistical analysis notebook that determines when an experiment has reached significance

Decision Systems

Adaptive experiments, decision frameworks, and experiment culture.

Multi-Armed Bandits and Adaptive Experiments

30 min Ship

Ship: Adaptive experiment system that automatically shifts traffic to winning variants

Decision Frameworks: RICE, ICE, and Opportunity Scoring

30 min Ship Build in Public

Ship: Prioritized backlog scored with RICE/ICE frameworks and opportunity analysis

Tweet: Tweet: 'Scored our entire backlog with RICE. The top 3 items were not what anyone expected. Data beats intuition.'

Preview — content blurs halfway through

Building an Experiment Culture

30 min

Unlock the rest of this course

$5 for full course or $25/mo for everything

See pricing

Next Course

AI Product Management

Product management for the AI era. Learn what makes AI products different, how to write specs for AI features, user research for trust and usability, roadmap planning with AI uncertainty, measuring AI product success, and communicating AI progress to stakeholders.