A/B Testing & Experimentation

Complete syllabus

Every chapter and lesson

1. 01

   Chapter 1 · 3 lessons

   Why Experiment at All

   1. 1.1

      Correlation vs. causation

      9 min · Quick check

      Lesson goal: By the end you can distinguish between correlation and causation in data analysis.

      • Define correlation and causation.
      • Explain the difference between correlation and causation.
      • Identify examples of correlation that do not imply causation.
      • Discuss the implications of confusing correlation with causation.
   2. 1.2

      What an A/B test really is

      9 min · Quick check

      Lesson goal: By the end you can describe the fundamental principles of A/B testing.

      • Define what an A/B test is.
      • Explain the purpose of A/B testing in experimentation.
      • Identify the components of a typical A/B test.
      • Discuss the importance of randomization in A/B testing.
   3. 1.3

      When not to A/B test

      8 min · Quick check

      Lesson goal: By the end you can recognize scenarios where A/B testing is not appropriate.

      • Identify situations where A/B testing may lead to misleading results.
      • Discuss ethical considerations in A/B testing.
      • Explain the limitations of A/B testing in certain contexts.
2. 02

   Chapter 2 · 3 lessons

   Designing a Clean Test

   1. 2.1

      Hypotheses & success metrics

      9 min · Quick check

      Lesson goal: By the end you can define hypotheses and success metrics in the context of A/B testing.

      • Understand the definition of a hypothesis in experimentation.
      • Identify key components of success metrics.
      • Differentiate between primary and secondary success metrics.
      • Formulate a clear hypothesis for an A/B test.
   2. 2.2

      Randomisation & assignment

      9 min · Quick check

      Lesson goal: By the end you can explain the importance of randomisation and assignment in A/B testing.

      • Define randomisation and its role in reducing bias.
      • Explain the concept of assignment in the context of test groups.
      • Discuss methods for implementing randomisation in experiments.
      • Understand the implications of improper randomisation.
   3. 2.3

      Guardrail metrics

      8 min · Quick check

      Lesson goal: By the end you can identify and define guardrail metrics to monitor during A/B testing.

      • Define guardrail metrics and their purpose in experimentation.
      • Identify common types of guardrail metrics.
      • Explain how guardrail metrics help in maintaining test integrity.
      • Discuss the importance of monitoring guardrail metrics throughout the test.
3. 03

   Chapter 3 · 3 lessons

   Sample Size & Power

   1. 3.1

      Effect size & MDE

      9 min · Quick check

      Lesson goal: By the end you can define effect size and minimum detectable effect (MDE) in the context of A/B testing.

      • Understand the concept of effect size and its importance in experiments.
      • Define minimum detectable effect (MDE) and its role in determining sample size.
      • Differentiate between small, medium, and large effect sizes.
   2. 3.2

      Statistical power

      9 min · Quick check

      Lesson goal: By the end you can explain statistical power and its significance in A/B testing.

      • Define statistical power and its relationship to Type I and Type II errors.
      • Discuss the factors that influence statistical power, including sample size and effect size.
      • Explain the importance of achieving adequate power in experimental design.
   3. 3.3

      Calculating sample size

      9 min · Quick check

      Lesson goal: By the end you can calculate the required sample size for an A/B test given specific parameters.

      • Understand the formula for calculating sample size based on desired power and effect size.
      • Apply the concepts of statistical power and effect size to determine sample size.
      • Practice calculating sample size using different scenarios and parameters.
4. 04

   Chapter 4 · 3 lessons

   Reading the Results

   1. 4.1

      p-values, the honest version

      9 min · Quick check

      Lesson goal: By the end you can explain the concept of p-values and their significance in A/B testing.

      • Define p-value and its role in hypothesis testing.
      • Explain the meaning of a low p-value in the context of statistical significance.
      • Discuss common misconceptions about p-values and their interpretation.
   2. 4.2

      Confidence intervals

      9 min · Quick check

      Lesson goal: By the end you can describe confidence intervals and their importance in estimating parameters.

      • Define confidence interval and its components.
      • Explain how confidence intervals provide a range of plausible values for a population parameter.
      • Discuss the relationship between sample size and the width of confidence intervals.
   3. 4.3

      Significance vs. importance

      9 min · Quick check

      Lesson goal: By the end you can differentiate between statistical significance and practical importance in A/B testing results.

      • Define statistical significance and practical importance.
      • Discuss how a result can be statistically significant but not practically important.
      • Provide examples to illustrate the difference between significance and importance.
5. 05

   Chapter 5 · 3 lessons

   Traps That Fool Smart Teams

   1. 5.1

      Peeking & early stopping

      9 min · Quick check

      Lesson goal: By the end you can explain the concepts of peeking and early stopping in A/B testing.

      • Define peeking and its implications on statistical validity.
      • Explain early stopping and its potential to introduce bias.
      • Discuss the importance of pre-defined stopping rules in experiments.
   2. 5.2

      Multiple comparisons

      9 min · Quick check

      Lesson goal: By the end you can identify the issues related to multiple comparisons in A/B testing.

      • Define multiple comparisons and its impact on Type I error rates.
      • Explain the concept of family-wise error rate.
      • Discuss methods to control for multiple comparisons in experiments.
   3. 5.3

      Novelty & sample-ratio bias

      9 min · Quick check

      Lesson goal: By the end you can describe novelty and sample-ratio bias in A/B testing.

      • Define novelty bias and its effects on experiment outcomes.
      • Explain sample-ratio bias and its implications for data interpretation.
      • Discuss strategies to mitigate novelty and sample-ratio bias in experiments.
6. 06

   Chapter 6 · 3 lessons

   From Result to Decision

   1. 6.1

      Segmenting results

      9 min · Quick check

      Lesson goal: By the end you can define segmentation in the context of A/B testing and identify key segments in your results.

      • Understand the concept of segmentation in A/B testing.
      • Identify different user segments based on behavior or demographics.
      • Analyze results within each segment to draw insights.
   2. 6.2

      Writing the readout

      9 min · Quick check

      Lesson goal: By the end you can write a comprehensive readout that effectively communicates A/B testing results.

      • Define what a readout is and its purpose in A/B testing.
      • Identify key components that should be included in a readout.
      • Learn how to present data clearly and concisely for stakeholders.
   3. 6.3

      Ship, iterate or kill

      9 min · Quick check

      Lesson goal: By the end you can determine whether to ship, iterate, or kill a product based on A/B testing results.

      • Understand the criteria for deciding to ship, iterate, or kill a product.
      • Evaluate A/B testing results to make informed decisions.
      • Learn how to communicate your decision-making process to stakeholders.