DataInterview

2.1 Intro

2.3 Key Members

2.4 Product KPI

2.7 Guardrail Metrics

2.8 Secondary Metrics

2.9 Segmentation Metrics

2.10 Launch Criteria

2.11 Experiment Data

2.12 Exercise - Amazon AB Test (Question)

2.13 Exercise - Amazon AB Test (Solution)

2.14 Code Walkthrough

3.1 State the Experiment Hypothesis

3.2 Hypothesis Statement

3.3 Hypothesis Statement - Question 1

3.4 Hypothesis Statement - Solution 1

3.5 Hypothesis Statement - Question 2

3.6 Hypothesis Statement - Solution 2

3.7 Significance Level

3.8 Significance Level - Question

3.9 Significance Level - Solution

3.10 Statistical Power

3.11 Statistical Power - Question 1

3.12 Statistical Power - Solution 1

3.13 Statistical Power - Question 2

3.14 Statistical Power - Solution 2

3.16 Minimum Detectable Effect - Question 1

3.17 Minimum Detectable Effect - Solution 1

3.18 Code Walkthrough

4.1 Design the Experiment

4.2 Randomization Unit

4.4 Sample Size Calculation - Question 1

4.5 Sample Size Calculation - Solution 1

4.6 Experiment Duration

4.7 Experiment Duration - Question 1

4.8 Experiment Duration - Solution 1

4.9 Code Walkthrough

5.1 Run the Experiment

5.2 Experiment Platform

5.3 Peeking at P-Values

5.4 Code Walkthrough

6.1 Assess Validity Threats

6.2 Stable Unit Treatment Value Assumption

6.3 Survivorship Bias

6.4 Sample Ratio Mismatch

6.5 Primacy Effect

6.6 Novelty Effect

6.7 Holiday Effect

6.8 Other Effects to Consider

6.9 AA Test

6.10 Code Walkthrough

7.1 Conduct Statistical Inference

7.2 Chi-Squared Test of Independence

7.3 Chi-Squared Test of Independence - Question

7.4 Chi-Squared Test of Independence - Solution

7.5 Z-Test for Proportions

7.6 Z-Test for Proportions - Question

7.7 Z-Test for Proportions - Solution

7.8 T-Test for Means

7.9 T-Test for Means - Question

7.10 T-Test for Means - Solution

7.11 Code Walkthrough

8.1 Decide Whether to Launch

8.2 Interpret Test Results (Relative Difference)

8.3 Launch Decisions

8.4 Decision Rule

8.5 Decision Rule with Confidence Interval

8.6 Holdout Set

8.7 Code Walkthrough

AB Testing Failed to Generalize

AB Test with Multiple Changes (UI + Algorithm)

P-Value is Higher than the Significance Level at 0.05

Statistical Parameters in AB Testing

[Amazon] Testing Recommender System

[Uber] Improving Driver Enrollment Rate