A/B Testing Power Simulator for Different Sample Allocations and Uplifts

This web application explores the impact of sample size allocation and effect size (uplift) on the statistical power of A/B tests. It allows you to visualize how power changes across different uplift percentages and allocation proportions.

Simulation Results

The simulator will generate a series of subplots, each representing a different sample size. Within each subplot:

• X-axis: Displays the uplift percentage (effect size).
• Y-axis: Represents the statistical power of the A/B test at that uplift percentage.
• Lines: Each line corresponds to a different allocation proportion, showing how power changes with increasing uplift.
• Grey Dashed Line: A horizontal line at y=0.8, representing a common threshold for desired power in A/B testing.

Interpretation

Analyze the plots to understand how the power of your A/B tests is influenced by:

• Sample Size: Larger sample sizes typically yield higher power.
• Uplift Percentage: Larger uplifts are easier to detect, leading to higher power.
• Allocation Proportions: Different splits between control and treatment groups can affect power.

Use these insights to make informed decisions about sample size allocation and minimum detectable effect size for your A/B tests, ensuring you have enough power to reliably detect meaningful differences.

The application leverages Python, Flask, and matplotlib to create interactive visualizations based on your simulation configurations. Start exploring the impact of different factors on A/B testing power!