Designs and implements statistically valid A/B tests to optimize conversion rates and product performance.
The A/B Test Setup skill provides a comprehensive framework for running rigorous experiments within your development workflow. It guides users through the entire experimentation lifecycle, from formulating data-driven hypotheses and calculating required sample sizes to selecting guardrail metrics and analyzing statistical significance. By enforcing best practices like testing single variables and avoiding the 'peeking problem,' this skill ensures that your optimizations are based on reliable data rather than intuition, helping you make informed decisions about UI changes, copy variations, and feature rollouts.
Key Features
01Post-test analysis checklists to interpret significance and confidence intervals
020 GitHub stars
03Sample size and duration calculation guides for statistical rigor
04Traffic allocation and variant design best practices
05Hypothesis framework for creating specific, measurable experiment goals
06Metric strategy for primary, secondary, and guardrail measurement
Use Cases
01Planning a multivariate test for a new landing page layout
02Evaluating experiment results to determine if a UI change should be permanently implemented
03Designing a split-test for a high-traffic pricing page to increase conversions
