When should I avoid running an A/B test?

Avoid A/B testing when traffic is too low to reach statistical significance in a reasonable timeframe, when there are ethical concerns, or when changes are foundational and affect the entire system.

Why is 'peeking' at results considered a pitfall?

Peeking at results before the pre-calculated sample size is reached significantly increases the risk of 'Type I' errors (false positives), leading to incorrect conclusions.

How does this skill calculate A/B test sample sizes?

The skill uses statistical parameters including baseline conversion rate, minimum detectable effect (MDE), statistical power (typically 80%), and significance level (typically 95%) to determine the required user count.

What is a guardrail metric in A/B testing?

A guardrail metric is a secondary measure used to ensure that while your primary metric improves, you aren't accidentally harming other critical areas of the business or user experience.

A/B Test Design

Name: A/B Test Design
Author: sethdford

bysethdford

•

营销自动化

Designs rigorous A/B experiments with statistically sound hypotheses, metrics, and sample size calculations to drive data-informed product decisions.

The A/B Test Design skill empowers developers and product teams to conduct scientifically valid experiments within their workflow. It provides a structured framework for defining clear hypotheses, isolating variables between control and treatment groups, and selecting primary and secondary metrics. By incorporating statistical rigor—including sample size and duration calculations based on power and significance—this skill helps users avoid common pitfalls like 'peeking' at results or running underpowered tests, ensuring that every experiment produces actionable and reliable insights.

主要功能

01Statistical power and sample size calculation based on baseline conversion and MDE

025 GitHub stars

03Anti-pattern detection for common pitfalls like novelty effects and segmentation errors

04Experimental duration planning based on weekly cycles and traffic

05Structured hypothesis generation following the 'If-Then-Because' framework

06Primary, secondary, and guardrail metric identification

使用场景

01Validating the impact of new feature rollouts using controlled treatment groups

02Measuring the effect of backend performance improvements on user engagement metrics

03Optimizing conversion funnels by testing specific UI variations against a baseline

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add sethdford/claude-skills a-b-test-design

For use in Claude.ai and ChatGPT

主要功能

01Statistical power and sample size calculation based on baseline conversion and MDE

025 GitHub stars

03Anti-pattern detection for common pitfalls like novelty effects and segmentation errors

04Experimental duration planning based on weekly cycles and traffic

05Structured hypothesis generation following the 'If-Then-Because' framework

06Primary, secondary, and guardrail metric identification

使用场景

01Validating the impact of new feature rollouts using controlled treatment groups

02Measuring the effect of backend performance improvements on user engagement metrics

03Optimizing conversion funnels by testing specific UI variations against a baseline