How does this skill help with sample size calculation?

It guides you through setting a baseline conversion rate, minimum detectable effect (MDE), statistical significance, and power to determine exactly how many users are needed before you start.

Can I use this to design tests for small user bases?

Yes, the skill helps you determine if a test is viable; if your sample size is too low for your desired effect, it will recommend increasing the MDE or finding alternative validation methods.

What is the primary benefit of the Experiment Design skill?

It ensures your product tests are statistically sound by providing structured templates for hypotheses, sample size calculations, and metric selection, preventing inconclusive or misleading results.

Why does the skill emphasize guardrail metrics?

Guardrail metrics prevent 'false wins' where a primary metric improves (like signups) but a crucial secondary metric (like long-term retention) is negatively impacted.

Experiment Design & A/B Testing

Name: Experiment Design & A/B Testing
Author: assimovt

byassimovt

•

Analytics & Monitoring

Designs rigorous, hypothesis-driven product experiments and A/B tests using scientific methodology to ensure actionable results.

The Experiment Design skill empowers product teams and developers to move beyond guesswork by implementing a structured, evidence-based approach to testing. It provides a standardized framework for drafting precise hypotheses, calculating necessary sample sizes, and establishing guardrail metrics to ensure data integrity. By enforcing best practices—such as preventing result peeking and isolating variables—the skill helps users design experiments that yield actionable, statistically significant insights for product optimization and strategic decision-making.

Key Features

01Sample size and statistical power calculation guidance

0228 GitHub stars

03Evidence-based hypothesis generation using standardized templates

04Best practice enforcement to eliminate common pitfalls like peeking

05Primary and guardrail metric definition to prevent unintended regressions

06Pre-committed analysis planning to avoid post-hoc storytelling

Use Cases

01Validating a feature change hypothesis before committing engineering resources

02Designing a split test for a new user onboarding flow

03Calculating the required duration and traffic for a conversion rate optimization test

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add assimovt/productskills experiment-design

For use in Claude.ai and ChatGPT

Download Skill