How does this skill help with sample size calculation?

The skill uses your current baseline metric, desired Minimum Detectable Effect (MDE), and available daily traffic to calculate the exact number of users needed per variant to reach statistical power.

Can it help identify if an experiment was flawed?

Yes, it specifically checks for and flags risks such as novelty effects, seasonal confounds, and Sample Ratio Mismatch (SRM) during the results interpretation phase.

Does it support tracking multiple metrics?

The skill enforces the best practice of having one primary success metric and up to three guardrail metrics to maintain focus and prevent the 'multiple testing' problem.

What kind of recommendations does the Experiment Designer provide?

After analyzing the data, it provides a clear recommendation to Ship, Iterate, Kill, or Run a follow-up test, supported by a detailed rationale regarding significance and business impact.

Experiment Designer

Name: Experiment Designer
Author: mohitagw15856

bymohitagw15856

•

295

•

Analytics & Monitoring

Designs statistically rigorous A/B tests and interprets experiment results to drive data-driven product decisions.

The Experiment Designer skill provides a structured framework for product teams to design and analyze A/B tests with scientific rigor. It guides users through defining hypotheses, calculating required sample sizes based on Minimum Detectable Effect (MDE), and establishing success criteria before a test begins. By automating the interpretation of p-values, confidence intervals, and practical significance, this skill helps users avoid common pitfalls like 'peeking' or ignoring sample ratio mismatches, resulting in clear, defensible recommendations to ship, iterate, or kill features.

Key Features

01Risk flagging for novelty effects and sample ratio mismatch

02295 GitHub stars

03Automated sample size and run-time duration calculations

04Standardized Ship/Iterate/Kill recommendation engine

05Primary and guardrail metric framework for focused analysis

06Statistical and practical significance assessment

Use Cases

01Determining if a completed A/B test has reached statistical significance

02Defending product decisions to data scientists and engineering leads

03Designing a high-stakes experiment for a new conversion funnel

Key Features

01Risk flagging for novelty effects and sample ratio mismatch

02295 GitHub stars

03Automated sample size and run-time duration calculations

04Standardized Ship/Iterate/Kill recommendation engine

05Primary and guardrail metric framework for focused analysis

06Statistical and practical significance assessment

Use Cases

01Determining if a completed A/B test has reached statistical significance

02Defending product decisions to data scientists and engineering leads

03Designing a high-stakes experiment for a new conversion funnel