Can I use this skill for multivariate tests (MVT)?

Yes, the skill supports various test types including A/B, A/B/n, and Multivariate tests, though it defaults to A/B for simplicity unless traffic justifies higher complexity.

How does this skill prevent 'peeking' in A/B tests?

The skill enforces a pre-defined sample size and duration based on statistical power, requiring users to commit to success criteria before the test begins.

What is the difference between primary and guardrail metrics?

A primary metric is the main success indicator for your hypothesis, while guardrail metrics are health indicators that must not degrade significantly during the test.

What happens if my traffic volume is too low?

The skill includes a refusal condition that warns users if traffic is insufficient to detect the MDE, recommending a redesign or delay instead of running an invalid test.

A/B Test Setup

Name: A/B Test Setup
Author: claudiodearaujo

byclaudiodearaujo

0•

Analíticas y Monitorización

Facilitates the creation of rigorous, statistically sound A/B tests through mandatory validation gates and metric definitions.

The A/B Test Setup skill provides a structured framework for designing and validating experiments before any code is written. It enforces statistical rigor by guiding users through mandatory 'gates' including hypothesis locking, sample size estimation, and validity checks. By requiring clearly defined primary, secondary, and guardrail metrics, this skill prevents common experimentation pitfalls such as 'peeking,' insufficient statistical power, and metric shifting, ensuring that every test provides actionable and reliable data.

Características Principales

01Statistical power and sample size estimation based on Minimum Detectable Effect (MDE).

02Structured metric hierarchy including primary, secondary, and guardrail metrics.

03Execution readiness checklist to prevent premature implementation.

04Mandatory Hypothesis Lock gate to ensure specific and measurable goals.

05Validity checks for traffic stability, user independence, and randomization.

060 GitHub stars

Casos de Uso

01Designing a high-integrity experiment for a new checkout flow.

02Defining guardrail metrics to ensure performance doesn't degrade during UI testing.

03Calculating the required traffic and duration for an A/B/n test.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/izacenter ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill