What is a 'Hard Gate' in this skill?

A Hard Gate is a mandatory checkpoint where the user must explicitly confirm specific criteria, such as a locked hypothesis or primary metric, before the skill allows the workflow to proceed to implementation.

Can I use this for multivariate testing (MVT)?

Yes, the skill includes logic for selecting various test types including A/B, A/B/n, Multivariate (MVT), and Split URL tests based on traffic and goals.

How does this skill prevent invalid A/B tests?

It enforces statistical rigor by requiring Minimum Detectable Effect (MDE) calculations, baseline rates, and defined sample sizes, and will refuse to proceed if traffic or metrics are insufficient.

Does this skill provide code for the A/B test?

This skill focuses on the architectural and statistical setup of the test. While it prepares the requirements, its primary purpose is to ensure the design is valid before any code is written.

A/B Test Rigor & Setup

Name: A/B Test Rigor & Setup
Author: claudiodearaujo

byclaudiodearaujo

•

Analytics & Monitoring

Enforces a structured, statistically sound framework for designing and validating A/B tests to ensure data-driven confidence.

About

The A/B Test Setup skill provides a rigorous framework for developers and growth engineers to design experiments that are scientifically valid. By implementing mandatory gates for hypothesis locking, metric definition, and sample size estimation, it prevents common experimentation pitfalls like 'peeking' or insufficient statistical power. It guides users through selecting the right test type, establishing guardrail metrics, and verifying execution readiness, ensuring that every experiment delivers actionable insights rather than noise.

Key Features

Mandatory 'Hard Gates' for hypothesis and metric validation
1 GitHub stars
Automated refusal conditions for invalid test designs
Guardrail metric definition to prevent harmful deployments
Statistical power and sample size estimation guidance
Strict anti-peeking and analysis discipline protocols

Use Cases

Validating backend algorithm changes against control groups
Setting up high-traffic UI tests with proper statistical significance
Designing conversion rate optimization experiments for product features

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill

GitHub