What is the 'Hypothesis Lock' feature?

Hypothesis Lock is a mandatory gate that requires you to finalize your target audience, primary metric, and expected outcome before designing variants, preventing bias during the analysis phase.

Can this skill help with sample size estimation?

Yes, the skill requires you to define a Minimum Detectable Effect (MDE) and baseline rate to estimate the necessary sample size and test duration before proceeding.

Does it support tests other than simple A/B?

Yes, it supports A/B/n, Multivariate (MVT), and Split URL tests, though it defaults to standard A/B testing unless a clear reason for complexity is provided.

What happens if my traffic is too low for a test?

The skill includes safety checks that will warn you or refuse to proceed if the estimated traffic volume cannot support the statistical power required for a valid test.

A/B Test Setup Framework

Name: A/B Test Setup Framework
Author: claudiodearaujo

byclaudiodearaujo

0•

Analytics & Monitoring

Enforces statistical rigor and logical validity in A/B test designs through mandatory quality gates and success criteria.

The A/B Test Setup skill provides a structured, fail-safe methodology for designing and executing experiments within Claude Code. It prevents common experimental errors by implementing mandatory 'gates'—such as Hypothesis Lock and Execution Readiness—ensuring that every test has a defined audience, primary metric, and sufficient statistical power before any code is modified. This skill is essential for teams that require high-confidence results and want to avoid pitfalls like 'peeking' at data or running underpowered tests.

Key Features

01Rigorous 'Refusal Conditions' for statistically invalid experiment designs

02Standardized experiment documentation for long-term learning and records

03Automated validation of sample size and test duration requirements

04Hierarchical metric definition (Primary, Secondary, and Guardrail)

05Mandatory Hypothesis Lock to prevent post-hoc goalpost shifting

060 GitHub stars

Use Cases

01Designing high-stakes UI/UX experiments where statistical confidence is required

02Calculating if existing traffic volume is sufficient to detect a desired conversion lift

03Establishing safety guardrails to ensure experiments do not negatively impact core business KPIs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro-front ab-test-setup

For use in Claude.ai and ChatGPT

Key Features

01Rigorous 'Refusal Conditions' for statistically invalid experiment designs

02Standardized experiment documentation for long-term learning and records

03Automated validation of sample size and test duration requirements

04Hierarchical metric definition (Primary, Secondary, and Guardrail)

05Mandatory Hypothesis Lock to prevent post-hoc goalpost shifting

060 GitHub stars

Use Cases

01Designing high-stakes UI/UX experiments where statistical confidence is required

02Calculating if existing traffic volume is sufficient to detect a desired conversion lift

03Establishing safety guardrails to ensure experiments do not negatively impact core business KPIs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro-front ab-test-setup

For use in Claude.ai and ChatGPT