What is the 'peeking problem' in A/B testing?

Peeking involves checking results before reaching the required sample size and stopping early. This significantly increases the risk of false positives and making decisions based on random noise.

Can this skill help with multivariate testing (MVT)?

Yes, it provides guidance for A/B, A/B/n, and MVT experiments, helping you determine which type is appropriate based on your current traffic volume.

Does it support server-side testing implementation?

Yes, it provides implementation strategies for both client-side (flicker-prone but fast) and server-side (rigorous but requires dev work) testing using tools like PostHog or LaunchDarkly.

How do I know if my test result is significant?

The skill provides an analysis checklist including p-value thresholds (typically < 0.05), confidence intervals, and effect size comparisons against your Minimum Detectable Effect (MDE).

A/B Test Setup

Name: A/B Test Setup
Author: fianchettogianni

byfianchettogianni

0•

Marketing Automation

Designs and implements statistically valid A/B tests and experiments to optimize product conversion rates and marketing performance.

This skill transforms Claude into an experimentation expert that helps users move from intuitive guessing to rigorous, data-driven testing. It provides structured frameworks for writing strong hypotheses, calculating required sample sizes, selecting the right primary and guardrail metrics, and determining the best implementation method (client-side vs. server-side). Whether you're optimizing button colors or complex pricing models, this skill ensures your tests produce actionable results while avoiding common pitfalls like 'peeking' or testing too many variables at once.

Key Features

01Statistical significance and results analysis checklist

02Rigorous Hypothesis Framework generator

03Sample size and duration calculation guidance

040 GitHub stars

05Implementation strategy for client-side and server-side tests

06Primary, secondary, and guardrail metric selection

Use Cases

01Setting up server-side feature flags for pricing and checkout tests

02Analyzing post-test data to determine statistical significance and actionable learnings

03Designing a high-impact landing page conversion experiment

Key Features

01Statistical significance and results analysis checklist

02Rigorous Hypothesis Framework generator

03Sample size and duration calculation guidance

040 GitHub stars

05Implementation strategy for client-side and server-side tests

06Primary, secondary, and guardrail metric selection

Use Cases

01Setting up server-side feature flags for pricing and checkout tests

02Analyzing post-test data to determine statistical significance and actionable learnings

03Designing a high-impact landing page conversion experiment