What is the 'peeking problem' mentioned in the skill?

Peeking refers to looking at results and stopping a test early before reaching the pre-determined sample size, which often leads to false positives and misleading data.

How does this skill help ensure my A/B tests are statistically valid?

The skill provides sample size reference tables and duration calculators to prevent 'peeking,' while enforcing a strict hypothesis framework and confidence interval checks.

Does this skill help with mobile app testing?

While it focuses on web patterns, the core principles of hypothesis testing, traffic allocation, and metric selection are applicable to iOS, Android, and cross-platform apps.

Can this skill help with technical implementation?

Yes, it provides implementation patterns for both client-side (JavaScript/CSS) and server-side (feature flags) testing using tools like PostHog, Optimizely, and LaunchDarkly.

A/B Test & Experimentation Setup

Name: A/B Test & Experimentation Setup
Author: monmacllcapp

bymonmacllcapp

0•

营销自动化

Designs and implements statistically valid A/B tests and experimentation frameworks to optimize product conversion rates.

The A/B Test Setup skill provides a comprehensive framework for planning and executing data-driven experiments. It guides users through the entire experimentation lifecycle, from crafting strong hypotheses and calculating required sample sizes to selecting primary and guardrail metrics. By enforcing statistical rigor and providing clear implementation patterns for both client-side and server-side tests, it ensures that teams make product decisions based on actionable, valid results rather than noise or intuition.

主要功能

01Comprehensive metric strategy including primary, secondary, and guardrail metrics

02Guidance on variant design for headlines, CTAs, and layout changes

030 GitHub stars

04Statistical power calculations for sample size and test duration

05Analysis checklists for interpreting significance and confidence intervals

06Structured hypothesis framework based on data and observations

使用场景

01Planning a landing page headline test to improve sign-up conversion rates

02Designing a multivariate test (MVT) to optimize checkout flow hierarchy

03Setting up a server-side experiment for a new pricing model with a 90/10 traffic split

主要功能

01Comprehensive metric strategy including primary, secondary, and guardrail metrics

02Guidance on variant design for headlines, CTAs, and layout changes

030 GitHub stars

04Statistical power calculations for sample size and test duration

05Analysis checklists for interpreting significance and confidence intervals

06Structured hypothesis framework based on data and observations

使用场景

01Planning a landing page headline test to improve sign-up conversion rates

02Designing a multivariate test (MVT) to optimize checkout flow hierarchy

03Setting up a server-side experiment for a new pricing model with a 90/10 traffic split