How does this skill help avoid the 'peeking problem'?

The skill emphasizes pre-determining sample sizes and committing to a methodology before launching, preventing false positives caused by stopping tests early based on preliminary data.

How does it help with metric selection?

It helps you define a single primary 'North Star' metric for the win condition, secondary metrics to explain user behavior, and guardrail metrics to ensure the change doesn't negatively impact other business KPIs.

What tools does this skill support?

The guidance is tool-agnostic but provides specific implementation patterns for popular platforms like PostHog, Optimizely, LaunchDarkly, VWO, and custom in-house solutions.

Can I use this for complex multivariate tests?

Yes, while standard A/B tests are recommended for most cases, the skill provides guidance on A/B/n and MVT (Multivariate Testing) for high-traffic environments where multiple variables need testing simultaneously.

A/B Test Setup & Experimentation

Name: A/B Test Setup & Experimentation
Author: Microck

byMicrock

•

营销自动化

Designs and implements statistically valid A/B tests and experiments to optimize product performance and conversion rates.

This skill transforms Claude into an experimentation expert, guiding users through the complete lifecycle of A/B and multivariate testing. It provides structured frameworks for crafting rigorous hypotheses, calculating required sample sizes to ensure statistical power, and selecting primary, secondary, and guardrail metrics. By following established best practices, the skill helps teams avoid common pitfalls like premature 'peeking' or over-interpreting inconclusive results, leading to more confident, data-driven product decisions whether you are testing UI copy or complex backend logic.

主要功能

012 GitHub stars

02Structured hypothesis generation using data-driven frameworks

03Post-test analysis checklists for interpreting significance and effect size

04Multi-tier metric selection including primary, secondary, and guardrail metrics

05Technical implementation guidance for client-side and server-side experiments

06Sample size and duration calculation for statistical rigor

使用场景

01Planning a conversion rate optimization (CRO) test for a high-traffic landing page

02Analyzing experiment results to determine statistical significance and business impact

03Setting up a server-side feature flag experiment for a new product capability

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add microck/opendots-microck ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill