What is the 'peeking problem' in A/B testing?

The peeking problem occurs when you check results and stop a test early before reaching the pre-determined sample size, which significantly increases the risk of false positives.

Does this skill help with multivariate testing (MVT)?

Yes, it provides guidance and traffic requirements for various test types including A/B, A/B/n, Multivariate Testing (MVT), and Split URL tests.

Can I use this for server-side experiments?

Absolutely. The skill includes implementation strategies for both client-side (JavaScript/flicker-prone) and server-side (render-based) testing environments.

What are guardrail metrics?

Guardrail metrics are secondary data points that shouldn't get worse during a test, such as page load speed or support ticket volume, used to ensure a winning variant doesn't cause unintended harm.

A/B Test Setup & Experimentation

Name: A/B Test Setup & Experimentation
Author: max0n232

bymax0n232

0•

营销自动化

Designs and implements statistically valid A/B tests to optimize product performance and conversion rates.

This skill empowers Claude to act as an experimentation expert, guiding users through the end-to-end process of setting up scientific split tests. It provides structured frameworks for creating data-driven hypotheses, calculating required sample sizes to avoid the 'peeking problem,' and selecting meaningful primary, secondary, and guardrail metrics. Whether you are testing UI changes, copy variations, or backend logic, this skill ensures your experiments produce actionable, statistically significant results while avoiding common pitfalls like underpowered tests or misinterpreted data.

主要功能

01Implementation strategies for client-side and server-side testing

02Post-test analysis checklists for statistical significance and confidence

03Structured hypothesis framework based on data and observations

040 GitHub stars

05Statistical power and sample size calculation guidance

06Selection of primary, secondary, and guardrail metrics

使用场景

01Setting up guardrail metrics to ensure a pricing change doesn't increase churn

02Designing a headline split test to increase landing page conversion

03Calculating the required traffic and duration for a new feature experiment

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add max0n232/arkhos ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill