How does this skill help with statistical significance?

It provides pre-determined sample size tables and p-value thresholds to ensure your test results are statistically valid and not due to random chance.

What is the 'peeking problem' mentioned in the skill?

The peeking problem occurs when researchers stop a test early after seeing a preliminary result, leading to false positives. This skill enforces pre-committed sample sizes to avoid this error.

Can it help with multivariate testing (MVT)?

Yes, it supports A/B, A/B/n, and MVT designs, providing guidance on the traffic volume required for each type of experiment.

Does it integrate with specific testing tools?

The skill provides implementation guidance for both client-side tools like PostHog and Optimizely, and server-side platforms like LaunchDarkly and Split.

What is a guardrail metric?

A guardrail metric is a business value, such as refund rate or support volume, that you monitor to ensure an experiment doesn't cause unintended harm while trying to improve a primary conversion metric.

A/B Test Setup & Experimentation

Name: A/B Test Setup & Experimentation
Author: oabdelmaksoud

byoabdelmaksoud

0•

Automatización de Marketing

Designs statistically valid A/B tests and experiments to optimize conversion rates and marketing performance.

This skill transforms Claude into an experimentation expert, guiding users through the end-to-end process of planning, designing, and implementing A/B tests. It provides structured frameworks for writing strong hypotheses, calculating required sample sizes, selecting primary and guardrail metrics, and analyzing results with statistical rigor. Whether you are testing UI changes, copy variants, or pricing strategies, this skill ensures your experiments produce actionable, data-driven insights while avoiding common pitfalls like the 'peeking problem' or lack of statistical significance.

Características Principales

01Structured Hypothesis Framework (Observation -> Change -> Outcome)

02Sample Size and Duration Calculators based on Baseline and Lift

03Metric Classification (Primary, Secondary, and Guardrail)

04Comprehensive Pre-Launch and Analysis Checklists

05Guidelines for Statistical Significance and Result Interpretation

060 GitHub stars

Casos de Uso

01Validating new feature performance using server-side or client-side split testing

02Optimizing landing page conversion rates through copy or design variations

03Designing pricing experiments with safety guardrails to prevent revenue loss

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add oabdelmaksoud/superclaw ab-test-setup

For use in Claude.ai and ChatGPT

Características Principales

01Structured Hypothesis Framework (Observation -> Change -> Outcome)

02Sample Size and Duration Calculators based on Baseline and Lift

03Metric Classification (Primary, Secondary, and Guardrail)

04Comprehensive Pre-Launch and Analysis Checklists

05Guidelines for Statistical Significance and Result Interpretation

060 GitHub stars

Casos de Uso

01Validating new feature performance using server-side or client-side split testing

02Optimizing landing page conversion rates through copy or design variations

03Designing pricing experiments with safety guardrails to prevent revenue loss

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add oabdelmaksoud/superclaw ab-test-setup

For use in Claude.ai and ChatGPT