What is ICE scoring in experiment prioritization?

ICE stands for Impact, Confidence, and Ease. This framework helps you rank experiments by multiplying these three factors to identify high-value, low-effort opportunities.

How does this skill help with sample size calculation?

The skill includes a dedicated Python script that computes the required sample size per variant based on your baseline conversion rate, minimum detectable effect (MDE), and statistical power.

Can this skill help interpret p-values and confidence intervals?

Yes, it provides specific guardrails for reading statistical output, helping you distinguish between statistical significance and practical business significance while avoiding common pitfalls like 'peeking' at results.

What is the If/Then/Because format for hypotheses?

It is a structured framework where you state the intervention (If), the expected metric change (Then), and the underlying behavioral mechanism (Because) to ensure tests are logically sound.

Experiment Designer

Name: Experiment Designer
Author: micsapp

bymicsapp

0•

Analytics & Monitoring

Designs and evaluates product experiments with statistical rigor, hypothesis frameworks, and sample size calculations.

Experiment Designer is a specialized Claude Code skill that empowers product teams and developers to run defensible A/B and multivariate tests. It provides structured frameworks for writing testable hypotheses using the If/Then/Because model, defining primary and guardrail metrics, and calculating required sample sizes using built-in scripts. By integrating ICE scoring for prioritization and providing guidelines for statistical interpretation, the skill ensures that experiment results lead to confident product decisions rather than false positives or underpowered conclusions.

Key Features

01Metric definition guidance for primary and guardrail metrics

02ICE scoring framework for test prioritization

03Result interpretation and statistical significance guardrails

04Structured If/Then/Because hypothesis generation

050 GitHub stars

06Statistical sample size and MDE calculator

Use Cases

01Planning A/B tests for product features and UI changes

02Calculating required traffic and duration for statistical significance

03Prioritizing a roadmap of potential experiments based on impact and ease

Key Features

01Metric definition guidance for primary and guardrail metrics

02ICE scoring framework for test prioritization

03Result interpretation and statistical significance guardrails

04Structured If/Then/Because hypothesis generation

050 GitHub stars

06Statistical sample size and MDE calculator

Use Cases

01Planning A/B tests for product features and UI changes

02Calculating required traffic and duration for statistical significance

03Prioritizing a roadmap of potential experiments based on impact and ease