What is the ICE scoring method used in this skill?

ICE stands for Impact, Confidence, and Ease. The skill helps you calculate an ICE score for every experiment to prioritize tests with the highest potential upside and lowest complexity.

Does this skill provide guidance on interpreting results?

Absolutely. It includes statistical interpretation guardrails to help you distinguish between statistical significance and business significance while avoiding common errors like 'peeking'.

How does this skill help with statistical significance?

It provides a built-in sample size calculator to determine the traffic needed for a specific Minimum Detectable Effect (MDE), ensuring your tests have enough power to reach valid conclusions.

Can it help me write better hypotheses?

Yes, it uses a standardized If/Then/Because format that forces you to specify the intervention, the expected metric change, and the underlying behavioral mechanism.

Experiment Designer

Name: Experiment Designer
Author: zhangzhang-111-i

byzhangzhang-111-i

0•

Data Science & ML

Streamlines product experiment planning by generating testable hypotheses, calculating sample sizes, and interpreting A/B test results with statistical rigor.

The Experiment Designer skill provides a robust framework for product teams and developers to design, prioritize, and evaluate experiments with scientific precision. It guides users through creating 'If/Then/Because' hypotheses, defining primary and guardrail metrics, and calculating required sample sizes using integrated statistical tools. By incorporating ICE scoring for prioritization and providing clear interpretation guardrails for p-values and confidence intervals, this skill ensures that product decisions are backed by defensible data and avoids common pitfalls like underpowered tests or false positives.

Key Features

01Automated sample size and MDE calculation via Python scripts

02Standardized If/Then/Because hypothesis generation

03Statistical interpretation guidance for A/B test outcomes and p-values

04Experiment prioritization using the ICE (Impact, Confidence, Ease) framework

05Comprehensive metric definition including primary, secondary, and guardrail metrics

060 GitHub stars

Use Cases

01Calculating the minimum traffic required to reach statistical significance for a UI change

02Prioritizing a product roadmap by scoring potential experiments based on expected ROI and confidence

03Planning a new feature rollout with a controlled A/B test to measure conversion impact

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add zhangzhang-111-i/claude-skills111 experiment-designer

For use in Claude.ai and ChatGPT

Key Features

01Automated sample size and MDE calculation via Python scripts

02Standardized If/Then/Because hypothesis generation

03Statistical interpretation guidance for A/B test outcomes and p-values

04Experiment prioritization using the ICE (Impact, Confidence, Ease) framework

05Comprehensive metric definition including primary, secondary, and guardrail metrics

060 GitHub stars

Use Cases

01Calculating the minimum traffic required to reach statistical significance for a UI change

02Prioritizing a product roadmap by scoring potential experiments based on expected ROI and confidence

03Planning a new feature rollout with a controlled A/B test to measure conversion impact

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add zhangzhang-111-i/claude-skills111 experiment-designer

For use in Claude.ai and ChatGPT