Designs statistically rigorous A/B tests and interprets experiment results to drive data-driven product decisions.
The Experiment Designer skill empowers product teams to transition from gut-feeling hypotheses to data-backed decisions by automating the creation of rigorous experiment frameworks. It assists in calculating required sample sizes, estimating run times, and identifying potential design risks like novelty effects or sample ratio mismatches. Beyond initial design, the skill interprets complex statistical results, distinguishing between statistical and practical significance to provide clear, defensible recommendations on whether to ship, iterate, or kill a feature.
主な機能
01Statistical and practical significance assessment for raw test results
02Automated sample size and run time calculations based on MDE and baseline metrics
03295 GitHub stars
04Standardized output for 'Ship, Iterate, or Kill' decision frameworks
05Comprehensive risk flagging for novelty effects, seasonal confounds, and peeking problems
06Structured hypothesis generation focusing on specific changes and measurable outcomes
ユースケース
01Planning a new feature rollout to determine the necessary traffic and duration for a valid test
02Interpreting A/B test data to defend product decisions to engineering leads and data scientists
03Validating existing experiment results for integrity issues like sample ratio mismatch (SRM)
