Does it support advanced statistical monitoring?

Yes, it includes templates for Sequential Probability Ratio Tests (SPRT) which allow for continuous monitoring of experiment data while controlling for Type I and Type II error rates.

How does this skill help with reproducibility?

It provides standardized patterns for setting random seeds across libraries like NumPy and Python's random module, while also logging environment metadata such as library versions and system info.

Can I use this for A/B testing?

Yes, it includes specific patterns for power analysis to determine required sample sizes and O'Brien-Fleming alpha-spending functions for safe interim monitoring of results.

What is the benefit of the pre-registration feature?

Pre-registration helps you document your hypothesis and analysis plan before examining data, which prevents bias and clearly distinguishes confirmatory results from exploratory observations.

Experiment Design & Reproducibility

Name: Experiment Design & Reproducibility
Author: Yeachan-Heo

byYeachan-Heo

•

120

•

Data Science & ML

Implements best practices for designing reproducible, statistically valid scientific experiments and machine learning workflows.

This skill provides a comprehensive framework for researchers and data scientists to design experiments that are robust and reproducible. It offers automated patterns for setting random seeds, environment recording, and advanced statistical controls like power analysis and sequential probability ratio tests (SPRT). By formalizing pre-registration concepts and stopping rules, it helps prevent common pitfalls like p-hacking and distinguishes between confirmatory and exploratory findings, ensuring that your data-driven decisions are backed by rigorous statistical methodology.

Key Features

01Statistical power analysis for determining required sample sizes across various test types.

02Automated reproducibility setup including random seed synchronization and environment logging.

03Advanced experimental controls like stratified train/test splits and cross-validation patterns.

04Sequential Analysis (SPRT) tools to monitor experiments in real-time with controlled error rates.

05Rigorous A/B testing frameworks including pre-registration plans and alpha-spending functions.

06120 GitHub stars

Use Cases

01Setting up a reproducible machine learning pipeline with synchronized seeds and environment snapshots.

02Designing a statistically powered A/B test for a new product feature to ensure valid results.

03Creating a pre-registered analysis plan to prevent HARKing and improve research transparency.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yeachan-heo/my-jogyo experiment-design

For use in Claude.ai and ChatGPT

Download Skill