What is the recommended sample size for statistical assertions?

The skill provides a calibrated scale: N=400 for standard moderate precision (+/-5%) and N=10,000 for critical validations requiring high precision (+/-1%).

When should I use Bootstrap versus Clopper-Pearson patterns?

Use Clopper-Pearson for simple binary outcomes (success/failure) and use Bootstrap for more complex statistics like medians, percentiles, or custom metrics.

Can I test time-series simulations with this skill?

Yes, it includes patterns for 'Tolerance Bands,' which are specifically designed to test simulated paths and trends over time.

How does this skill prevent flaky tests in CI?

It implements a strategy of using fixed, documented seeds for CI environments, ensuring that random processes produce reproducible results every time the test suite runs.

Statistical Testing & Simulation Validation

Name: Statistical Testing & Simulation Validation
Author: nikhillinit

bynikhillinit

Security & Testing

Ensures the reliability of probabilistic code and Monte Carlo simulations through rigorous statistical assertions and stable testing patterns.

About

This skill provides a comprehensive framework for testing non-deterministic code, such as simulations, statistical estimates, and random sampling, without the common pitfall of flaky CI tests. It guides developers through choosing the right testing approach—ranging from deterministic mocking for control flows to seeded statistical tests for distributional properties. By implementing these patterns, teams can maintain high confidence in their probabilistic algorithms while ensuring that CI/CD pipelines remain stable, reproducible, and mathematically sound.

Key Features

Standardizes sample size (N) selection based on desired precision for proportions and means.
0 GitHub stars
Eliminates flaky CI tests using seeded statistical validation and deterministic mocking.
Provides a decision tree for choosing between mock, seeded, and random-seed testing approaches.
Includes implementation patterns for Clopper-Pearson, Bootstrap, and Tolerance Band confidence intervals.
Integrates property-based testing to validate invariants across diverse simulation outcomes.

Use Cases

Testing financial models or portfolio simulations for expected returns and risk profiles.
Validating machine learning model predictions and distributional properties in a pipeline.
Verifying the robustness of stochastic algorithms against edge cases using fast-check patterns.

About

Key Features

Standardizes sample size (N) selection based on desired precision for proportions and means.
0 GitHub stars
Eliminates flaky CI tests using seeded statistical validation and deterministic mocking.
Provides a decision tree for choosing between mock, seeded, and random-seed testing approaches.
Includes implementation patterns for Clopper-Pearson, Bootstrap, and Tolerance Band confidence intervals.
Integrates property-based testing to validate invariants across diverse simulation outcomes.

Use Cases

Testing financial models or portfolio simulations for expected returns and risk profiles.
Validating machine learning model predictions and distributional properties in a pipeline.
Verifying the robustness of stochastic algorithms against edge cases using fast-check patterns.