Why does the framework include a grace period?

The grace period acts as a human veto window (default 30 minutes), allowing developers to review and cancel proposed experiments before resources are spent.

How does the skill handle budget overruns?

The system monitors spending in real-time; tests are automatically moved to a REVERTED status if they exceed the pre-defined maximum cost (default $2.00) or duration.

What parameters are forbidden in A/B testing?

Parameters critical to security or system integrity, such as safety gates, cost limits, rate limits, and authentication settings, are strictly forbidden and cannot be tested.

Where are the experiment results stored?

All test results, verdicts, and cost data are recorded in a centralized evidence store, making past findings searchable for future decision-making.

A/B Testing Framework

Name: A/B Testing Framework
Author: supportersimulator

bysupportersimulator

0•

보안 및 테스팅

Automates the end-to-end A/B testing lifecycle for prompts and model parameters with built-in safety guardrails and budget caps.

This skill enables developers to empirically validate changes to LLM configurations by running controlled experiments between different variants. It manages the entire experimental lifecycle, from initial proposal and human veto periods to live activation and automated performance measurement. By integrating strict safety constraints on forbidden parameters and enforcing budget limits, it ensures that experiments never compromise system integrity or exceed financial boundaries, while the persistent evidence store provides a searchable history of past optimizations to inform future development.

주요 기능

01Real-time performance measurement and metric comparison

02Full experimental lifecycle management from proposal to conclusion

03Automated cost and duration caps to prevent resource exhaustion

040 GitHub stars

05Safety-first design with forbidden parameter protections and human veto windows

06Evidence store integration for searchable, historical experiment results

사용 사례

01Optimizing model parameters like temperature for better output consistency

02Comparing system prompt variants to determine which produces higher accuracy

03Validating the efficiency of new caching strategies against current implementations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add supportersimulator/3-surgeons ab-testing

For use in Claude.ai and ChatGPT

주요 기능

01Real-time performance measurement and metric comparison

02Full experimental lifecycle management from proposal to conclusion

03Automated cost and duration caps to prevent resource exhaustion

040 GitHub stars

05Safety-first design with forbidden parameter protections and human veto windows

06Evidence store integration for searchable, historical experiment results

사용 사례

01Optimizing model parameters like temperature for better output consistency

02Comparing system prompt variants to determine which produces higher accuracy

03Validating the efficiency of new caching strategies against current implementations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add supportersimulator/3-surgeons ab-testing

For use in Claude.ai and ChatGPT