How do I start an experiment analysis with this skill?

Simply provide the Amplitude experiment URL or ID to Claude. If you don't have the ID, you can ask Claude to search for the experiment by its name.

Can I see how different user segments performed?

Yes, the skill automatically generates detailed markdown tables breaking down results by platform, user tenure, plan type, and geography.

What statistical checks does the skill perform?

It performs Sample Ratio Mismatch (SRM) checks, statistical power analysis, 95% confidence interval calculations, and validates statistical assumptions like variance and mean validity.

Does it support guardrail metrics?

Yes, it specifically monitors guardrail metrics to identify any unintended negative consequences or regressions in revenue, retention, or engagement.

Amplitude Experiment Analyst

Name: Amplitude Experiment Analyst
Author: amplitude

byamplitude

•

Analytics & Monitoring

Performs rigorous statistical analysis of A/B tests and experiments to drive data-driven ship/no-ship decisions.

The Amplitude Experiment Analyst skill transforms Claude into a specialized data scientist capable of performing deep-dive evaluations of your A/B tests. It goes beyond surface-level summaries to provide comprehensive statistical rigor, including Sample Ratio Mismatch (SRM) detection, power analysis, and precision checks. By analyzing primary, secondary, and guardrail metrics alongside automated segment breakdowns (like platform and user tenure), it helps product teams identify not just if a variant won, but why it won and what the practical business implications are.

Key Features

0112 GitHub stars

02Automated Primary and Secondary Metric Deep-Dives

03Comprehensive Data Quality & SRM Validation

04Multi-Segment Breakdown Tables (Platform, Tenure, Geography)

05Actionable Ship/No-Ship Decision Recommendations

06Statistical Power and Precision Analysis

Use Cases

01Investigating segment-level effects to understand divergent user behavior.

02Monitoring running experiments for early signals or data quality issues like SRM.

03Analyzing completed experiment results to finalize product ship decisions.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add amplitude/mcp-marketplace analyze-experiment

For use in Claude.ai and ChatGPT

Key Features

0112 GitHub stars

02Automated Primary and Secondary Metric Deep-Dives

03Comprehensive Data Quality & SRM Validation

04Multi-Segment Breakdown Tables (Platform, Tenure, Geography)

05Actionable Ship/No-Ship Decision Recommendations

06Statistical Power and Precision Analysis

Use Cases

01Investigating segment-level effects to understand divergent user behavior.

02Monitoring running experiments for early signals or data quality issues like SRM.

03Analyzing completed experiment results to finalize product ship decisions.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add amplitude/mcp-marketplace analyze-experiment

For use in Claude.ai and ChatGPT