PostHog Experiment Health Diagnostics provides automated troubleshooting and statistical health checks for PostHog A/B testing and experimentation workflows. It enables Claude to evaluate experiment configurations, identify sample ratio mismatches (SRM), detect zero-exposure or skew issues, and uncover identity resolution bugs like users showing up across multiple variants. The skill inspects raw exposure criteria, historical experiment changes, and metric definitions while guiding teams through statistical interpretation pitfalls such as peeking bias, A/A testing variances, sequential testing thresholds, and CUPED adjustments to ensure experiment results are reliable and scientifically sound.
主要功能
01Discrepancy reconciliation between experiment funnels, insights, and raw SQL queries
02Sample ratio mismatch (SRM) and variant skew root-cause analysis
03Multi-variant exposure detection and distinct_id migration audit
04Zero-exposure and missing flag evaluation diagnostic checks
0579 GitHub stars
06Statistical validity analysis covering Bayesian, Frequentist, and sequential testing
使用场景
01Auditing experiment setup when users are observed entering multiple test variants or experiencing UI flickering
02Troubleshooting experiments showing zero exposures or skewed split ratios (e.g., 58/42 instead of 50/50)
03Investigating discrepancies between experiment reporting dashboards and internal SQL analytics data