What is Population Stability Index (PSI) in LLM monitoring?

PSI is a statistical metric used to measure how much a variable's distribution has shifted between two points in time, helping identify data drift in model inputs or outputs.

How does dynamic thresholding reduce alert fatigue?

By using Exponential Weighted Moving Averages (EWMA), thresholds adapt to natural fluctuations in data, only triggering alerts when shifts are statistically significant relative to historical baselines.

Can this skill help with RAG systems?

Yes, it includes patterns for monitoring embedding drift and semantic shifts, which are critical for maintaining the retrieval quality of RAG (Retrieval-Augmented Generation) applications.

Does this skill integrate with existing observability tools?

Yes, it provides implementation patterns for popular tools like Langfuse, Evidently AI, and Arize Phoenix to create a unified monitoring pipeline.

Drift Detection & Quality Monitoring

Name: Drift Detection & Quality Monitoring
Author: yonatangross

byyonatangross

•

分析と監視

Monitors LLM application quality and input distribution shifts to detect performance degradation in production environments.

This skill provides advanced monitoring capabilities for LLM-powered applications by detecting statistical and quality drift. It implements production-ready patterns for calculating the Population Stability Index (PSI), setting dynamic thresholds using Exponential Weighted Moving Averages (EWMA), and integrating with observability tools like Langfuse and Evidently. By identifying input distribution shifts and output quality degradation early, it helps developers maintain model reliability, reduce alert fatigue, and automate evaluation triggers before performance impacts the user experience.

主な機能

01Automated quality regression alerts and performance trend monitoring

0269 GitHub stars

03Statistical drift analysis using Population Stability Index (PSI) and KS tests

04Dynamic thresholding with Exponential Weighted Moving Averages (EWMA)

05Integration patterns for Langfuse, Evidently, and Arize Phoenix

06Semantic and embedding drift detection for RAG systems

ユースケース

01Monitoring semantic shifts in user queries to identify emerging data patterns

02Detecting when production LLM outputs deviate from baseline quality scores

03Reducing false-positive alerts by replacing static thresholds with adaptive control limits

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit drift-detection

For use in Claude.ai and ChatGPT

Download Skill