Monitors and detects quality degradation, input distribution shifts, and semantic drift in LLM-powered applications.
The Drift Detection skill provides a comprehensive suite of statistical tools and production-ready patterns for monitoring LLM behavior over time. It enables developers to implement Population Stability Index (PSI), Exponential Weighted Moving Average (EWMA) dynamic thresholds, and embedding-based semantic drift analysis to identify when a model's performance deviates from its baseline. By integrating with observability tools like Langfuse and Arize Phoenix, this skill helps teams proactively catch concept drift and quality regression before they impact end-users, ensuring that AI-driven features remain reliable and accurate in production environments.
主要功能
01Seamless integration with Langfuse quality scores and trace monitoring
02Dynamic thresholding with Exponential Weighted Moving Average (EWMA)
03Semantic embedding drift detection for RAG and cluster analysis
04Automated alerting patterns for LLM output quality degradation
0569 GitHub stars
06Statistical drift analysis using PSI, KS test, and KL divergence
使用场景
01Detecting when user input distributions change (data drift) requiring prompt adjustments
02Monitoring for gradual degradation in LLM response accuracy and consistency
03Implementing automated quality gates for model updates or canary deployments
