What is 'silent failure' detection in the context of LLMs?

It refers to identifying issues where an AI agent might skip necessary tools or provide degraded answers without throwing a formal code error, using heuristics and LLM-as-a-judge patterns.

How does it handle LLM drift detection?

The skill provides patterns for statistical drift detection using Population Stability Index (PSI) and KS tests, as well as quality drift monitoring through score regression and baseline comparisons.

Can this skill help reduce LLM operational costs?

Yes, it includes specific rules for LLM cost tracking, token usage monitoring, and spend alerts to help developers manage and optimize API expenses effectively.

What monitoring tools does this skill support?

It focuses on industry standards like Prometheus for metrics, Grafana for dashboards, OpenTelemetry for tracing, and Langfuse for LLM-specific observability.

Monitoring & Observability

Name: Monitoring & Observability
Author: yonatangross

byyonatangross

•

116

•

分析と監視

Implements comprehensive monitoring, LLM tracing, and drift detection patterns for production-ready AI applications.

This skill provides a complete framework for maintaining system health and LLM performance through standardized observability patterns. It integrates infrastructure monitoring using Prometheus and Grafana with specialized AI observability through Langfuse for tracing and cost tracking. By implementing statistical drift detection and silent failure monitoring, it ensures that LLM-driven features remain reliable, cost-effective, and free from quality regression over time, making it essential for developers deploying AI agents at scale.

主な機能

01Statistical drift detection (PSI, KS tests) to identify quality regressions in production models

02Infrastructure monitoring with Prometheus metrics and Grafana dashboard templates

03Standardized alerting rules and severity levels for effective incident management

04116 GitHub stars

05Silent failure detection to alert on tool skipping, token spikes, and quality degradation

06Advanced LLM observability using Langfuse for tracing, cost tracking, and evaluation scoring

ユースケース

01Setting up full-stack observability for a production LLM application

02Monitoring LLM API costs and token usage to prevent unexpected budget overruns

03Detecting when model output quality drifts from established baselines in production

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit monitoring-observability

For use in Claude.ai and ChatGPT

Download Skill