What is LLM-as-judge in this context?

LLM-as-judge is an automated evaluation pattern where a high-capability model is used to score agent responses based on specific criteria like helpfulness, safety, or tone.

Which observability frameworks are supported?

The skill defines universal span attributes that work with most systems, but provides specific implementation examples for Langfuse and Braintrust.

How does it help with hallucination detection?

The skill provides a structured claim-checking pattern that compares agent responses against source documents to calculate a groundedness score.

Can this skill track human feedback?

Yes, it includes standardized patterns for capturing user ratings, corrections, and comments, and linking that feedback directly to the specific execution trace for debugging.

Agent Quality & Evaluation Instrumentation

Name: Agent Quality & Evaluation Instrumentation
Author: nexus-labs-automation

bynexus-labs-automation

0•

분석 및 모니터링

Instruments AI agent evaluation systems to track response quality, detect hallucinations, and monitor performance through automated and human feedback loops.

The Agent Quality & Evaluation skill empowers developers to build robust observability into AI agents by implementing structured evaluation metrics and feedback loops. It provides standardized patterns for LLM-as-judge scoring, human feedback collection, and ground truth comparisons across dimensions like correctness, helpfulness, and safety. By linking evaluations directly to execution traces, it enables granular debugging of failures and ensures that agent performance is quantitatively measured and continuously improved from development through to production.

주요 기능

01Integration patterns for Langfuse and Braintrust observability frameworks

02Hallucination detection and factual accuracy metrics

03Human feedback instrumentation for thumbs-up/down and user comments

040 GitHub stars

05Automated LLM-as-judge scoring and heuristic patterns

06Standardized span attributes for quality, safety, and relevance

사용 사례

01Detecting and preventing hallucinations in RAG-based AI applications

02Building a continuous evaluation pipeline to compare agent versions against a baseline

03Measuring the impact of prompt engineering changes on agent response quality

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nexus-labs-automation/agent-observability evaluation-quality

For use in Claude.ai and ChatGPT

Download Skill