How does the LLM-as-judge feature work?

It utilizes structured prompts to direct a stronger model to evaluate the outputs of other models based on specific criteria like accuracy, helpfulness, and clarity.

Does it support human-in-the-loop evaluation?

Yes, it provides frameworks for structuring human annotation tasks and calculating inter-rater agreement scores like Cohen's Kappa.

Can I detect regressions automatically?

Yes, the skill includes a RegressionDetector that compares new results against established baselines to flag significant performance drops during development.

What metrics are supported for RAG applications?

The skill includes specialized retrieval metrics such as Mean Reciprocal Rank (MRR), NDCG, and Precision@K, along with custom groundedness checks using NLI models.

LLM Evaluation & Metrics

Name: LLM Evaluation & Metrics
Author: Krosebrook

byKrosebrook

•

Data Science & ML

Implements comprehensive evaluation frameworks for LLM applications using automated metrics, human feedback, and LLM-as-judge patterns.

This skill provides a robust framework for measuring and improving the performance of Large Language Model applications. It equips developers with standardized tools for automated metrics like BLEU and ROUGE, advanced semantic evaluation via BERTScore, and sophisticated 'LLM-as-judge' patterns for subjective quality assessment. By integrating regression detection and statistical A/B testing, it enables data-driven decision-making for prompt engineering, model selection, and production monitoring, ensuring AI outputs remain accurate, safe, and helpful.

Key Features

01Automated NLP metrics including BLEU, ROUGE, and BERTScore

02LLM-as-Judge patterns for pairwise and pointwise scoring

03Statistical A/B testing with Cohen's d effect size analysis

04Retrieval Augmented Generation (RAG) metrics like MRR and NDCG

05Automated regression detection for CI/CD pipeline integration

061 GitHub stars

Use Cases

01Validating prompt improvements through systematic A/B testing

02Measuring RAG system performance, retrieval quality, and groundedness

03Detecting performance regressions before deploying model or configuration updates

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add krosebrook/source-of-truth-monorepo llm-evaluation

For use in Claude.ai and ChatGPT

Download Skill