How does the LLM-as-Judge pattern work?

It utilizes a high-capability model to evaluate the outputs of other models, offering pointwise scoring for individual responses or pairwise comparisons to determine which model version is superior.

Can I use this for RAG (Retrieval-Augmented Generation) apps?

Yes, it includes specialized metrics for retrieval performance like Mean Reciprocal Rank (MRR) and NDCG, alongside groundedness checks to ensure responses are based on provided context.

What automated metrics are supported for text generation?

The skill supports standard NLP metrics including BLEU for translation, ROUGE for summarization, METEOR for semantic similarity, and BERTScore for embedding-based evaluation.

Does it support statistical significance testing?

Yes, it includes a built-in A/B testing framework that performs T-tests and calculates p-values and Cohen's d to ensure improvements are statistically significant.

LLM Performance Evaluation

Name: LLM Performance Evaluation
Author: Activer007

byActiver007

0•

数据科学与机器学习

Implement comprehensive evaluation frameworks for Large Language Model applications using automated metrics, human feedback, and LLM-as-judge patterns.

This skill provides a structured framework for measuring the quality, accuracy, and reliability of LLM-powered applications. It enables developers to implement automated metrics like BERTScore and ROUGE, establish LLM-as-judge evaluation patterns, and manage human annotation workflows. By integrating systematic benchmarking and A/B testing, the skill helps teams detect performance regressions, compare model variations, and build confidence in production AI systems through rigorous statistical validation and factual grounding checks.

主要功能

01RAG-specific evaluation for retrieval quality and groundedness

02Regression detection to identify performance drops before deployment

03LLM-as-Judge patterns for automated pointwise and pairwise scoring

04Automated NLP metrics including BLEU, ROUGE, METEOR, and BERTScore

05Statistical A/B testing framework with Cohen’s d effect size calculation

060 GitHub stars

使用场景

01Systematically comparing prompt iterations and different model versions

02Validating RAG pipeline accuracy and information retrieval quality

03Establishing automated quality gates in CI/CD pipelines for AI applications

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add activer007/ordinary-claude-skills llm-evaluation

For use in Claude.ai and ChatGPT

Download Skill