What metrics are supported by this skill?

It supports standard NLP metrics like BLEU and ROUGE, semantic metrics like BERTScore, and specialized custom metrics for toxicity, groundedness, and factuality.

Can I use this for RAG application testing?

Yes, it includes frameworks for measuring retrieval metrics like Precision@K and NDCG, as well as NLI-based groundedness checks to ensure responses are based on the provided context.

What is regression detection in this context?

Regression detection compares current evaluation results against a saved baseline. If scores drop beyond a specific threshold (e.g., 5%), the system flags it as a performance regression.

How does the LLM-as-Judge pattern work?

It uses a stronger model (like GPT-4 or Claude 3.5 Sonnet) to evaluate the outputs of other models based on defined criteria like accuracy, helpfulness, and clarity.

LLM Evaluation

Name: LLM Evaluation
Author: ccf

byccf

データサイエンスとML

Implements comprehensive evaluation frameworks to measure LLM application quality using automated metrics, human feedback, and comparative benchmarks.

概要

The LLM Evaluation skill provides a robust toolkit for systematically assessing the performance of Large Language Model applications. It bridges the gap between development and production by offering standardized methods for automated metric calculation (like BERTScore, ROUGE, and BLEU), LLM-as-Judge patterns, and human annotation workflows. Whether you are validating prompt iterations, comparing model providers, or protecting against regressions, this skill provides the statistical and procedural rigor necessary to build reliable AI systems.

主な機能

Automated regression detection against performance baselines
0 GitHub stars
Statistical A/B testing framework with significance and effect size calculation
LLM-as-Judge patterns for pointwise and pairwise qualitative assessments
Automated NLP metrics including BLEU, ROUGE, and BERTScore
Customizable metrics for RAG groundedness, toxicity, and factuality

ユースケース

Validating prompt engineering changes and system instructions before deployment
Comparing performance across different foundation models or fine-tuned versions
Establishing quality benchmarks for RAG-based knowledge retrieval systems

概要

主な機能

Automated regression detection against performance baselines
0 GitHub stars
Statistical A/B testing framework with significance and effect size calculation
LLM-as-Judge patterns for pointwise and pairwise qualitative assessments
Automated NLP metrics including BLEU, ROUGE, and BERTScore
Customizable metrics for RAG groundedness, toxicity, and factuality

ユースケース

Validating prompt engineering changes and system instructions before deployment
Comparing performance across different foundation models or fine-tuned versions
Establishing quality benchmarks for RAG-based knowledge retrieval systems