What metrics are supported for text generation?

The skill supports standard n-gram metrics like BLEU and ROUGE, as well as semantic metrics like BERTScore and METEOR to measure similarity and fluency.

How does LLM-as-Judge improve evaluation?

It uses a more capable model to grade outputs on subjective criteria like helpfulness and coherence, providing a scalable alternative to manual human review.

How does the regression detection feature work?

It compares new model outputs against a set of baseline results and flags any significant statistical decreases in performance based on a configurable threshold.

Does it support statistical significance testing?

Yes, it includes a statistical testing framework using T-tests and Cohen’s d to determine if improvements between version A and B are truly significant.

Can I use this for RAG applications?

Yes, it includes specialized retrieval metrics such as MRR (Mean Reciprocal Rank), NDCG, and Precision@K to evaluate the quality of your context retrieval.

LLM Application Evaluation

Name: LLM Application Evaluation
Author: HermeticOrmus

byHermeticOrmus

Ciencia de Datos y ML

Implement comprehensive evaluation frameworks for LLM applications using automated metrics, human feedback, and benchmarking.

Acerca de

The llm-evaluation skill provides a systematic approach to measuring and improving the quality of AI-driven applications. It enables developers to implement a multi-layered evaluation strategy encompassing automated text metrics (BLEU, ROUGE, BERTScore), LLM-as-Judge patterns for semantic assessment, and structured human evaluation frameworks. By integrating statistical A/B testing and regression detection, this skill helps teams confidently validate prompt changes, compare model performance, and ensure production-grade reliability across text generation, classification, and RAG tasks.

Características Principales

LLM-as-Judge patterns for automated pointwise and pairwise evaluation
Statistical A/B testing framework with Cohen's d effect size calculation
Automated metrics for text generation and retrieval (RAG) performance
Regression detection to prevent performance drops during deployment
Human annotation structures with inter-rater agreement (Cohen's Kappa) analysis
0 GitHub stars

Casos de Uso

Systematically comparing different foundation models or prompt iterations
Validating RAG system accuracy, groundedness, and retrieval relevance
Building automated CI/CD pipelines to catch LLM performance regressions

Acerca de

Características Principales

LLM-as-Judge patterns for automated pointwise and pairwise evaluation
Statistical A/B testing framework with Cohen's d effect size calculation
Automated metrics for text generation and retrieval (RAG) performance
Regression detection to prevent performance drops during deployment
Human annotation structures with inter-rater agreement (Cohen's Kappa) analysis
0 GitHub stars

Casos de Uso

Systematically comparing different foundation models or prompt iterations
Validating RAG system accuracy, groundedness, and retrieval relevance
Building automated CI/CD pipelines to catch LLM performance regressions