How does the LLM-as-judge pattern work?

It uses a highly capable model (like Claude 3.5 Sonnet or GPT-4o) to evaluate the outputs of other models based on defined rubrics for accuracy, helpfulness, and safety.

Can I use this to evaluate RAG applications?

Yes, the skill provides specific strategies for checking groundedness, retrieval precision, and how well an LLM utilizes provided context.

Does this skill help with regression testing?

Absolutely. It includes a RegressionDetector framework to compare new model outputs against established baselines to ensure performance hasn't degraded.

What automated metrics are supported in this skill?

It includes implementation patterns for text generation metrics like BLEU and ROUGE, semantic metrics like BERTScore, and retrieval metrics such as MRR and NDCG.

LLM Evaluation Framework

Name: LLM Evaluation Framework
Author: HermeticOrmus

byHermeticOrmus

0•

데이터 과학 및 ML

Implements rigorous evaluation strategies for LLM applications using automated metrics, human-in-the-loop feedback, and systematic benchmarking.

The LLM Evaluation skill provides a comprehensive toolkit for measuring, monitoring, and improving AI application quality. It enables developers to implement automated metrics like BLEU and ROUGE, deploy LLM-as-judge patterns for semantic validation, and establish human evaluation workflows. By integrating these strategies, teams can detect regressions, validate prompt improvements, and build production-ready systems with confidence through data-driven performance analysis and statistical A/B testing.

주요 기능

01Statistical A/B testing and performance regression detection

020 GitHub stars

03Automated NLP metrics including BLEU, ROUGE, and BERTScore

04Human annotation frameworks with inter-rater agreement tracking

05LLM-as-judge patterns for qualitative and semantic assessment

06Retrieval-augmented generation (RAG) specific evaluation (MRR, NDCG)

사용 사례

01Validating RAG system groundedness and factual accuracy against context

02Comparing model performance and accuracy during prompt engineering

03Detecting quality regressions before deploying AI updates to production

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add hermeticormus/hermetic-academy llm-evaluation

For use in Claude.ai and ChatGPT

Download Skill