LLM Performance Evaluation

소개

This skill provides a complete toolkit for measuring and improving AI application quality through rigorous evaluation pipelines. It integrates automated metrics like BLEU, ROUGE, and BERTScore with sophisticated patterns like LLM-as-Judge and RAG-specific retrieval metrics. Whether you are comparing model providers, validating prompt engineering improvements, or setting up regression testing for production, this skill offers the frameworks needed to build confidence in your AI systems using statistical analysis and systematic benchmarking.

주요 기능

• Statistical A/B testing and Cohen’s d effect size analysis
• 81 GitHub stars
• LLM-as-Judge implementation for qualitative scoring
• Human evaluation frameworks and inter-rater agreement tools
• Automated regression detection against performance baselines
• Automated metrics for text generation and RAG performance

사용 사례

• Comparing cost-performance trade-offs between different model tiers
• Measuring RAG groundedness and retrieval quality in production
• Detecting performance regressions before deploying new prompt versions