How do I initialize a new evaluation project?

You can use the provided init-deepeval.sh script to automatically set up the necessary directory structure and configuration files.

What metrics can I measure with this skill?

You can measure answer relevancy, faithfulness, contextual relevancy, hallucination rates, toxicity, and bias using built-in DeepEval metrics.

Can I create my own custom metrics?

Yes, the skill supports the creation of custom evaluators, allowing you to define specific criteria for your unique AI application needs.

Does this skill help with RAG evaluation?

Absolutely. It includes specific templates and metrics like Contextual Relevancy designed specifically to evaluate Retrieval-Augmented Generation pipelines.

Can I use this with my existing pytest suite?

Yes, DeepEval is designed to be fully compatible with pytest, and this skill provides templates and configurations for seamless integration.

DeepEval LLM Testing

Name: DeepEval LLM Testing
Author: vanman2024

byvanman2024

•

Security & Testing

Implements robust pytest-style evaluation frameworks to measure LLM performance, RAG quality, and output faithfulness.

DeepEval Testing provides a comprehensive suite for evaluating Large Language Model outputs through standardized metrics and automated pytest workflows. It enables developers to rigorously measure RAG performance, detect hallucinations, and ensure answer relevance using built-in metrics like toxicity and bias detection. By integrating seamlessly with CI/CD pipelines, this skill ensures that AI applications maintain high-quality standards through every iteration, offering templates for both basic unit tests and complex RAG evaluation scenarios.

Key Features

01Comprehensive RAG quality metrics including faithfulness and relevance

02Custom evaluator and metric creation support

03Standardized pytest-compatible evaluation patterns

04Automated project initialization and test execution scripts

052 GitHub stars

06Asynchronous test execution for high-throughput evaluation

Use Cases

01Integrating automated LLM quality gates into CI/CD workflows

02Detecting hallucinations and toxicity in LLM-generated content

03Benchmarking RAG pipeline accuracy and context relevance

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add vanman2024/ai-dev-marketplace deepeval-testing

For use in Claude.ai and ChatGPT

Download Skill