RagScore

소개

RagScore offers a swift and privacy-conscious solution for auditing RAG systems. It empowers users to quickly generate comprehensive QA datasets from their documents and evaluate RAG endpoints with just two commands, whether in Colab, Jupyter notebooks, or via the command line interface. Supporting both cloud and local LLMs like Ollama, RagScore prioritizes data privacy while delivering insightful visual reports and detailed multi-metric evaluations to pinpoint RAG performance issues.

주요 기능

• Generate QA datasets from diverse documents (PDFs, etc.) with configurable settings
• 28 GitHub stars
• Evaluate RAG systems using a simple 2-line CLI or a flexible Python API
• Ensure 100% data privacy with support for local LLMs like Ollama, requiring no cloud API keys
• Integrate with various LLM providers including OpenAI, Anthropic, DashScope, vLLM, and any OpenAI-compatible API
• Conduct detailed multi-metric evaluations across five diagnostic dimensions (Correctness, Completeness, Relevance, Conciseness, Faithfulness)

사용 사례

• Securely test RAG applications with confidential data in privacy-sensitive sectors (e.g., healthcare, legal, finance) using local LLMs
• Automate RAG evaluation in production workflows via CLI for continuous integration and AI agent integration
• Rapidly audit and iterate on RAG system performance in interactive notebook environments (Jupyter, Colab)