Evaluates Agentic GraphRAG performance using automated benchmarking and LLM-as-judge scoring rubrics.
The AgRAG LLM Judge skill enables developers to systematically assess the performance of Agentic GraphRAG systems, particularly those tailored for complex domains like telecommunications test scope analysis. It automates the process of running batch prompts in headless mode while maintaining thread consistency to ensure reliable evaluations of retrieval quality, grounding, and tool usage. By applying a structured rubric that measures metrics like graph path validity and tool efficiency, the skill generates detailed transcripts and diagnostic reports, helping developers identify failure modes and apply recommended fixes to improve agentic accuracy.
主要功能
01Automated tool usage auditing to detect redundant searches and missing results
02Detailed transcript generation including entity IDs and graph traversal paths
03Diagnostic script for identifying RAG-specific failure modes and proposing fixes
04Headless batch processing with consistent thread-id context tracking
05Structured LLM-as-judge scoring based on a 5-point performance rubric
060 GitHub stars
使用场景
01Validating graph path integrity for domain-specific knowledge retrieval
02Benchmarking retrieval accuracy in complex GraphRAG implementations
03Debugging hallucination and grounding issues in agentic tool-calling workflows
