Analyzes AI agent evaluation traces and performance metrics to optimize LLM application behavior and cost.
The AgentV Trace Analyst skill provides deep visibility into AI agent performance by processing AgentV evaluation traces and result files. It enables developers to compute detailed percentile statistics for scores, latency, and costs, visualize execution paths through tree views, and perform head-to-head A/B comparisons between different runs. By identifying failure patterns, tool usage bottlenecks, and regression deltas, it helps refine agent logic and ensure high-quality, cost-effective tool trajectories.
Características Principales
01Detailed A/B comparison between baseline and candidate runs to detect regressions
02Advanced jq-powered querying for custom data filtering and tool frequency analysis
03Execution path visualization showing LLM calls and tool invocations
04Percentile-based statistical analysis for score, latency, and cost metrics
05Chronological listing and inspection of AgentV evaluation result files
0611 GitHub stars
Casos de Uso
01Debugging agent failures by inspecting failed assertions and execution trajectories
02Validating prompt and configuration changes through regression testing and delta analysis
03Optimizing LLM costs and performance by identifying high-latency and high-token outliers
