What kind of improvements can I identify using this skill?

You can identify gaps such as missing tool parameters, unclear instructions that lead to guesswork, and unnecessary steps caused by tool limitations.

Does this skill judge if the agent's answer was correct?

No, the primary goal is not to judge correctness but to identify what system changes (like better docs or tools) would help the agent take a more efficient path.

What is a trajectory analysis?

It is a comprehensive review of the specific steps, tool calls, and internal reasoning an AI agent used to solve a task, aimed at finding system-level improvements.

Where are the analysis findings stored?

Findings are written to a structured markdown file (findings.md) within a specific data directory, organized by user and chat ID.

Can I see the specific code the agent executed?

Yes, the skill extracts and organizes all tool calls, including SQL, Python scripts, and large text results into dedicated directories for inspection.

Trajectory Analysis & Evaluation

Name: Trajectory Analysis & Evaluation
Author: josca42

byjosca42

•

Analytics & Monitoring

Analyzes agent conversation trajectories to identify concrete improvements for system instructions, tool definitions, and documentation.

This skill enables developers to perform deep-dive reviews of AI agent performance by generating structured trajectory reports from chat histories. Instead of judging answer correctness, it focuses on identifying system-level gaps—such as ambiguous instructions, incomplete tool outputs, or inefficient exploration paths—that hinder agent performance. By transforming raw logs into a standardized evaluation framework, it provides actionable insights to refine the agent's environment, optimize tool usage, and enhance overall trajectory efficiency for complex tasks.

Key Features

01Standardized reporting format for specific, high-impact system improvements

02Automated generation of turn-by-turn trajectory files and artifacts

031 GitHub stars

04Idempotent file generation to prevent redundant processing of existing turns

05Structured evaluation framework for instructions, tools, and efficiency

06Detailed artifact extraction including SQL queries, Python code, and visual plots

Use Cases

01Refining system prompts based on observed agent confusion or guesswork during complex workflows

02Debugging why an agent failed to find specific data despite having the necessary tools

03Auditing tool usage to consolidate multiple redundant calls into a single efficient tool

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add josca42/varro analyse-trajectory

For use in Claude.ai and ChatGPT