What is the 'Compound' phase in the optimization loop?

The compounding phase refers to turning failures into new test cases and persisting decisions in a journal, ensuring your evaluation dataset grows and agent performance improves every iteration.

Does this require an existing Langfuse setup?

Yes, it works best when your agent is already generating Langfuse traces with key steps instrumented so the advisor can analyze actual performance data.

Where are the iteration outcomes stored?

Outcomes are recorded in a standardized journal file located at .claude/optimization-loops/ /journal.yaml for long-term tracking and knowledge persistence.

Can it help if I don't have production data yet?

Yes, the skill can propose synthetic data generation strategies to build an initial evaluation set until you have real production traces to work with.

How does this skill help with agent evaluation?

It provides a structured framework for measuring output quality, trajectory efficiency, and safety, helping you move from manual testing to automated evaluations using Langfuse.

Langfuse Agent Advisor

Name: Langfuse Agent Advisor
Author: mberto10

bymberto10

Analytics & Monitoring

Provides strategic guidance for evaluating and optimizing AI agents using Langfuse traces and data-driven iteration loops.

About

Langfuse Agent Advisor is a specialized skill designed to help developers systematically evaluate and improve AI agents. It guides users through establishing evaluation frameworks that cover output quality, trajectory/process efficiency, and safety. By implementing a structured 'Hypothesize-Experiment-Analyze-Compound' loop, the skill helps teams move beyond vibes-based development to rigorous optimization, utilizing Langfuse traces to build high-quality datasets and track performance improvements over time in a persistent optimization journal.

Key Features

Structured optimization loop for testable hypotheses and metric-driven experiments.
Strategic dataset construction including golden sets, edge cases, and adversarial inputs.
Phase-aligned checklists for running experiments and comparing traces.
0 GitHub stars
Automated logging of iteration outcomes in a persistent journal format.
Comprehensive evaluation frameworks covering output quality, trajectory, and safety.

Use Cases

Debugging complex agent trajectories where reasoning steps need validation alongside final outputs.
Building a growing, persistent evaluation dataset based on identified production failures.
Transitioning experimental agent prototypes into production-ready systems with high reliability.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mberto10/mberto-compound langfuse-agent-advisor

For use in Claude.ai and ChatGPT

Download Skill

GitHub