Can I use production data for my agent evaluations?

Yes, the skill can utilize existing Langfuse traces from production to create realistic and diverse datasets for testing.

How does the skill identify my agent's logic?

The skill scans your codebase for specific entry points, tool definitions, and prompt templates to infer the execution flow and necessary evaluation criteria.

Do I need an existing Langfuse account to use this skill?

Yes, this skill leverages Langfuse services to store datasets and manage judge prompts, so an active project and API access are required.

Where are the evaluation configurations stored?

The skill generates a standardized YAML configuration file within the .codex/agent-eval/ directory of your project.

What does the Langfuse Agent Eval Setup skill do?

It automates the configuration of evaluation pipelines for AI agents, including dataset creation, prompt management, and quality dimension mapping within the Langfuse ecosystem.

Langfuse Agent Evaluation Setup

Name: Langfuse Agent Evaluation Setup
Author: mberto10

bymberto10

•

Security & Testing

Automates the creation of comprehensive evaluation pipelines for AI agents using Langfuse observability and testing frameworks.

This skill streamlines the complex process of setting up evaluation frameworks for AI agents within your development environment. It automatically discovers agent logic and execution flows, helps define quality dimensions like accuracy and cost, manages Langfuse datasets and judge prompts, and generates a structured local configuration file. By bridging the gap between development and production observability, it ensures developers can systematically measure agent performance and catch regressions early in the lifecycle.

Key Features

01Dynamic definition of quality dimensions and scoring thresholds

021 GitHub stars

03Integration with Langfuse for dataset and prompt management

04Generation of structured YAML evaluation configurations

05Built-in smoke testing to verify evaluation pipeline integrity

06Automated discovery of agent entry points and execution flows

Use Cases

01Establishing a performance baseline for a newly developed AI agent

02Converting production traces into regression test datasets

03Implementing LLM-as-a-judge workflows for automated quality scoring

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mberto10/mberto-compound langfuse-agent-eval-setup

For use in Claude.ai and ChatGPT

Download Skill