Where are the evaluation configurations stored?

The skill generates a YAML configuration file located at .codex/agent-eval/ .yaml within your project directory.

Can I use real production data for my evaluations?

Yes, Phase 3 of the process allows you to use the langfuse-data-retrieval tool to pull production traces and populate your test datasets.

Does this skill require an existing Langfuse account?

Yes, it utilizes Langfuse-specific tools for data retrieval, dataset management, and prompt management to create the required evaluation assets.

How does this skill explore my agent's codebase?

The skill searches for agent entry points, traces execution flows such as LLM calls and tool usage, and extracts prompt instructions to infer necessary quality dimensions.

What is the primary purpose of the Langfuse Agent Eval Setup skill?

Its primary purpose is to automate the complex process of setting up an evaluation pipeline for AI agents, including code discovery, dataset creation, and judge prompt configuration.

Langfuse Agent Evaluation Setup

Name: Langfuse Agent Evaluation Setup
Author: mberto10

bymberto10

Security & Testing

Automates the end-to-end setup of evaluation pipelines for AI agents using Langfuse observability and datasets.

About

The Langfuse Agent Evaluation Setup skill streamlines the process of establishing rigorous testing frameworks for LLM-powered agents. It intelligently analyzes your codebase to map execution flows, identifies critical quality dimensions, and manages the creation of Langfuse assets like datasets and judge prompts. By generating standardized configuration files and performing initial smoke tests, this skill enables developers to transition from manual 'vibes-based' testing to data-driven performance metrics, ensuring agent reliability through automated evaluation loops.

Key Features

0 GitHub stars
Automated discovery of agent entry points and execution flows
Dynamic quality dimension and threshold definition
Seamless integration with Langfuse dataset and prompt management
Standardized YAML configuration generation for local experiments
Built-in smoke testing for evaluation pipeline verification

Use Cases

Standardizing judge-based evaluation across multiple AI agent components
Creating regression tests using production traces from Langfuse
Establishing a benchmarking suite for agentic workflows

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mberto10/mberto-compound langfuse-agent-eval-setup

For use in Claude.ai and ChatGPT

Download Skill

GitHub