Configures comprehensive Langfuse datasets with custom evaluation dimensions and LLM-as-judge prompts.
This skill streamlines the process of establishing evaluation environments within Langfuse, helping developers define regression sets, A/B tests, and golden datasets for AI models. It guides users through requirement gathering, CLI-based dataset creation, and the configuration of scoring mechanisms. By integrating with judge prompts, it enables automated LLM-as-judge workflows, ensuring that your AI applications maintain high standards of accuracy, relevance, and safety throughout the development lifecycle.
Key Features
01Supports both human review and automated scoring configurations
02Facilitates golden set creation for regression and A/B testing
03Streamlines LLM-as-judge prompt setup and management
04Automated dataset creation with purpose-driven metadata
05Configures evaluation dimensions like accuracy, tone, and completeness
061 GitHub stars
Use Cases
01Establishing a golden dataset for LLM regression testing
02Setting up an automated LLM-as-judge framework for production monitoring
03Configuring A/B testing dimensions for comparing different prompt versions
