What happens if my data is in CSV format?

The skill includes a workflow to read CSV files and convert them into the required JSONL format, confirming the schema with you before proceeding.

What does the BYOB skill do in NeMo Evaluator?

BYOB stands for 'Bring Your Own Benchmark.' It provides a decorator-based framework to define custom datasets, prompts, and scoring logic to evaluate Large Language Models effectively.

How do I run the benchmarks I create?

The skill provides CLI commands to compile your benchmark into a package and can even containerize it into a Docker image for deployment in CI/CD pipelines.

Can I use datasets directly from HuggingFace?

Yes, the skill supports 'hf://' URIs to download and process HuggingFace datasets automatically during the benchmark compilation process.

Does this skill support subjective human-like scoring?

Yes, it includes an LLM-as-judge feature with built-in templates for binary QA, Likert scales, and safety assessments to grade responses subjectively.

NeMo Evaluator BYOB

Name: NeMo Evaluator BYOB
Author: NVIDIA-NeMo

byNVIDIA-NeMo

•

273

•

Data Science & ML

Creates and deploys custom LLM evaluation benchmarks using the BYOB decorator framework for scalable model testing.

The BYOB (Bring Your Own Benchmark) skill for NeMo Evaluator streamlines the process of building specialized evaluation pipelines for AI models. It guides developers through a five-step workflow—covering dataset preparation, prompt templating, and scoring logic—to transform raw data into reproducible benchmarks. With support for built-in metrics, custom Python scorers, and LLM-as-judge evaluation, this skill enables precise quality control and performance tracking for any domain-specific language model task.

Key Features

01Support for both standard metrics (F1, ROUGE, BLEU) and sophisticated LLM-as-judge scoring

02One-command compilation and Docker containerization for scalable evaluation runs

03Automatic data conversion and field mapping for CSV, JSON, and JSONL formats

04Built-in smoke testing to validate scoring logic before deployment

05Step-by-step onboarding for creating custom benchmarks from local files or HuggingFace datasets

06273 GitHub stars

Use Cases

01Standardizing evaluation workflows across research teams via containerized benchmarks

02Building domain-specific evaluation sets for medical, legal, or financial LLM applications

03Implementing subjective quality assessments using Llama-3 or other models as evaluators

Key Features

01Support for both standard metrics (F1, ROUGE, BLEU) and sophisticated LLM-as-judge scoring

02One-command compilation and Docker containerization for scalable evaluation runs

03Automatic data conversion and field mapping for CSV, JSON, and JSONL formats

04Built-in smoke testing to validate scoring logic before deployment

05Step-by-step onboarding for creating custom benchmarks from local files or HuggingFace datasets

06273 GitHub stars

Use Cases

01Standardizing evaluation workflows across research teams via containerized benchmarks

02Building domain-specific evaluation sets for medical, legal, or financial LLM applications

03Implementing subjective quality assessments using Llama-3 or other models as evaluators