What benchmarks can I run with this skill?

It supports a wide range of benchmarks categorized by model type, including General Knowledge, Coding, Core Reasoning, Agentic tasks, and Multilingual evaluations.

What is the NeMo Evaluator Launcher (NEL)?

NEL is an open-source library from NVIDIA designed for scalable, reproducible evaluation of AI models across various benchmarks and deployment backends.

Can this skill help with multi-node deployments?

Yes, the assistant supports advanced patterns including multi-instance independent deployments with HAProxy and multi-node single-instance setups using Ray for pipeline parallelism.

Does it support HuggingFace models?

Absolutely. You can provide a HuggingFace handle, and the skill will use WebSearch to find the model card and extract optimal sampling, context, and deployment parameters.

NeMo Evaluator Assistant

Name: NeMo Evaluator Assistant
Author: NVIDIA-NeMo

byNVIDIA-NeMo

•

230

•

数据科学与机器学习

Streamlines the configuration and execution of large-scale AI model evaluations using the NVIDIA NeMo Evaluator Launcher.

The NeMo Evaluator Assistant is a specialized workflow tool for managing the lifecycle of AI model benchmarking. It guides users through an interactive process to generate production-ready YAML configurations, optimize deployment parameters via automated model card analysis, and handle complex scaling requirements like multi-node Ray clusters or HAProxy load balancing. By automating environment checks, task selection, and deployment-specific arguments for backends like vLLM and SGLang, it ensures reproducible and high-performance evaluation runs on local hardware or SLURM clusters.

主要功能

01Built-in task management for core reasoning, coding, and agentic benchmarks

02Support for multiple deployment backends including vLLM, SGLang, NIM, and TRT-LLM

03Advanced multi-node scaling patterns for massive models (>120B parameters)

04Interactive YAML configuration builder for local and SLURM environments

05Automated model parameter extraction from HuggingFace and NVIDIA model cards

06230 GitHub stars

使用场景

01Scaling model evaluations across multi-node SLURM clusters using Ray or HAProxy

02Integrating automated evaluation results with MLflow or Weights & Biases tracking

03Generating reproducible benchmark configurations for new LLM checkpoints

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nvidia-nemo/evaluator nel-assistant

For use in Claude.ai and ChatGPT

Download Skill