Can I run evaluations on my local GPU?

Yes, this skill supports local execution via vLLM and lighteval if your machine has the necessary GPU hardware and the 'uv' package manager installed.

Does it support the official Open LLM Leaderboard format?

Yes, it specifically manages the 'model-index' metadata format required for leaderboard integration and Papers with Code specification.

What evaluation frameworks are supported?

The skill integrates with lighteval (HuggingFace), inspect-ai (UK AI Safety Institute), and supports both vLLM and accelerate backends.

How does it prevent duplicate pull requests?

The skill includes a mandatory check for existing open PRs via the 'get-prs' command, warning the user before creating new evaluation pull requests.

Hugging Face Evaluation Manager

Name: Hugging Face Evaluation Manager
Author: sickn33

bysickn33

•

31,432

•

データサイエンスとML

Manages and automates structured evaluation results for Hugging Face model cards using local scripts, API imports, and custom GPU evaluations.

This skill streamlines the process of maintaining Hugging Face model documentation by automating the management of evaluation results. It enables developers to extract benchmarks directly from README tables, import verified scores via the Artificial Analysis API, and execute custom model evaluations using high-performance backends like vLLM and lighteval. By ensuring compliance with the model-index metadata format, it simplifies the preparation of models for official leaderboards and Papers with Code integration while preventing duplicate pull requests through built-in repository checks.

主な機能

01Direct integration with Artificial Analysis API for importing benchmark scores.

02Model-index metadata management for Hugging Face leaderboard compatibility.

03Automated extraction of evaluation tables from Markdown READMEs into structured YAML.

04High-performance custom model evaluation using vLLM and lighteval backends.

0531,432 GitHub stars

06Built-in PR management workflow to prevent duplicate evaluation submissions.

ユースケース

01Running standardized MMLU or GSM8K evaluations on custom models using GPU-accelerated infrastructure.

02Automating the sync between local benchmark results and a remote Hugging Face model card.

03Preparing a new LLM release for the Open LLM Leaderboard by generating valid model-index metadata.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add sickn33/antigravity-awesome-skills hugging-face-evaluation

For use in Claude.ai and ChatGPT

主な機能

01Direct integration with Artificial Analysis API for importing benchmark scores.

02Model-index metadata management for Hugging Face leaderboard compatibility.

03Automated extraction of evaluation tables from Markdown READMEs into structured YAML.

04High-performance custom model evaluation using vLLM and lighteval backends.

0531,432 GitHub stars

06Built-in PR management workflow to prevent duplicate evaluation submissions.

ユースケース

01Running standardized MMLU or GSM8K evaluations on custom models using GPU-accelerated infrastructure.

02Automating the sync between local benchmark results and a remote Hugging Face model card.

03Preparing a new LLM release for the Open LLM Leaderboard by generating valid model-index metadata.