Evaluates AI code generation models using industry-standard benchmarks and pass@k metrics.
The BigCode Evaluation Harness is a comprehensive benchmarking suite designed to measure the functional correctness of code generation models. It provides a standardized framework for running 15+ benchmarks—including HumanEval, MBPP, and MultiPL-E—across 18 programming languages. By automating code execution in secure environments and calculating pass@k metrics, it enables researchers and engineers to rigorously compare model performance, test instruction-tuning effectiveness, and validate multi-language capabilities against the same standards used by the HuggingFace leaderboards.
主な機能
01Support for 15+ benchmarks including HumanEval, MBPP, and APPS
02Instruction-tuned model testing with custom prompt templates and tokens
03Standardized pass@k metric calculation for objective performance measurement
04Multi-language evaluation across 18 different programming languages via MultiPL-E
053,983 GitHub stars
06Secure code execution using Docker containers to prevent host contamination
ユースケース
01Measuring the impact of fine-tuning or quantization on a model's functional correctness
02Benchmarking custom-trained code models against industry leaders like StarCoder or CodeLlama
03Validating the multi-language coding proficiency of a new LLM across 18 languages
