Measures inter-rater agreement across multiple large language models for robust AI evaluation pipelines.
Sponsored
Conkurrence is a comprehensive AI evaluation toolkit designed to assess the consensus among various large language models (LLMs). By treating multiple AI models as independent raters, it applies statistically validated methods like Fleiss' kappa with bootstrap confidence intervals to measure inter-rater reliability. The tool can also identify and route contested evaluation items to human experts, ensuring a robust and reliable AI assessment process. It integrates seamlessly with MCP-compatible clients like Claude Desktop, offering features for multi-model evaluation, schema design, trend tracking, and cost estimation.
主な機能
01Multi-model evaluation across Bedrock, OpenAI, and Gemini models simultaneously
02Statistical rigor with Fleiss' kappa, bootstrap confidence intervals, and Kendall's W
03Self-consistency mode that works without API keys using host model via MCP Sampling
04AI-powered schema design and suggestion from your data
05Trend tracking to compare runs over time and detect agreement degradation
060 GitHub stars
ユースケース
01Evaluating the consistency and reliability of AI models in an evaluation pipeline
02Identifying disagreements and discrepancies when comparing multiple LLM providers
03Designing, validating, and optimizing AI evaluation schemas for specific tasks
