tools.csv

name,category,availability,status,primary_url,description
Confident AI,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.confident-ai.com,Confident AI is the AI quality platform built for enterprise platform teams to standardize evals and observability across the org — one consistent bar for how different product teams measure and monitor their AI. Docs
Braintrust,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.braintrust.dev,"Braintrust is an eval-first AI engineering platform with experiments, datasets, a prompt playground, logging, and online scoring using its open-source autoevals scorers."
LangSmith,Full-Stack LLM Evaluation Platforms,commercial,active,https://smith.langchain.com,"LangSmith is LangChain's platform for tracing, dataset-based evaluation, annotation queues, and prompt management; it works with or without the LangChain framework. Docs"
Langfuse,Full-Stack LLM Evaluation Platforms,open-source,active,https://langfuse.com,"Langfuse is an open-source, self-hostable LLM engineering platform with tracing, LLM-as-a-judge and custom evaluations, prompt management, datasets, and dashboards. GitHub"
Arize AX,Full-Stack LLM Evaluation Platforms,commercial,active,https://arize.com,"Arize AX is an enterprise AI observability and evaluation platform; its open-source foundation, Phoenix, is listed separately under observability."
Opik,Full-Stack LLM Evaluation Platforms,open-core,active,https://www.comet.com/site/products/opik/,"Opik, from Comet, is an open-source platform to debug, evaluate, and monitor LLM applications, RAG systems, and agents, with tracing, evaluation metrics, datasets, and dashboards. GitHub"
Galileo,Full-Stack LLM Evaluation Platforms,commercial,active,https://galileo.ai,"Galileo is an enterprise evaluation and observability platform with proprietary evaluation models (e.g., Luna), agent evaluations, guardrailing, and production monitoring."
Patronus AI,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.patronus.ai,"Patronus AI is an automated AI evaluation and testing platform with proprietary evaluators (Lynx for hallucination detection, the Glider judge model), red teaming, and monitoring APIs."
Weights & Biases Weave,Full-Stack LLM Evaluation Platforms,open-core,active,https://wandb.ai/site/weave/,"W&B Weave is a tracing and evaluation toolkit for LLM applications with scorers, comparisons, leaderboards, and production monitoring. GitHub"
Vellum,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.vellum.ai,"Vellum is a platform for building and evaluating LLM workflows with test suites, side-by-side comparisons, and deployment monitoring."
HoneyHive,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.honeyhive.ai,"HoneyHive is an AI observability and evaluation platform with tracing, offline and online evaluators, dataset curation, and human review."
Maxim AI,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.getmaxim.ai,"Maxim AI provides end-to-end simulation, evaluation, and observability for AI agents, including prompt experiments, agent simulation runs, and production quality checks."
Freeplay,Full-Stack LLM Evaluation Platforms,commercial,active,https://freeplay.ai,"Freeplay combines prompt management, evaluation, and observability with human-in-the-loop review workflows for product teams."
Athina AI,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.athina.ai,"Athina AI is a collaborative platform to prototype, evaluate, and monitor LLM applications, with a spreadsheet-style experimentation UI and preset plus custom evals."
Orq.ai,Full-Stack LLM Evaluation Platforms,commercial,active,https://orq.ai,"Orq.ai is a generative AI collaboration platform with built-in experiments, evaluators, guardrails, and deployment management."
Agenta,Full-Stack LLM Evaluation Platforms,open-source,active,https://agenta.ai,"Agenta is an open-source LLMOps platform with a prompt playground, versioning, human and automatic evaluation, and observability. GitHub"
LangWatch,Full-Stack LLM Evaluation Platforms,open-source,active,https://langwatch.ai,"LangWatch is an open-source platform for LLM evaluation, agent simulation testing (Scenario), and observability. GitHub"
Openlayer,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.openlayer.com,"Openlayer provides testing and monitoring for AI systems (LLM and classic ML), with automated tests in CI and production quality alerts."
PromptLayer,Full-Stack LLM Evaluation Platforms,commercial,active,https://www.promptlayer.com,"PromptLayer offers prompt management and evaluation: a visual prompt registry, batch evals, A/B testing, and usage analytics."
Giskard Hub,Full-Stack LLM Evaluation Platforms,open-core,active,https://www.giskard.ai,"Giskard Hub is an enterprise agent-testing platform built on the open-source Giskard library, adding vulnerability scanning, continuous red teaming, and business-failure detection."
Scale GenAI Platform,Full-Stack LLM Evaluation Platforms,commercial,active,https://scale.com,"Scale AI's GenAI platform provides enterprise model evaluation and a data engine, including expert human evaluation and the SEAL leaderboards."
DeepEval,LLM Evaluation Frameworks,open-source,active,https://github.com/confident-ai/deepeval,"DeepEval is an open-source LLM evaluation framework (""Pytest for LLMs"") with 50+ metrics for RAG, agents, multi-turn conversations, safety, and custom criteria, plus CI/CD and benchmark support."
Ragas,LLM Evaluation Frameworks,open-source,active,https://github.com/vibrantlabsai/ragas,"Ragas is an evaluation toolkit for LLM applications, best known for reference-free RAG metrics such as faithfulness, answer relevancy, and context precision/recall. Paper"
promptfoo,LLM Evaluation Frameworks,open-source,active,https://github.com/promptfoo/promptfoo,"promptfoo is a developer-first CLI for testing prompts, models, and RAG/agent applications, with declarative test cases, comparison matrices, CI integration, and red teaming."
OpenAI Evals,LLM Evaluation Frameworks,open-source,active,https://github.com/openai/evals,"OpenAI Evals is OpenAI's original open-source framework and registry of community-contributed evals; the repository is in maintenance mode, and OpenAI's separate hosted Evals product is scheduled for shutdown in late 2026."
openai/simple-evals,LLM Evaluation Frameworks,open-source,active,https://github.com/openai/simple-evals,"simple-evals contains OpenAI's lightweight reference implementations of its reported benchmarks (MMLU, GPQA, SimpleQA, HumanEval, HealthBench, and more)."
lm-evaluation-harness,LLM Evaluation Frameworks,open-source,active,https://github.com/EleutherAI/lm-evaluation-harness,EleutherAI's lm-evaluation-harness is a widely used harness for few-shot LLM benchmarking with hundreds of academic tasks; it powered the Hugging Face Open LLM Leaderboard.
Inspect AI,LLM Evaluation Frameworks,open-source,active,https://github.com/UKGovernmentBEIS/inspect_ai,"Inspect AI is the UK AI Security Institute's framework for LLM evaluations, with solvers, scorers, sandboxed tool use, and agent evals. See also Inspect Evals, a community repository of ready-to-run benchmark implementations. Docs"
HELM,LLM Evaluation Frameworks,open-source,active,https://github.com/stanford-crfm/helm,"HELM (Holistic Evaluation of Language Models), from Stanford CRFM, provides transparent, reproducible, multi-metric model evaluation. Paper"
OpenCompass,LLM Evaluation Frameworks,open-source,active,https://github.com/open-compass/opencompass,OpenCompass is a large-scale evaluation platform supporting 100+ datasets and a wide range of open and API models.
LightEval,LLM Evaluation Frameworks,open-source,active,https://github.com/huggingface/lighteval,"LightEval is Hugging Face's all-in-one LLM evaluation toolkit across multiple backends (transformers, vLLM, inference endpoints); Hugging Face recommends it over the older Evaluate library for LLM work."
Pydantic Evals,LLM Evaluation Frameworks,open-source,active,https://ai.pydantic.dev/evals/,"Pydantic Evals is the evaluation library from the Pydantic team for scoring LLM and agent outputs with datasets, evaluators, and reports. GitHub"
MLflow LLM Evaluate,LLM Evaluation Frameworks,open-source,active,https://mlflow.org/docs/latest/genai/eval-monitor/,MLflow's GenAI evaluation module provides heuristic and LLM-judge metrics integrated with MLflow experiment tracking and tracing. GitHub
NeMo Evaluator,LLM Evaluation Frameworks,open-source,active,https://github.com/NVIDIA-NeMo/Evaluator,"NVIDIA NeMo Evaluator is an open-source platform for scalable, reproducible evaluation of LLMs across popular benchmarks."
Prompt flow,LLM Evaluation Frameworks,open-source,active,https://github.com/microsoft/promptflow,Microsoft Prompt flow is a development suite for LLM apps that includes built-in and custom evaluation flows for quality assessment in CI.
Evidently,LLM Evaluation Frameworks,open-core,active,https://github.com/evidentlyai/evidently,"Evidently is an open-source evaluation and monitoring library for ML and LLM systems with 100+ metrics, test suites, and reports, plus the commercial Evidently Cloud."
TruLens,LLM Evaluation Frameworks,open-source,active,https://github.com/truera/trulens,"TruLens provides evaluation and tracking for LLM apps using feedback functions, and originated the ""RAG triad"" (context relevance, groundedness, answer relevance)."
OpenEvals,LLM Evaluation Frameworks,open-source,active,https://github.com/langchain-ai/openevals,"OpenEvals is LangChain's collection of ready-made evaluators for correctness, conciseness, hallucination, and structured output."
Evalite,LLM Evaluation Frameworks,open-source,active,https://github.com/mattpocock/evalite,"Evalite is a TypeScript-native eval runner built on Vitest that treats evals as tests, with a local UI."
Giskard,LLM Evaluation Frameworks,open-source,active,https://github.com/Giskard-AI/giskard-oss,"Giskard is an open-source testing library for LLM agents and ML models with automatic vulnerability scanning for hallucination, bias, and injection."
EvalScope,LLM Evaluation Frameworks,open-source,active,https://github.com/modelscope/evalscope,EvalScope is ModelScope's framework for LLM/VLM/AIGC evaluation and inference performance benchmarking.
OLMES,LLM Evaluation Frameworks,open-source,active,https://github.com/allenai/olmes,"OLMES is AI2's reproducible standard for LLM evaluations, used for the OLMo and Tülu model reports."
Eureka ML Insights,LLM Evaluation Frameworks,open-source,active,https://github.com/microsoft/eureka-ml-insights,"Eureka ML Insights is Microsoft's framework for standardized, reproducible evaluation of large foundation models."
Weave Evaluations,LLM Evaluation Frameworks,open-source,active,https://github.com/wandb/weave,"Weave Evaluations is W&B Weave's code-first evaluation harness with scorers, datasets, and model comparisons."
LLM Comparator,LLM Evaluation Frameworks,open-source,active,https://github.com/PAIR-code/llm-comparator,"LLM Comparator, from Google's PAIR team, is an interactive tool for side-by-side analysis of LLM evaluation results."
Parea,LLM Evaluation Frameworks,open-source,active,https://github.com/parea-ai/parea-sdk-py,Parea's SDK provides experiment tracking and evaluation for LLM applications.
continuous-eval,LLM Evaluation Frameworks,open-source,active,https://github.com/relari-ai/continuous-eval,"continuous-eval, from Relari, offers data-driven, modular evaluation for LLM pipelines with metric ensembles. Repository activity has slowed since early 2025."
Phoenix Evals,LLM Evaluation Frameworks,open-source,active,https://github.com/Arize-ai/phoenix,"Phoenix Evals is Arize Phoenix's evaluation library with pre-tested LLM-judge templates for hallucination, relevance, toxicity, and more."
UpTrain,LLM Evaluation Frameworks,open-source,active,https://github.com/uptrain-ai/uptrain,UpTrain is an open-source tool to evaluate and improve LLM applications with 20+ preconfigured checks. Repository activity has slowed since 2024.
AlpacaEval,LLM Evaluation Frameworks,open-source,active,https://github.com/tatsu-lab/alpaca_eval,"AlpacaEval is a fast, inexpensive, replicable LLM-based automatic evaluator for instruction-following models."
Evalchemy,LLM Evaluation Frameworks,open-source,active,https://github.com/mlfoundations/evalchemy,"Evalchemy, from ML Foundations, is a unified toolkit for running many popular benchmarks against any model."
Evaluate,LLM Evaluation Frameworks,open-source,active,https://github.com/huggingface/evaluate,"Hugging Face Evaluate is a library of standard NLP/ML metrics (BLEU, ROUGE, BERTScore) with a unified API; it is in maintenance mode, with LightEval recommended for new LLM evaluation work."
FastChat,LLM Evaluation Frameworks,open-source,active,https://github.com/lm-sys/FastChat,"FastChat is LMSYS's platform for serving and evaluating chat models, and the original home of MT-Bench and the Chatbot Arena codebase; development has slowed and it is now mostly of historical reference value."
Microsoft Foundry Evaluations,Cloud Provider Evaluation Services,commercial,active,https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/observability,"Microsoft Foundry (formerly Azure AI Foundry) includes built-in evaluators for generative and agentic applications, covering intent resolution, tool-call accuracy, task adherence, groundedness, and safety."
Vertex AI Gen AI Evaluation,Cloud Provider Evaluation Services,commercial,active,https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview,"Google Cloud's Gen AI Evaluation Service provides model-based and computation-based metrics, including agent trajectory evaluation, integrated with Vertex AI."
Amazon Bedrock Evaluations,Cloud Provider Evaluation Services,commercial,active,https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html,"Amazon Bedrock Evaluations runs managed model and RAG evaluation jobs using automatic metrics, LLM-as-a-judge, and human evaluation workflows."
Mosaic AI Agent Evaluation,Cloud Provider Evaluation Services,commercial,active,https://docs.databricks.com/en/generative-ai/agent-evaluation/index.html,"Databricks' Mosaic AI Agent Evaluation provides managed judges, a human review app, and MLflow integration for evaluating GenAI apps on Databricks."
OpenAI Evals (hosted),Cloud Provider Evaluation Services,commercial,active,https://platform.openai.com/docs/guides/evals,OpenAI's platform-integrated evaluation product for testing prompts and models in the OpenAI dashboard; OpenAI has announced a shutdown of the hosted Evals product scheduled for late 2026.
Anthropic Console Evaluations,Cloud Provider Evaluation Services,commercial,active,https://docs.anthropic.com/en/docs/test-and-evaluate/eval-tool,The Anthropic Console includes an evaluation tool for generating test cases and comparing prompt outputs for Claude models.
Langfuse,LLM Observability and Tracing,open-source,active,https://github.com/langfuse/langfuse,"Langfuse is a widely adopted open-source LLM observability platform with traces, sessions, cost tracking, evaluations, and prompt management; it is self-hostable."
Arize Phoenix,LLM Observability and Tracing,open-source,active,https://github.com/Arize-ai/phoenix,"Phoenix is an OpenTelemetry-native tracing and evaluation library for LLM apps and agents that runs locally, in notebooks, or self-hosted; it underpins the commercial Arize AX platform."
Helicone,LLM Observability and Tracing,open-source,active,https://github.com/Helicone/helicone,"Helicone is an LLM observability proxy/gateway added with one line of code, providing logging, caching, scoring, and experiments."
Opik,LLM Observability and Tracing,open-source,active,https://github.com/comet-ml/opik,Opik is Comet's open-source tracing and evaluation platform for LLM and agentic applications.
OpenLLMetry,LLM Observability and Tracing,open-source,active,https://github.com/traceloop/openllmetry,"OpenLLMetry, from Traceloop, provides OpenTelemetry-based instrumentation for LLM apps, exporting traces to any OTel-compatible backend."
OpenInference,LLM Observability and Tracing,open-source,active,https://github.com/Arize-ai/openinference,"OpenInference is an open specification and set of instrumentation libraries for capturing LLM application traces, used by Phoenix and other OTel backends."
OpenLIT,LLM Observability and Tracing,open-source,active,https://github.com/openlit/openlit,"OpenLIT is an OpenTelemetry-native AI engineering platform with observability, evaluations, guardrails, and a prompt hub."
AgentOps,LLM Observability and Tracing,open-source,active,https://github.com/AgentOps-AI/agentops,"AgentOps provides session replays, metrics, and monitoring purpose-built for AI agents across popular agent frameworks."
Laminar,LLM Observability and Tracing,open-source,active,https://github.com/lmnr-ai/lmnr,"Laminar is an open-source, OTel-based platform for tracing and evaluating AI applications."
Pydantic Logfire,LLM Observability and Tracing,open-core,active,https://github.com/pydantic/logfire,"Logfire, from the Pydantic team, is an observability platform with first-class LLM and agent tracing built on OpenTelemetry."
Lunary,LLM Observability and Tracing,open-source,active,https://github.com/lunary-ai/lunary,"Lunary is a production toolkit for LLM apps covering observability, prompt management, and analytics."
Langtrace,LLM Observability and Tracing,open-source,active,https://github.com/Scale3-Labs/langtrace,"Langtrace is an open-source, OTel-based observability tool with evaluations for LLM applications."
Portkey,LLM Observability and Tracing,open-core,active,https://github.com/Portkey-AI/gateway,"Portkey combines an open-source AI gateway with a commercial observability and governance suite: routing, guardrails, logs, and feedback."
Datadog LLM Observability,LLM Observability and Tracing,commercial,active,https://www.datadoghq.com/product/llm-observability/,"Datadog LLM Observability adds LLM tracing, quality and security scanning, and evaluation to the Datadog platform."
New Relic AI Monitoring,LLM Observability and Tracing,commercial,active,https://newrelic.com/platform/ai-monitoring,"New Relic AI Monitoring provides APM-style monitoring for LLM-powered applications, covering traces, cost, and response quality signals."
LangSmith,LLM Observability and Tracing,commercial,active,https://smith.langchain.com,LangSmith provides tracing and monitoring with online evaluators and alerts (see the [full-stack platforms](#full-stack-llm-evaluation-platforms) section).
Arthur,LLM Observability and Tracing,open-core,active,https://www.arthur.ai,Arthur is an AI performance monitoring platform; its open-source Arthur Bench LLM evaluation tool is no longer actively developed.
LangKit,LLM Observability and Tracing,open-source,active,https://github.com/whylabs/langkit,"LangKit is an open-source library of text-quality and safety telemetry metrics for LLMs, built on whylogs. The hosted WhyLabs platform it fed into has been discontinued (see [Discontinued](#discontinued-and-historical-tools))."
Ragas,RAG Evaluation,open-source,active,https://github.com/vibrantlabsai/ragas,"Ragas is a popular toolkit for RAG metrics: faithfulness, answer relevancy, context precision, and context recall."
DeepEval RAG metrics,RAG Evaluation,open-source,active,https://deepeval.com/docs/metrics-introduction,"DeepEval's RAG metrics cover answer relevancy, faithfulness, and contextual precision/recall/relevancy, with explainable per-test-case reasons."
TruLens,RAG Evaluation,open-source,active,https://github.com/truera/trulens,"TruLens implements RAG triad evaluation (context relevance, groundedness, answer relevance) with application instrumentation."
ARES,RAG Evaluation,open-source,active,https://github.com/stanford-futuredata/ARES,"ARES, from Stanford, automates RAG evaluation using synthetic data and fine-tuned judge models with statistical guarantees. Paper"
AutoRAG,RAG Evaluation,open-source,active,https://github.com/Marker-Inc-Korea/AutoRAG,AutoRAG is an AutoML-style tool that evaluates RAG pipeline configurations on your own data to find the best-performing setup.
RAGChecker,RAG Evaluation,open-source,active,https://github.com/amazon-science/RAGChecker,"RAGChecker, from Amazon Science, is a fine-grained diagnostic framework for RAG using claim-level entailment checks on both retriever and generator."
BEIR,RAG Evaluation,open-source,active,https://github.com/beir-cellar/beir,BEIR is a heterogeneous zero-shot benchmark for information retrieval spanning 18 datasets across 9 task types.
MTEB,RAG Evaluation,open-source,active,https://github.com/embeddings-benchmark/mteb,"MTEB (Massive Text Embedding Benchmark) is the benchmark suite for evaluating embedding models, with a public leaderboard."
CRAG,RAG Evaluation,open-source,active,https://github.com/facebookresearch/CRAG,"CRAG (Comprehensive RAG Benchmark), from Meta, provides 4,400+ QA pairs and mock APIs simulating web and knowledge-graph search, used in the KDD Cup 2024."
RAGBench,RAG Evaluation,open-source,active,https://huggingface.co/datasets/rungalileo/ragbench,"RAGBench is a 100k-example, explainable RAG benchmark across five industry domains, with the TRACe evaluation framework."
BRIGHT,RAG Evaluation,open-source,active,https://github.com/xlang-ai/BRIGHT,BRIGHT is a benchmark for reasoning-intensive retrieval where queries require deep reasoning rather than keyword matching.
MTRAG,RAG Evaluation,open-source,active,https://github.com/IBM/mt-rag-benchmark,"MTRAG, from IBM, is a human-generated benchmark for multi-turn RAG conversations across four domains."
Tonic Validate,RAG Evaluation,open-source,active,https://github.com/TonicAI/tonic_validate,Tonic Validate is a RAG benchmarking and evaluation framework with a free visualization UI.
continuous-eval,RAG Evaluation,open-source,active,https://github.com/relari-ai/continuous-eval,"continuous-eval offers modular, granular evaluation of each RAG pipeline stage: retrieval, reranking, and generation."
FlashRAG,RAG Evaluation,open-source,active,https://github.com/RUC-NLPIR/FlashRAG,"FlashRAG is a Python toolkit for reproducing and evaluating RAG research, with 20+ algorithms and 16+ datasets."
RAGTruth,RAG Evaluation,open-source,active,https://github.com/ParticleMedia/RAGTruth,"RAGTruth is a word-level, hallucination-annotated corpus for training and evaluating trustworthy RAG systems."
RGB,RAG Evaluation,open-source,active,https://github.com/chen700564/RGB,"RGB benchmarks RAG robustness across noise, negative rejection, information integration, and counterfactuals."
DeepEval agentic evals,AI Agent Evaluation,open-source,active,https://deepeval.com/docs/metrics-task-completion,"DeepEval's agentic metrics cover task completion, tool correctness, and argument correctness, supporting end-to-end and component-level agent testing over traces."
LangWatch Scenario,AI Agent Evaluation,open-source,active,https://github.com/langwatch/scenario,"Scenario, from LangWatch, is an agent simulation testing framework that simulates users over multi-turn scenarios and asserts on outcomes."
AgentEvals,AI Agent Evaluation,open-source,active,https://github.com/langchain-ai/agentevals,"AgentEvals is LangChain's set of ready-made evaluators for agent trajectories, including trajectory match and LLM-judge over trajectories."
Inspect AI,AI Agent Evaluation,open-source,active,https://github.com/UKGovernmentBEIS/inspect_ai,"Inspect AI has first-class agent evaluation support: sandboxed tool execution, multi-step solvers, and agent bridges."
Rogue,AI Agent Evaluation,open-source,active,https://github.com/qualifire-dev/rogue,"Rogue, from Qualifire, evaluates and red-teams agents by probing them through realistic adversarial conversations."
any-agent,AI Agent Evaluation,open-source,active,https://github.com/mozilla-ai/any-agent,"any-agent, from Mozilla AI, is a single interface for building and evaluating agents across different agent frameworks."
Vivaria,AI Agent Evaluation,open-source,active,https://github.com/METR/vivaria,Vivaria is METR's platform for running agent evaluations and conducting elicitation research.
Harbor,AI Agent Evaluation,open-source,active,https://github.com/harbor-framework/harbor,Harbor is a framework for running agent evaluations and creating RL environments at scale.
ToolSandbox,AI Agent Evaluation,open-source,active,https://github.com/apple/ToolSandbox,"ToolSandbox, from Apple, is a stateful, conversational, interactive evaluation benchmark for LLM tool-use capabilities."
Coval,Voice AI and Conversational Evaluation,commercial,active,https://www.coval.dev,"Coval is a simulation and evaluation platform for voice and chat agents, applying scenario-based automated testing practices borrowed from self-driving-car simulation."
Hamming,Voice AI and Conversational Evaluation,commercial,active,https://hamming.ai,"Hamming automates voice-agent testing at scale with thousands of simulated phone calls, prompt experiments, and production call analytics."
Cekura,Voice AI and Conversational Evaluation,commercial,active,https://www.cekura.ai,Cekura (formerly Vocera) provides testing and observability for voice and chat agents via simulated conversations and production monitoring.
DeepEval conversational metrics,Voice AI and Conversational Evaluation,open-source,active,https://deepeval.com/docs/metrics-conversational-geval,"DeepEval's conversational metrics evaluate multi-turn conversations at the turn and conversation level (role adherence, knowledge retention, conversation completeness) and include user simulation."
Bland AI Testing,Voice AI and Conversational Evaluation,commercial,active,https://www.bland.ai,Bland AI includes built-in test suites for phone-call agents on the Bland platform.
MultiChallenge,Voice AI and Conversational Evaluation,open-source,active,https://github.com/ScaleAI/MultiChallenge,"MultiChallenge, from Scale AI, benchmarks realistic multi-turn conversation ability of frontier LLMs."
DeepTeam,LLM Red Teaming and Security Testing,open-source,active,https://github.com/confident-ai/deepteam,"DeepTeam is an open-source LLM red-teaming framework covering 120+ vulnerabilities and 20+ attack methods, with mappings to the OWASP Top 10, NIST AI RMF, and MITRE ATLAS."
garak,LLM Red Teaming and Security Testing,open-source,active,https://github.com/NVIDIA/garak,"garak, from NVIDIA, is an LLM vulnerability scanner (""nmap for LLMs"") probing hallucination, jailbreaks, toxicity, and data leakage. Paper"
PyRIT,LLM Red Teaming and Security Testing,open-source,active,https://github.com/Azure/PyRIT,"PyRIT is Microsoft's Python Risk Identification Toolkit for generative AI, used by Microsoft's AI Red Team for automated adversarial probing."
promptfoo red teaming,LLM Red Teaming and Security Testing,open-source,active,https://www.promptfoo.dev/docs/red-team/,"promptfoo's red-teaming module runs automated scans for injection, jailbreaks, PII leakage, and RBAC issues, with OWASP and NIST mappings."
HarmBench,LLM Red Teaming and Security Testing,open-source,active,https://github.com/centerforaisafety/HarmBench,"HarmBench, from the Center for AI Safety, is a standardized evaluation framework for automated red teaming and robust refusal."
Giskard scan,LLM Red Teaming and Security Testing,open-source,active,https://github.com/Giskard-AI/giskard-oss,"Giskard's scan feature automatically detects vulnerabilities in LLM agents, including injection, harmful content, and sycophancy."
FuzzyAI,LLM Red Teaming and Security Testing,open-source,active,https://github.com/cyberark/FuzzyAI,"FuzzyAI, from CyberArk, is an automated LLM fuzzer for discovering jailbreaks and adversarial weaknesses in model guardrails."
CyberSecEval,LLM Red Teaming and Security Testing,open-source,active,https://github.com/meta-llama/PurpleLlama,"CyberSecEval, part of Meta's Purple Llama project, benchmarks LLM cybersecurity risks including insecure code generation and cyberattack helpfulness."
Lakera Red,LLM Red Teaming and Security Testing,commercial,active,https://www.lakera.ai,Lakera Red is an AI red-teaming service from the team behind the Gandalf prompt-injection game.
Mindgard,LLM Red Teaming and Security Testing,commercial,active,https://www.mindgard.ai,Mindgard is an automated AI red teaming and security testing platform (offensive security for AI systems).
HiddenLayer,LLM Red Teaming and Security Testing,commercial,active,https://hiddenlayer.com,HiddenLayer is an AI security platform covering model scanning and adversarial ML detection and response.
Cisco AI Defense,LLM Red Teaming and Security Testing,commercial,active,https://www.cisco.com/site/us/en/products/security/ai-defense/index.html,"Cisco AI Defense, built on the Robust Intelligence acquisition, provides algorithmic red teaming and runtime guardrails for enterprises."
Haize Labs,LLM Red Teaming and Security Testing,commercial,active,https://www.haizelabs.com,"Haize Labs provides automated stress-testing/jailbreaking (""haizing"") suites and judge infrastructure for frontier AI systems."
AgentDojo,LLM Red Teaming and Security Testing,open-source,active,https://github.com/ethz-spylab/agentdojo,"AgentDojo, from ETH Zurich, is a dynamic environment for evaluating prompt-injection attacks and defenses in tool-using LLM agents."
JailbreakBench,LLM Red Teaming and Security Testing,open-source,active,https://github.com/JailbreakBench/jailbreakbench,"JailbreakBench is an open robustness benchmark for jailbreaking, with a leaderboard and artifact repository."
Moonshot,LLM Red Teaming and Security Testing,open-source,active,https://github.com/aiverify-foundation/moonshot,"Moonshot, from the AI Verify Foundation, is a modular tool for benchmarking and red-teaming LLM applications."
NeMo Guardrails,Guardrails and Runtime Safety,open-source,active,https://github.com/NVIDIA/NeMo-Guardrails,"NeMo Guardrails is NVIDIA's toolkit for programmable guardrails (via Colang) on conversational systems: topic control, jailbreak detection, and fact-checking rails."
Guardrails AI,Guardrails and Runtime Safety,open-source,active,https://github.com/guardrails-ai/guardrails,"Guardrails AI is a Python framework for input/output validation, with a community Guardrails Hub of validators."
LLM Guard,Guardrails and Runtime Safety,open-source,active,https://github.com/protectai/llm-guard,"LLM Guard, from Protect AI, is a security toolkit providing sanitization, PII detection, and prompt-injection and toxicity scanners."
LlamaFirewall,Guardrails and Runtime Safety,open-source,active,https://github.com/meta-llama/PurpleLlama/tree/main/LlamaFirewall,"LlamaFirewall, from Meta, is an open-source guardrail framework for agentic systems, combining PromptGuard, agent-alignment checks, and code-safety scanning."
OpenAI Guardrails,Guardrails and Runtime Safety,open-source,active,https://github.com/openai/openai-guardrails-python,OpenAI Guardrails is a modular safety framework for validating LLM inputs and outputs with configurable pipelines.
Invariant Guardrails,Guardrails and Runtime Safety,open-source,active,https://github.com/invariantlabs-ai/invariant,"Invariant, from Invariant Labs, provides guardrailing rules and analysis for agentic systems, including tool-call policy enforcement."
Lakera Guard,Guardrails and Runtime Safety,commercial,active,https://www.lakera.ai/lakera-guard,"Lakera Guard is a low-latency API for detecting prompt injection, content violations, and PII in production."
Amazon Bedrock Guardrails,Guardrails and Runtime Safety,commercial,active,https://aws.amazon.com/bedrock/guardrails/,"Amazon Bedrock Guardrails applies configurable safety policies — content filters, denied topics, and contextual grounding checks — to Bedrock applications."
Azure AI Content Safety,Guardrails and Runtime Safety,commercial,active,https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety,Azure AI Content Safety provides content moderation plus Prompt Shields (jailbreak and indirect injection detection) and groundedness detection.
Google Model Armor,Guardrails and Runtime Safety,commercial,active,https://cloud.google.com/security-command-center/docs/model-armor-overview,"Model Armor is Google Cloud's service for screening LLM prompts and responses for injection, jailbreaks, sensitive data, and harmful content."
Llama Guard,Guardrails and Runtime Safety,open-weights,active,https://github.com/meta-llama/PurpleLlama,"Llama Guard is Meta's family of downloadable safety classifier models (Llama Guard, Prompt Guard) under the Purple Llama project, released under Meta's community license."
OpenAI Moderation,Guardrails and Runtime Safety,commercial,active,https://platform.openai.com/docs/guides/moderation,The OpenAI Moderation endpoint is a free API for text and image safety classification.
Vectara HHEM,Hallucination Detection and Factuality,open-source,active,https://huggingface.co/vectara/hallucination_evaluation_model,"HHEM is Vectara's widely used open hallucination detection model, which powers the Hughes Hallucination Leaderboard."
Patronus Lynx,Hallucination Detection and Factuality,open-weights,active,https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct,"Lynx, from Patronus AI, is a hallucination detection model reported to outperform GPT-4 on the HaluBench benchmark (July 2024 paper). Paper"
SelfCheckGPT,Hallucination Detection and Factuality,open-source,active,https://github.com/potsawee/selfcheckgpt,"SelfCheckGPT performs zero-resource, black-box hallucination detection via sampling consistency. Paper"
FActScore,Hallucination Detection and Factuality,open-source,active,https://github.com/shmsw25/FActScore,"FActScore provides fine-grained, atomic-fact evaluation of factual precision in long-form generation. Paper"
LongFact & SAFE,Hallucination Detection and Factuality,open-source,active,https://github.com/google-deepmind/long-form-factuality,"LongFact (a long-form factuality benchmark) and SAFE (Search-Augmented Factuality Evaluator), from Google DeepMind, evaluate long-form factuality using search-grounded fact checking."
MiniCheck,Hallucination Detection and Factuality,open-source,active,https://github.com/Liyan06/MiniCheck,"MiniCheck is an efficient fact-checking model for verifying whether LLM output sentences are grounded in provided documents, at a fraction of GPT-4's cost."
AlignScore,Hallucination Detection and Factuality,open-source,active,https://github.com/yuh-zha/AlignScore,AlignScore is a unified metric for factual-consistency evaluation trained on a broad alignment corpus.
FacTool,Hallucination Detection and Factuality,open-source,active,https://github.com/GAIR-NLP/factool,"FacTool performs tool-augmented factuality detection across knowledge-based QA, code, math, and scientific review."
TruthfulQA,Hallucination Detection and Factuality,open-source,active,https://github.com/sylinrl/TruthfulQA,TruthfulQA is a benchmark measuring whether models reproduce common human falsehoods. Paper
HaluEval,Hallucination Detection and Factuality,open-source,active,https://github.com/RUCAIBox/HaluEval,HaluEval is a large-scale hallucination evaluation benchmark with 35K samples.
HaluBench,Hallucination Detection and Factuality,open-source,active,https://huggingface.co/datasets/PatronusAI/HaluBench,"HaluBench, from Patronus AI, is a hallucination evaluation benchmark of 15K real-world RAG samples spanning finance and medicine."
FACTS Grounding,Hallucination Detection and Factuality,commercial,active,https://www.kaggle.com/facts-leaderboard,"FACTS Grounding, from Google DeepMind, is a benchmark and leaderboard for grounding quality in long-form responses."
SimpleQA,Hallucination Detection and Factuality,open-source,active,https://github.com/openai/simple-evals,"SimpleQA, from OpenAI, benchmarks short-form factuality and serves as a useful hallucination-rate probe. Paper"
G-Eval,LLM-as-a-Judge Models and Libraries,open-source,active,https://arxiv.org/abs/2303.16634,"G-Eval is a chain-of-thought, form-filling LLM-judging algorithm (Liu et al., 2023); production implementations are available in DeepEval and other frameworks, letting you define custom criteria in plain language."
Prometheus 2,LLM-as-a-Judge Models and Libraries,open-weights,active,https://github.com/prometheus-eval/prometheus-eval,"Prometheus 2 is a family of open judge language models specialized in fine-grained, rubric-based evaluation of other models. Paper"
Glider,LLM-as-a-Judge Models and Libraries,open-weights,active,https://huggingface.co/PatronusAI/glider,"Glider, from Patronus AI, is a 3.8B-parameter explainable rubric-based judge model with span-level highlights."
Atla Selene,LLM-as-a-Judge Models and Libraries,open-weights,active,https://huggingface.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B,"Selene, from Atla, is a small judge model family purpose-trained for evaluation tasks."
JudgeLM,LLM-as-a-Judge Models and Libraries,open-source,active,https://github.com/baaivision/JudgeLM,"JudgeLM, from BAAI, provides fine-tuned, scalable judge models for open-ended benchmarks."
Verdict,LLM-as-a-Judge Models and Libraries,open-source,active,https://github.com/haizelabs/verdict,"Verdict, from Haize Labs, is a declarative library for building compound LLM-judge systems: ensembles, debates, and verification chains."
judges,LLM-as-a-Judge Models and Libraries,open-source,active,https://github.com/quotient-ai/judges,"judges, from Quotient AI, is a small library of curated, research-backed judge prompts."
autoevals,LLM-as-a-Judge Models and Libraries,open-source,active,https://github.com/braintrustdata/autoevals,"autoevals, from Braintrust, is a library of automatic evaluators (factuality, closed-QA, RAG scorers) for LLM outputs."
PandaLM,LLM-as-a-Judge Models and Libraries,open-source,active,https://github.com/WeOpenML/PandaLM,PandaLM is a reproducible judge model for pairwise LLM comparison.
Flow Judge,LLM-as-a-Judge Models and Libraries,open-weights,active,https://github.com/flowaicom/flow-judge,"Flow Judge, from Flow AI, is a compact 3.8B-parameter open evaluator model for customizable rubric evaluation."
RewardBench,Judge and Reward Model Meta-Evaluation,open-source,active,https://github.com/allenai/reward-bench,"RewardBench, from AI2, was the first benchmark for evaluating reward models; RewardBench 2 extends it with harder, unseen human prompts."
JudgeBench,Judge and Reward Model Meta-Evaluation,open-source,active,https://github.com/ScalerLab/JudgeBench,"JudgeBench evaluates LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding."
RM-Bench,Judge and Reward Model Meta-Evaluation,open-source,active,https://github.com/THU-KEG/RM-Bench,RM-Bench evaluates reward models' sensitivity to subtle content changes and resistance to style bias.
LLMBar,Judge and Reward Model Meta-Evaluation,open-source,active,https://github.com/princeton-nlp/LLMBar,"LLMBar, from Princeton NLP, is a meta-evaluation benchmark testing whether LLM evaluators can detect instruction-following correctly against adversarial distractors."
LM-Polygraph,Uncertainty and Calibration,open-source,active,https://github.com/IINemo/lm-polygraph,LM-Polygraph implements a broad suite of uncertainty-quantification methods for LLM outputs with a unified benchmark.
TruthTorchLM,Uncertainty and Calibration,open-source,active,https://github.com/Ybakman/TruthTorchLM,TruthTorchLM is an open-source library of truthfulness-prediction and uncertainty methods for generative LLMs.
Cleanlab TLM,Uncertainty and Calibration,commercial,active,https://cleanlab.ai/tlm/,Cleanlab's Trustworthy Language Model (TLM) adds real-time trustworthiness and confidence scores to any LLM output.
UQ360,Uncertainty and Calibration,open-source,active,https://github.com/IBM/UQ360,"Uncertainty Quantification 360, from IBM, is a toolkit for quantifying and communicating uncertainty in machine-learning models."
MMLU,General Capability Benchmarks,open-source,active,https://github.com/hendrycks/test,MMLU is a 57-subject multiple-choice knowledge benchmark that served for years as the default capability probe. Paper
MMLU-Pro,General Capability Benchmarks,open-source,active,https://github.com/TIGER-AI-Lab/MMLU-Pro,"MMLU-Pro is a harder, more robust successor to MMLU with 10 answer choices and more reasoning-heavy questions."
GPQA,General Capability Benchmarks,open-source,active,https://github.com/idavidrein/gpqa,"GPQA is a graduate-level, ""Google-proof"" science QA benchmark written by domain experts. Paper"
Humanity's Last Exam,General Capability Benchmarks,open-source,active,https://github.com/centerforaisafety/hle,"Humanity's Last Exam contains 2,500 extremely difficult expert-written questions across 100+ subjects, designed as a ""final"" closed-ended academic benchmark. Site"
BIG-bench,General Capability Benchmarks,open-source,active,https://github.com/google/BIG-bench,BIG-bench is a collaborative benchmark of 200+ tasks probing model capabilities and limitations; see also BIG-Bench Hard.
GSM8K,General Capability Benchmarks,open-source,active,https://github.com/openai/grade-school-math,"GSM8K is a benchmark of grade-school math word problems, a classic chain-of-thought reasoning test."
MATH,General Capability Benchmarks,open-source,active,https://github.com/hendrycks/math,"MATH contains 12,500 competition mathematics problems."
FrontierMath,General Capability Benchmarks,commercial,active,https://epoch.ai/frontiermath,"FrontierMath, from Epoch AI, is a private benchmark of exceptionally hard, unpublished research-level math problems."
AIME,General Capability Benchmarks,open-source,active,https://maa.org/aime-thresholds-are-available/,"AIME (American Invitational Mathematics Examination) problems are widely used to test frontier-model math, e.g., via MathArena."
IFEval,General Capability Benchmarks,open-source,active,https://github.com/google-research/google-research/tree/master/instruction_following_eval,"IFEval tests verifiable instruction following (e.g., ""answer in exactly 3 paragraphs""). Paper"
MT-Bench,General Capability Benchmarks,open-source,active,https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge,"MT-Bench is a multi-turn conversation benchmark scored by LLM judges, introduced alongside Chatbot Arena. Paper"
Arena-Hard,General Capability Benchmarks,open-source,active,https://github.com/lmarena/arena-hard-auto,"Arena-Hard is an automatic benchmark built from challenging live Arena prompts, with high correlation to human preference rankings."
LiveBench,General Capability Benchmarks,open-source,active,https://github.com/LiveBench/LiveBench,LiveBench is a contamination-resistant benchmark with monthly-refreshed questions and objective scoring.
ARC-AGI,General Capability Benchmarks,open-source,active,https://github.com/fchollet/ARC-AGI,ARC-AGI is François Chollet's abstraction-and-reasoning corpus and the basis of the ARC Prize.
SimpleBench,General Capability Benchmarks,open-source,active,https://simple-bench.com,SimpleBench is a trick-question benchmark on which average humans still outperform frontier models.
EQ-Bench,General Capability Benchmarks,open-source,active,https://github.com/EQ-bench/EQ-Bench,EQ-Bench benchmarks emotional intelligence and creative writing in LLMs.
HellaSwag,General Capability Benchmarks,open-source,active,https://github.com/rowanz/hellaswag,HellaSwag is a classic commonsense-reasoning benchmark using adversarially filtered sentence completions.
WinoGrande,General Capability Benchmarks,open-source,active,https://github.com/allenai/winogrande,WinoGrande is a large-scale Winograd-schema-style benchmark for commonsense pronoun resolution.
ARC (AI2 Reasoning Challenge),General Capability Benchmarks,open-source,active,https://allenai.org/data/arc,"ARC is AI2's grade-school science question benchmark, still common in base-model reports."
SWE-bench,Coding Benchmarks,open-source,active,https://github.com/SWE-bench/SWE-bench,"SWE-bench requires resolving real GitHub issues in real repositories and is the standard agentic coding benchmark, with Lite, Verified, and Multimodal variants. Paper"
HumanEval,Coding Benchmarks,open-source,active,https://github.com/openai/human-eval,"HumanEval, from OpenAI, contains 164 hand-written function-synthesis problems and originated the pass@k code metric."
MBPP,Coding Benchmarks,open-source,active,https://github.com/google-research/google-research/tree/master/mbpp,"MBPP, from Google Research, contains roughly 1,000 crowd-sourced entry-level Python problems."
EvalPlus,Coding Benchmarks,open-source,active,https://github.com/evalplus/evalplus,EvalPlus re-evaluates HumanEval and MBPP with 80×/35× more tests (HumanEval+ and MBPP+).
BigCodeBench,Coding Benchmarks,open-source,active,https://github.com/bigcode-project/bigcodebench,"BigCodeBench tests practical, tool-using code generation with rich function calls and branch coverage."
LiveCodeBench,Coding Benchmarks,open-source,active,https://github.com/LiveCodeBench/LiveCodeBench,"LiveCodeBench is a contamination-aware code benchmark continuously updated from LeetCode, AtCoder, and Codeforces problems."
Aider Polyglot,Coding Benchmarks,open-source,active,https://aider.chat/docs/leaderboards/,The Aider Polyglot benchmark tests code editing across 225 hard Exercism problems in six languages.
Terminal-Bench,Coding Benchmarks,open-source,active,https://github.com/laude-institute/terminal-bench,"Terminal-Bench evaluates agents on real tasks in a terminal environment: builds, system administration, and data wrangling."
SWE-Lancer,Coding Benchmarks,open-source,active,https://github.com/openai/SWELancer-Benchmark,"SWE-Lancer, from OpenAI, benchmarks 1,400+ real freelance software engineering tasks with a combined task value of $1M USD."
CodeElo,Coding Benchmarks,open-source,active,https://codeelo-bench.github.io,CodeElo benchmarks competition-level code generation with human-comparable Elo ratings from Codeforces.
SciCode,Coding Benchmarks,open-source,active,https://github.com/scicode-bench/SciCode,SciCode is a research-level benchmark of coding problems drawn from real scientific research workflows.
GAIA,Agent Benchmarks,open-source,active,https://huggingface.co/datasets/gaia-benchmark/GAIA,"GAIA contains 466 real-world assistant questions requiring reasoning, browsing, and tool use. Paper"
tau-bench,Agent Benchmarks,open-source,active,https://github.com/sierra-research/tau-bench,"tau-bench (τ-bench), from Sierra, benchmarks tool-using agents that interact with simulated users under domain policies (retail, airline)."
tau2-bench,Agent Benchmarks,open-source,active,https://github.com/sierra-research/tau2-bench,tau2-bench (τ²-bench) extends tau-bench with a dual-control telecom environment where both agent and simulated user act on shared state.
AgentBench,Agent Benchmarks,open-source,active,https://github.com/THUDM/AgentBench,"AgentBench is a comprehensive multi-environment benchmark for LLMs acting as agents across OS, database, web, and game environments."
OSWorld,Agent Benchmarks,open-source,active,https://github.com/xlang-ai/OSWorld,OSWorld benchmarks multimodal agents on real computer-use tasks in real operating systems.
AppWorld,Agent Benchmarks,open-source,active,https://github.com/StonyBrookNLP/appworld,AppWorld is a controllable world of nine simulated apps and 457 APIs for benchmarking interactive coding agents on complex everyday tasks.
WebArena,Agent Benchmarks,open-source,active,https://github.com/web-arena-x/webarena,"WebArena is a realistic, reproducible, self-hosted web environment for autonomous agents; VisualWebArena extends it to visual tasks."
Mind2Web,Agent Benchmarks,open-source,active,https://github.com/OSU-NLP-Group/Mind2Web,Mind2Web is a dataset and benchmark for generalist web agents across 137 real websites.
BrowserGym,Agent Benchmarks,open-source,active,https://github.com/ServiceNow/BrowserGym,"BrowserGym, from ServiceNow, is a gym environment unifying WebArena, MiniWoB++, WorkArena, and more for web-agent evaluation."
BFCL,Agent Benchmarks,open-source,active,https://gorilla.cs.berkeley.edu/leaderboard.html,The Berkeley Function-Calling Leaderboard (BFCL) is the standard benchmark for tool/function-calling accuracy. GitHub
MCPMark,Agent Benchmarks,open-source,active,https://github.com/eval-sys/mcpmark,MCPMark is a stress-testing benchmark for agent capabilities in realistic Model Context Protocol (MCP) environments.
BrowseComp,Agent Benchmarks,open-source,active,https://github.com/openai/simple-evals,"BrowseComp, from OpenAI, benchmarks browsing agents on hard-to-find, entangled information-seeking tasks."
TheAgentCompany,Agent Benchmarks,open-source,active,https://github.com/TheAgentCompany/TheAgentCompany,TheAgentCompany benchmarks LLM agents on consequential real-world professional tasks in a simulated software company.
RE-Bench,Agent Benchmarks,open-source,active,https://github.com/METR/ai-rd-tasks,"RE-Bench, from METR, evaluates frontier AI R&D capabilities of agents against human experts on realistic research-engineering tasks."
AgentHarm,Agent Benchmarks,open-source,active,https://huggingface.co/datasets/ai-safety-institute/AgentHarm,"AgentHarm, from the UK AI Security Institute, measures harmfulness of LLM agents on multi-step malicious tasks."
MLE-bench,Agent Benchmarks,open-source,active,https://github.com/openai/mle-bench,"MLE-bench, from OpenAI, benchmarks agents on Kaggle-style machine-learning engineering tasks."
PaperBench,Agent Benchmarks,open-source,active,https://github.com/openai/preparedness,"PaperBench, from OpenAI, evaluates agents on replicating state-of-the-art AI research papers."
MiniWoB++,Agent Benchmarks,open-source,active,https://github.com/Farama-Foundation/miniwob-plusplus,MiniWoB++ contains 100+ small web-interaction tasks and remains the classic web-agent testbed.
ClawBench,Agent Benchmarks,open-source,active,https://github.com/TIGER-AI-Lab/ClawBench,"ClawBench is an open-source benchmark framework for end-to-end browser-agent tasks on live websites, with isolated runs and replayable five-layer traces. Paper"
AILuminate,Safety and Alignment Benchmarks,open-source,active,https://mlcommons.org/ailuminate/,"AILuminate, from MLCommons, grades systems across 12 hazard categories and is developed by an industry consortium."
HarmBench,Safety and Alignment Benchmarks,open-source,active,https://github.com/centerforaisafety/HarmBench,HarmBench provides standardized red-teaming and refusal-robustness evaluation.
DecodingTrust,Safety and Alignment Benchmarks,open-source,active,https://github.com/AI-secure/DecodingTrust,"DecodingTrust is a comprehensive trustworthiness assessment covering toxicity, stereotype bias, robustness, privacy, ethics, and fairness (NeurIPS 2023 Outstanding Paper)."
SORRY-Bench,Safety and Alignment Benchmarks,open-source,active,https://github.com/SORRY-Bench/sorry-bench,SORRY-Bench systematically evaluates refusal behavior across 45 unsafe topic categories.
XSTest,Safety and Alignment Benchmarks,open-source,active,https://github.com/paul-rottger/xstest,XSTest tests exaggerated safety (over-refusal) using safe prompts that superficially look unsafe.
StrongREJECT,Safety and Alignment Benchmarks,open-source,active,https://github.com/alexandrasouly/strongreject,StrongREJECT is a rigorous benchmark for measuring jailbreak effectiveness.
Do-Not-Answer,Safety and Alignment Benchmarks,open-source,active,https://github.com/Libr-AI/do-not-answer,"Do-Not-Answer collects questions responsible models should refuse, for evaluating safeguards."
WildGuard,Safety and Alignment Benchmarks,open-weights,active,https://github.com/allenai/wildguard,"WildGuard, from AI2, is an open moderation model for prompt harmfulness, response harmfulness, and refusal detection."
TrustLLM,Safety and Alignment Benchmarks,open-source,active,https://github.com/HowieHwong/TrustLLM,"TrustLLM benchmarks trustworthiness across truthfulness, safety, fairness, robustness, privacy, and machine ethics."
BBQ,Safety and Alignment Benchmarks,open-source,active,https://github.com/nyu-mll/BBQ,BBQ (Bias Benchmark for QA) measures social bias across nine demographic dimensions.
ToxiGen,Safety and Alignment Benchmarks,open-source,active,https://github.com/microsoft/TOXIGEN,"ToxiGen, from Microsoft, is a large-scale dataset for implicit and adversarial hate-speech detection across 13 minority groups."
SafetyBench,Safety and Alignment Benchmarks,open-source,active,https://github.com/thu-coai/SafetyBench,SafetyBench is a multiple-choice LLM safety evaluation in English and Chinese.
sycophancy-eval,Safety and Alignment Benchmarks,open-source,active,https://github.com/meg-tong/sycophancy-eval,"sycophancy-eval contains the evaluations from Anthropic's paper ""Towards Understanding Sycophancy in Language Models""."
Global MMLU,Multilingual Evaluation,open-source,active,https://huggingface.co/datasets/CohereLabs/Global-MMLU,"Global MMLU, from Cohere Labs, is a culturally aware version of MMLU professionally translated and verified across 42 languages."
Belebele,Multilingual Evaluation,open-source,active,https://github.com/facebookresearch/belebele,"Belebele, from Meta, is a machine-reading-comprehension benchmark spanning 122 language variants."
MGSM,Multilingual Evaluation,open-source,active,https://huggingface.co/datasets/juletxara/mgsm,MGSM (Multilingual Grade School Math) translates GSM8K problems into 10 typologically diverse languages.
FLORES-200,Multilingual Evaluation,open-source,active,https://github.com/facebookresearch/flores,"FLORES-200, from Meta, is the standard machine-translation evaluation benchmark covering 200 languages."
INCLUDE,Multilingual Evaluation,open-source,active,https://huggingface.co/datasets/CohereLabs/include-base-44,"INCLUDE is a regional-knowledge benchmark built from native exam sources in 44 languages, testing knowledge in original cultural contexts."
C-Eval,Multilingual Evaluation,open-source,active,https://github.com/hkust-nlp/ceval,"C-Eval is a comprehensive Chinese evaluation suite of 13,948 multiple-choice questions across 52 disciplines."
CMMLU,Multilingual Evaluation,open-source,active,https://github.com/haonan-li/CMMLU,"CMMLU measures Chinese language and cultural knowledge across 67 subjects, including China-specific topics."
HealthBench,Domain-Specific Benchmarks,open-source,active,https://github.com/openai/simple-evals,"HealthBench, from OpenAI, evaluates health-related conversations using 5,000 realistic dialogues scored against physician-written rubrics."
LegalBench,Domain-Specific Benchmarks,open-source,active,https://github.com/HazyResearch/legalbench,LegalBench is a collaboratively built benchmark of 162 legal-reasoning tasks designed with legal professionals.
FinanceBench,Domain-Specific Benchmarks,open-source,active,https://github.com/patronus-ai/financebench,"FinanceBench, from Patronus AI, tests open-book financial question answering over real filings."
MedQA,Domain-Specific Benchmarks,open-source,active,https://github.com/jind11/MedQA,MedQA tests medical knowledge with USMLE-style board exam questions.
FinBen,Domain-Specific Benchmarks,open-source,active,https://github.com/The-FinAI/FinBen,FinBen is an extensive open benchmark suite for financial LLM evaluation across dozens of tasks.
Vals AI,Domain-Specific Benchmarks,commercial,active,https://www.vals.ai,"Vals AI runs industry-specific model evaluations (legal, finance, medical, tax) on private datasets and publishes leaderboards."
Needle In A Haystack,Long-Context and Memory Evaluation,open-source,active,https://github.com/gkamradt/LLMTest_NeedleInAHaystack,Needle In A Haystack is the original simple retrieval-from-long-context stress test.
RULER,Long-Context and Memory Evaluation,open-source,active,https://github.com/NVIDIA/RULER,"RULER, from NVIDIA, is a synthetic benchmark that reveals a model's _effective_ context length versus its claimed length."
LongBench v2,Long-Context and Memory Evaluation,open-source,active,https://github.com/THUDM/LongBench,LongBench v2 tests realistic long-context reasoning up to 2M words in English and Chinese.
HELMET,Long-Context and Memory Evaluation,open-source,active,https://github.com/princeton-nlp/HELMET,"HELMET, from Princeton, is an application-centric long-context evaluation suite at 128K+ tokens."
∞Bench,Long-Context and Memory Evaluation,open-source,active,https://github.com/OpenBMB/InfiniteBench,∞Bench (InfiniteBench) benchmarks context windows beyond 100K tokens.
LongMemEval,Long-Context and Memory Evaluation,open-source,active,https://github.com/xiaowu0162/LongMemEval,LongMemEval benchmarks long-term interactive memory in chat assistants across five core memory abilities (ICLR 2025).
LoCoMo,Long-Context and Memory Evaluation,open-source,active,https://github.com/snap-research/locomo,"LoCoMo, from Snap Research, evaluates very long-term conversational memory over multi-session dialogues."
lmms-eval,Multimodal and Vision Evaluation,open-source,active,https://github.com/EvolvingLMMs-Lab/lmms-eval,"lmms-eval is a one-for-all evaluation toolkit for large multimodal models across text, image, video, and audio tasks."
VLMEvalKit,Multimodal and Vision Evaluation,open-source,active,https://github.com/open-compass/VLMEvalKit,"VLMEvalKit, from OpenCompass, supports 220+ vision-language models and 80+ multimodal benchmarks."
MMMU,Multimodal and Vision Evaluation,open-source,active,https://github.com/MMMU-Benchmark/MMMU,MMMU is a massive multi-discipline multimodal understanding benchmark at college level. Paper
MMBench,Multimodal and Vision Evaluation,open-source,active,https://github.com/open-compass/MMBench,MMBench provides systematic multimodal capability evaluation using the CircularEval protocol.
MathVista,Multimodal and Vision Evaluation,open-source,active,https://github.com/lupantech/MathVista,MathVista benchmarks mathematical reasoning in visual contexts.
Video-MME,Multimodal and Vision Evaluation,open-source,active,https://github.com/MME-Benchmarks/Video-MME,Video-MME is a comprehensive benchmark for video analysis by multimodal LLMs (CVPR 2025).
ChartQA,Multimodal and Vision Evaluation,open-source,active,https://github.com/vis-nlp/ChartQA,ChartQA tests question answering about charts requiring visual and logical reasoning.
OmniDocBench,Multimodal and Vision Evaluation,open-source,active,https://github.com/opendatalab/OmniDocBench,OmniDocBench is a comprehensive document-parsing evaluation benchmark (CVPR 2025).
VoiceBench,Audio and Generative Media Evaluation,open-source,active,https://github.com/MatthewCYM/VoiceBench,"VoiceBench benchmarks LLM-based voice assistants on speech interactions across knowledge, instruction following, safety, and robustness."
AudioBench,Audio and Generative Media Evaluation,open-source,active,https://github.com/AudioLLMs/AudioBench,"AudioBench is a universal benchmark for audio LLMs covering speech understanding, audio-scene understanding, and voice understanding."
Full-Duplex-Bench,Audio and Generative Media Evaluation,open-source,active,https://github.com/DanielLin94144/Full-Duplex-Bench,"Full-Duplex-Bench evaluates full-duplex spoken dialogue models on turn-taking: pause handling, backchanneling, and interruption."
VBench,Audio and Generative Media Evaluation,open-source,active,https://github.com/Vchitect/VBench,"VBench is a comprehensive benchmark suite for video generative models, decomposing quality into 16 dimensions."
HEIM,Audio and Generative Media Evaluation,open-source,active,https://github.com/stanford-crfm/helm,"HEIM, from Stanford CRFM (part of the HELM family), is a holistic evaluation of text-to-image models."
GenEval,Audio and Generative Media Evaluation,open-source,active,https://github.com/djghosh13/geneval,GenEval is an object-focused framework for evaluating text-to-image alignment.
GuideLLM,Inference Performance and Load Testing,open-source,active,https://github.com/vllm-project/guidellm,"GuideLLM, from the vLLM project, evaluates and optimizes LLM deployments for real-world inference performance under load."
genai-bench,Inference Performance and Load Testing,open-source,active,https://github.com/sgl-project/genai-bench,"genai-bench, from the SGLang project, provides comprehensive token-level performance benchmarking of LLM serving systems."
GenAI-Perf,Inference Performance and Load Testing,open-source,active,https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_benchmark/genai-perf-README.html,"GenAI-Perf, from NVIDIA, measures TTFT, inter-token latency, tokens per second, and requests per second for generative models served via inference endpoints."
LLMPerf,Inference Performance and Load Testing,open-source,active,https://github.com/ray-project/llmperf,"LLMPerf, from the Ray project, load-tests and validates the performance of LLM APIs."
MLPerf Inference,Inference Performance and Load Testing,open-source,active,https://mlcommons.org/benchmarks/inference/,"MLPerf Inference, from MLCommons, is the industry consortium benchmark for ML and LLM inference performance across hardware and serving stacks."
inference-perf,Inference Performance and Load Testing,open-source,active,https://github.com/kubernetes-sigs/inference-perf,"inference-perf, a Kubernetes SIG project, is a GenAI inference performance benchmarking tool for Kubernetes deployments."
AIPerf,Inference Performance and Load Testing,open-source,active,https://github.com/ai-dynamo/aiperf,"AIPerf, from NVIDIA's Dynamo project, provides comprehensive benchmarking of generative AI serving performance at scale."
LMArena (Chatbot Arena),Leaderboards and Arenas,open-source,active,https://lmarena.ai,LMArena ranks models via crowdsourced pairwise human-preference battles with Elo-style ratings. Paper
Artificial Analysis,Leaderboards and Arenas,commercial,active,https://artificialanalysis.ai,"Artificial Analysis independently benchmarks models and providers on intelligence, price, latency, and throughput."
Open LLM Leaderboard (archived),Leaderboards and Arenas,open-source,active,https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard,Hugging Face's Open LLM Leaderboard ranked open models until its retirement in 2025; it remains a useful historical archive.
LiveBench,Leaderboards and Arenas,open-source,active,https://livebench.ai,LiveBench maintains a contamination-limited leaderboard with monthly-refreshed questions.
Scale SEAL,Leaderboards and Arenas,commercial,active,https://scale.com/leaderboard,"SEAL, from Scale AI, publishes private, expert-evaluated leaderboards designed to resist overfitting."
Vellum LLM Leaderboard,Leaderboards and Arenas,commercial,active,https://www.vellum.ai/llm-leaderboard,The Vellum LLM Leaderboard aggregates frontier-model comparisons across popular benchmarks.
Epoch AI Benchmarking Hub,Leaderboards and Arenas,open-source,active,https://epoch.ai/benchmarks,Epoch AI's Benchmarking Hub publishes independent evaluations and trend analyses of frontier models over time.
MTEB Leaderboard,Leaderboards and Arenas,open-source,active,https://huggingface.co/spaces/mteb/leaderboard,The MTEB Leaderboard ranks text-embedding models.
BFCL Leaderboard,Leaderboards and Arenas,open-source,active,https://gorilla.cs.berkeley.edu/leaderboard.html,The BFCL Leaderboard ranks models on function/tool-calling accuracy.
SWE-bench Leaderboard,Leaderboards and Arenas,open-source,active,https://www.swebench.com,The official SWE-bench leaderboard tracks agentic coding performance across SWE-bench variants.
ARC Prize Leaderboard,Leaderboards and Arenas,open-source,active,https://arcprize.org/leaderboard,The ARC Prize leaderboard tracks progress on ARC-AGI-1 and ARC-AGI-2.
Aider LLM Leaderboards,Leaderboards and Arenas,open-source,active,https://aider.chat/docs/leaderboards/,Aider's leaderboards rank models on polyglot code editing.
Vectara Hallucination Leaderboard,Leaderboards and Arenas,open-source,active,https://github.com/vectara/hallucination-leaderboard,"The Hughes Hallucination Leaderboard ranks models by hallucination rate when summarizing documents, scored with HHEM."
LLM Decontaminator,Benchmark Contamination and Evaluation Reliability,open-source,active,https://github.com/lm-sys/llm-decontaminator,"LLM Decontaminator, from LMSYS, detects rephrased benchmark samples in training data that string matching misses. Paper"
LiveBench,Benchmark Contamination and Evaluation Reliability,open-source,active,https://github.com/LiveBench/LiveBench,LiveBench limits contamination structurally by releasing new questions monthly.
LiveCodeBench,Benchmark Contamination and Evaluation Reliability,open-source,active,https://github.com/LiveCodeBench/LiveCodeBench,LiveCodeBench tags every problem with a release date so models can be scored only on problems published after their training cutoff.
Adding Error Bars to Evals,Benchmark Contamination and Evaluation Reliability,open-source,active,https://arxiv.org/abs/2411.00640,Anthropic's statistical guidance for reporting confidence intervals and significance in LLM evaluations.
DeepEval Synthesizer,Synthetic Data and Test Set Generation,open-source,active,https://deepeval.com/docs/synthesizer-introduction,"DeepEval's Synthesizer generates synthetic goldens from documents, contexts, or from scratch, with evolution-based complexity control."
Ragas Testset Generation,Synthetic Data and Test Set Generation,open-source,active,https://docs.ragas.io/en/stable/concepts/test_data_generation/,Ragas generates knowledge-graph-based synthetic test sets for RAG pipelines.
YourBench,Synthetic Data and Test Set Generation,open-source,active,https://github.com/huggingface/yourbench,"YourBench, from Hugging Face, generates custom benchmarks from your own documents, cheaply and without contamination."
distilabel,Synthetic Data and Test Set Generation,open-source,active,https://github.com/argilla-io/distilabel,"distilabel, from Argilla, is a framework for synthetic data and AI-feedback pipelines for dataset creation."
DataDreamer,Synthetic Data and Test Set Generation,open-source,active,https://github.com/datadreamer-dev/DataDreamer,DataDreamer is a Python library for synthetic data generation and reproducible LLM workflows.
Meta Synthetic Data Kit,Synthetic Data and Test Set Generation,open-source,active,https://github.com/meta-llama/synthetic-data-kit,"Synthetic Data Kit, from Meta, generates reasoning traces and QA pairs for fine-tuning and evaluation datasets."
NeMo Curator,Synthetic Data and Test Set Generation,open-source,active,https://github.com/NVIDIA-NeMo/Curator,"NeMo Curator, from NVIDIA, is a scalable data curation and synthetic-generation pipeline for training and evaluation corpora."
Gretel,Synthetic Data and Test Set Generation,commercial,active,https://gretel.ai,"Gretel (acquired by NVIDIA) is a general synthetic-data platform for tabular, text, and structured data with privacy guarantees."
Mostly AI,Synthetic Data and Test Set Generation,open-core,active,https://mostly.ai,Mostly AI provides a synthetic-data SDK and platform for privacy-safe tabular and structured data.
Snorkel,Synthetic Data and Test Set Generation,commercial,active,https://snorkel.ai,"Snorkel AI provides programmatic data labeling and curation (Snorkel Flow), including expert-driven LLM evaluation dataset development; it grew out of the open-source Snorkel research project."
Label Studio,Human Annotation and Labeling,open-core,active,https://github.com/HumanSignal/label-studio,"Label Studio is a widely used open-source data labeling tool supporting text, image, audio, video, and LLM response ranking."
Argilla,Human Annotation and Labeling,open-source,active,https://github.com/argilla-io/argilla,"Argilla is a collaboration tool for building high-quality LLM datasets: rating, ranking, and feedback workflows."
Prodigy,Human Annotation and Labeling,commercial,active,https://prodi.gy,"Prodigy, from the spaCy team, is a scriptable annotation tool with model-in-the-loop workflows."
Labelbox,Human Annotation and Labeling,commercial,active,https://labelbox.com,Labelbox combines a labeling platform with managed human evaluation for RLHF and evals.
Scale,Human Annotation and Labeling,commercial,active,https://scale.com,Scale AI provides managed expert data annotation and human evaluation at frontier-lab scale.
Surge AI,Human Annotation and Labeling,commercial,active,https://www.surgehq.ai,Surge AI provides human data labeling and RLHF workforces used by frontier labs.
Toloka,Human Annotation and Labeling,commercial,active,https://toloka.ai,Toloka provides a global expert workforce for AI data and human evaluation.
SuperAnnotate,Human Annotation and Labeling,commercial,active,https://www.superannotate.com,SuperAnnotate is an enterprise annotation platform including LLM evaluation and fine-tuning data workflows.
Encord,Human Annotation and Labeling,commercial,active,https://encord.com,"Encord is a data development platform with annotation and model-evaluation tooling, strongest in vision."
MLflow,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/mlflow/mlflow,"MLflow provides experiment tracking, a model registry, and evaluation for ML and GenAI."
Weights & Biases,Classic ML Evaluation and Monitoring,open-core,active,https://wandb.ai,"Weights & Biases provides experiment tracking, sweeps, a model registry, and reports, plus Weave for LLM work."
Comet,Classic ML Evaluation and Monitoring,open-core,active,https://www.comet.com,"Comet provides experiment management and production model monitoring, plus Opik for LLM work."
ClearML,Classic ML Evaluation and Monitoring,open-core,active,https://github.com/clearml/clearml,"ClearML is an open-source MLOps suite for experiments, orchestration, and serving."
DVC,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/iterative/dvc,DVC provides Git-based data and experiment versioning.
Deepchecks,Classic ML Evaluation and Monitoring,open-core,active,https://github.com/deepchecks/deepchecks,"Deepchecks tests and validates ML models and data from research to production, and offers an LLM evaluation product."
Great Expectations,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/great-expectations/great_expectations,Great Expectations is a widely used framework for data quality testing and validation.
whylogs,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/whylabs/whylogs,whylogs is an open standard for data logging and ML telemetry profiles.
NannyML,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/NannyML/nannyml,NannyML estimates post-deployment model performance and detects silent failure without ground truth.
Fairlearn,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/fairlearn/fairlearn,Fairlearn is a Python package for assessing and improving the fairness of ML models.
AI Fairness 360,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/Trusted-AI/AIF360,"AIF360, from IBM (now LF AI), is a toolkit of fairness metrics and bias-mitigation algorithms."
Alibi Detect,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/SeldonIO/alibi-detect,"Alibi Detect, from Seldon, provides outlier, adversarial, and drift detection for ML systems."
Cleanlab,Classic ML Evaluation and Monitoring,open-source,active,https://github.com/cleanlab/cleanlab,Cleanlab automatically detects label errors and data issues in ML datasets (data-centric evaluation).
Fiddler,Classic ML Evaluation and Monitoring,commercial,active,https://www.fiddler.ai,"Fiddler provides enterprise AI observability: model monitoring, explainability, and LLM monitoring."
Evidently,Classic ML Evaluation and Monitoring,open-core,active,https://github.com/evidentlyai/evidently,"Evidently provides drift detection, data quality checks, and performance monitoring for ML and LLM systems."
Literal AI,Discontinued and Historical Tools,discontinued,discontinued,https://literalai.com,"Literal AI, from the Chainlit team, announced its service shutdown effective October 31, 2025."
Neptune,Discontinued and Historical Tools,discontinued,discontinued,https://neptune.ai,Neptune's hosted experiment tracker was discontinued in March 2026 following the company's acquisition; existing users were migrated off the platform.
WhyLabs Platform,Discontinued and Historical Tools,discontinued,discontinued,https://whylabs.ai,The hosted WhyLabs observability platform discontinued operations; the open-source whylogs and LangKit libraries remain available.
OpenAI Evals (hosted),Discontinued and Historical Tools,discontinued,discontinued,https://platform.openai.com/docs/guides/evals,OpenAI announced the hosted Evals dashboard/API will shut down in late 2026; the open-source `openai/evals` repository remains available in maintenance mode.
Arthur Bench,Discontinued and Historical Tools,discontinued,discontinued,https://github.com/arthur-ai/bench,Arthur Bench is no longer actively developed; Arthur's commercial platform continues.