BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation

Introduction

We introduce BioVerge, a benchmark that integrates structured triplets from PubTator3 with textual PubMed literature, and apply a strict temporal cutoff of January 1, 2024 to prevent data contamination. The test set is further curated using journal impact factors to provide a reliable evaluation set.

We also propose BioVerge Agent, an LLM agent framework that alternates between Generation and Evaluation modules to propose, refine, and self-assess hypotheses. We develop both Single and Double Agent architectures, which differs in memory log sharing, to study to how design choices influence BioVerge Agent's reasoning strategies, exploration, and performance.

We document our benchmark and agent construction in the sections below.

---

Project Setup

Download the project repository. Ensure that conda or miniconda is installed on your device.

Under the home project directory, run:

conda env create -f environment.yml -n bioverge
conda activate bioverge

This would install the necessary dependencies for dataset construction and agent execution.

Dataset

Dataset Download

Our dataset is available for download here: https://drive.google.com/file/d/1VruZiXabiPCh6L1HcDGzBu1iq0g0uf-4/view?usp=sharing

Move the tarball to the project root. Under project root run:

mkdir -p data
tar -xzvf bioverge_dataset.tar.gz -C data
rm bioverge_dataset.tar.gz

Make sure all dataset files are placed under BioVerge/data/ since that is the default data directory. Note: If you download the dataset, you can skip the dataset construction below and continue to the Agent section.

Dataset Construction

To construct the dataset from processing scripts, follow the processing pipeline below. (note that raw data files require a significant storage, so please reserve enough disk space for download and processing)

1. From project root, change directory to dataset_constuction:

cd dataset_constuction/

2. Execute the python scripts in order:

python download_data.py
python get_data_info.py
python create_triplets.py
python create_knowledge_base.py
python create_test_set.py

This should download the necessary dataset files for agent execution.

3. Return to the project root directory.

---

Agent Execution

Ensure that you have set your OPENAI_API_KEY environment variable before execution.

export OPENAI_API_KEY=<your-openai-api-key>

To execute BioVerge Agent, follow the instructions below:

1. From project root, change directory to agents:

cd agents/

2. There are separate scripts for Single Agent and Double Agent execution.

• To run Single Agent, run:

python single_agent.py

• To run Double Agent, run:

python double_agent.py

• To set a particular evaluation threshold (1 to 100), provide an argument to the script:

python <single/double>_agent.py  --evaluation_threshold <value>

3. Return to project root directory.

---

Evaluation

To evaluate the results of agent executions, follow the instructions below:

1. From project root, change directory to evaluation:

cd evaluation/

2. Manually set DIR variable to the output folder from agent execution:

DIR="./../output/test_subset/gpt-4o-mini/<agent-setting-dir>/<date>/<time>/round1/"

3. Execute the evaluation script:

python eval.py -i $DIR

Evaluation results will be stored under BioVerge/evaluation/results/.