NEUROPOLY DATABASE EXPLORATION AND STANDARDIZATION TOOLS

This repository hosts a collection of tools to interact with metadata contained in the several NEUROPOLY databases.

• Streamlined setup of NeuroBagel nodes
• Parsing and standardization of BIDS datasets
• Automatic conversion of BIDS datasets to NeuroBagel
• Automated download from NeuroBagel queries
  • Support http(s), git and git-annex protocols

Contents

• npdb command line tool
  • Prerequisites
  • Installation
  • Usage guides
  • Commands
  • Developer guide
    • Developer installation
    • Components

npdb command line tool

The NeuroPoly Database Browser is a python command line tool that simplifies interaction with the many databases and hosting technologies (NeuroGitea, NeuroBagel, etc.) used at NeuroPoly and their associated data standards (DICOM, Nifti, BIDS, etc.). It offers, among others, the following functionalities :

• Standardization of BIDS datasets to a common NeuroPoly vocabulary and structure.
• Download of datasets from NeuroGitea using NeuroBagel queries.
• Conversion of local BIDS datasets — or datasets fetched from NeuroGitea/Forgejo — to NeuroBagel format for ingestion in a NeuroBagel graph database.

All npdb commands are interactive by default and require user input to proceed. However, most of them also offer assisted and automated modes to reduce (even replace) user interaction and speed up the process. Refer to the commands descriptions below for more details.

Important

New users are strongly encouraged to read the usage guides before using the CLI.

Prerequisites

• Install Python 3.12+
• Install uv

Installation

Important

To use the download functionalities, you'll need to query from NeuroBagel (unless you already have the query results you need). Until an official NeuroBagel node is deployed at NeuroPoly, you need to install a local NeuroBagel node to query datasets from.

Follow the steps in this documentation to install and furnish a local NeuroBagel node.

0. If not done already, clone or download this repository to your local machine. Then, open a terminal and navigate to its root.

1. Create a new virtual environment locally to host the CLI dependencies and libraries :

   uv venv .venv

   Answer yes if you see :

   A virtual environment already exists at .venv. Do you want to replace it?

   The above command might fail if some virtual environment has already been configured in the provided directory (.venv). If you experience issues, delete the content under the virtual environment's directory and re-run the command.

2. Synchronize the virtual environment with the CLI dependencies :

   uv sync --active

3. (Optional) If you intend on using the assisted or automated modes for BIDS standardization and conversion to NeuroBagel (see commands below), you need to install additional dependencies. Run the following commands to install them :

   uv sync --active --quiet --extra annotation-automation
   uv run playwright install --with-deps chromium

Usage guides

This guide explains how to :

• query datasets using the NeuroBagel web interface,
• save the query results to file and interpret them,
• download the query results from NeuroGitea using npdb

This guide explains how to :

• convert a local BIDS dataset with npdb convert bagel local when the dataset is already on disk,
• convert a NeuroGitea/Forgejo-hosted dataset with npdb convert bagel gitea when the dataset must be fetched from a forge first,
• continue to the annotation and standardization modes used by both workflows.

Commands

npdb standardize bids [options] <bids_dir>

🢖 Standardization options and customization

npdb download [options] <query-results.tsv>

🢖 Guide: download from NeuroBagel queries

npdb convert bagel local [options] <input_dir> <online_url> <output>

🢖 Guide: ingest a local dataset into NeuroBagel

npdb convert bagel gitea [options] <dataset> <output>

🢖 Guide: ingest a NeuroGitea/Forgejo dataset into NeuroBagel

Which command should I use?

• Use npdb convert bagel local when the dataset is already available locally.
• Use npdb convert bagel gitea when the dataset must be resolved from NeuroGitea/Forgejo.

🢖 Annotation and standardization modes

Developer guide

Developer installation

First, run the installation procedure above. Then, install the full development environment using :

uv sync --active --quiet --all-extras

Components

• Database exploration

  Complete and structured deployment of a local NeuroBagel node, extended with NeuroPoly-specific imaging modality vocabulary :

  • NeuroBagel deployment
  • NeuroBagel extensions
  • NeuroBagel management

• Database ingestion

  A set of command line tools (under npdb) to ingest local or NeuroGitea/Forgejo-hosted BIDS datasets into a local NeuroBagel node:

  • Local and NeuroGitea/Forgejo database ingestion

• Metadata standardization

  A set of command line tools (under npdb standardize) to manipulate common standards (e.g. BIDS, Bagel).

  • BIDS datasets standardization