Development Environment Setup

Development: environment setup

← Development

This page sets up a full development environment: every optional dependency, browser automation, native validation and conversion tools, and the test suite.

flowchart TD
   START[Clone repository] --> CHOICE{Choose setup}
   CHOICE -- VS Code --> CONTAINER[Dev container]
   CHOICE -- local --> VENV[Python + uv environment]
   CONTAINER --> EXTRAS[Provider extras + dev tools]
   VENV --> EXTRAS
   EXTRAS --> BROWSER[Playwright + Chromium]
   EXTRAS --> NATIVE[Native tools: git, validator, dcm2niix]
   BROWSER --> TEST[Offline test suite]
   NATIVE --> TEST

Loading

• Option A: VS Code dev container (recommended)
• Option B: local installation
• Dependency extras and groups
• Native tools
• Environment variables
• Running the tests

Option A: VS Code dev container (recommended)

1. Install Docker (see Neurobagel deployment for versions) and VS Code with the Dev Containers extension.

2. Clone the repository with its submodules:

   git clone https://github.com/neuropoly/neuropoly-db.git
   cd neuropoly-db
   git submodule update --init --recursive

   Without access to the private seed datasets repository, initialize only the recipes:

   git submodule update --init --recursive neurobagel-recipes

3. Open the folder in VS Code and run Dev Containers: Reopen in Container.

The container (.devcontainer/Dockerfile) builds the Rust BIDS validator with Rust 1.90.0 and installs dcm2niix. On creation, .devcontainer/postCreate.sh:

• creates .venv and runs uv sync --active --dev --all-extras (re-run automatically when pyproject.toml or uv.lock change),
• installs Playwright Chromium with its system dependencies,
• registers a Python (neuropoly-db) Jupyter kernel,
• links the native validator into .venv/bin.

Note

The Docker daemon runs outside the dev container. Paths inside /workspaces/... are not valid host bind-mount paths. For smoke tests in other containers, docker cp files in instead of mounting them.

Option B: local installation

Prerequisites: Python 3.12+, uv, Rust 1.85+ with Cargo, git, git-annex.

git clone https://github.com/neuropoly/neuropoly-db.git
cd neuropoly-db
git submodule update --init --recursive neurobagel-recipes

uv venv .venv
source .venv/bin/activate
uv sync --active --dev --all-extras

# Browser automation for --mode assist|auto|full-auto
uv run playwright install --with-deps chromium

If uv venv asks whether to replace an existing .venv, answer yes. If it still fails, delete the content of .venv and run it again.

Then install the native tools.

Dependency extras and groups

Every provider backend has its own extra; development tools are a dependency group. The core install has no provider libraries and every command still loads: a provider whose extra is missing stops with ... requires the '<extra>' extra. Install it with: pip install 'npdb[<extra>]'. Missing system tools (git, git-annex, dcm2niix, bids-validator-rust) are reported the same way.

Extra	Python packages	Used by
gitea	py-gitea	npdb download, convert bagel gitea
git	none (system git)	convert bagel git
kaggle	kagglehub	convert bagel kaggle
mendeley	none (core httpx)	convert bagel mendeley
figshare	none (core httpx)	convert bagel figshare
midrc	gen3	convert bagel midrc
openneuro	openneuro-py	convert bagel openneuro
zenodo	zenodo_get	convert bagel zenodo
download	= gitea	npdb download only
all	every provider above	recommended install
annotation-automation	playwright	--mode assist|auto|full-auto (then playwright install --with-deps chromium)
full	all + annotation-automation	curators

uv sync --active --extra midrc              # one provider
uv sync --active --extra all                # every provider
uv sync --active --dev --all-extras         # development: everything + test tools
pip install 'npdb[midrc] @ git+https://github.com/neuropoly/neuropoly-db'   # pip, single provider

Note

A bare npdb cannot install every provider by default while npdb[midrc] installs only MIDRC: that needs PEP 771 default extras, which pip and uv don't support yet. Until then, docs install npdb[all]. When support lands, add default-optional-dependency-keys = ["all"] to [project] in pyproject.toml.

End users install with a single command, without cloning:

uv tool install "npdb[all] @ git+https://github.com/neuropoly/neuropoly-db"

To test that path, or a limited install, from a checkout:

T=$(mktemp -d)
UV_TOOL_DIR=$T/tools UV_TOOL_BIN_DIR=$T/bin uv tool install "npdb[midrc] @ $PWD"
$T/bin/npdb convert bagel kaggle u/d ./out --cache-dir $T/c   # -> requires the 'kaggle' extra
rm -rf "$T"

Native tools

npdb uses dcm2niix to convert MIDRC DICOM series and the Rust-only bids-validator-rust executable to validate datasets before Neurobagel imaging table generation. Missing tools cause actionable errors; no JavaScript, Deno, or Python dataset-validator fallback is invoked. npdb download needs neither tool.

Installation

# Debian/Ubuntu (run system package installation yourself)
sudo apt-get update
sudo apt-get install -y dcm2niix build-essential

# macOS alternative
brew install dcm2niix

Build the pinned validator from the repository root:

bash scripts/install_bids_validator.sh
.venv/bin/bids-validator-rust --version
dcm2niix --version

The script defaults to $VIRTUAL_ENV or the project's .venv. Pass a different installation prefix as its first argument if needed. It reuses a matching installed binary or builds with cargo install --locked; Rust dependencies are pinned by Cargo.toml and Cargo.lock. uv sync and requirements.txt install Python packages, not these native executables.

The Docker build context of the dev container contains only the validator source and build files; credentials, environment files, dataset caches and extracted images are excluded. Neurobagel API, database and UI services do not use these tools.

Validation

.venv/bin/bids-validator-rust /path/to/bids > validation.json

The runner checks the raw root and each immediate derivative dataset. It emits JSON and exits nonzero for errors. npdb convert bagel uses this same executable before Neurobagel's imaging-table export, replacing its PyBIDS validation gate; the upstream table-generation and indexing libraries remain in use.

The pinned bids-validate 0.0.3 library is a lightweight structural validator. It checks dataset descriptions, subject-file naming, and derivative provenance; it does not establish full sidecar or NIfTI semantic compliance. A zero-error report must not be described as full BIDS semantic certification. Warnings are retained in the report. The Rust runner is separately licensed GPL-3.0-or-later.

Environment variables

Copy the template and fill in only what you need:

cp template.env .env

npdb download and npdb convert bagel gitea load .env from the current working directory (searching parent directories), then from the repository root as a fallback. Variables already set in the shell are never overridden. Other provider commands read the environment only: export first with set -a; source .env; set +a.

Variables	Used by
NP_GITEA_APP_URL, NP_GITEA_APP_USER, NP_GITEA_APP_TOKEN	npdb download, npdb convert bagel gitea
NP_NPDB_CACHE_DIR	Kaggle, MIDRC Discovery, optional Mendeley and Zenodo cache location
NP_GIT_USER, NP_GIT_TOKEN	npdb convert bagel git for private HTTP repositories
NP_KAGGLE_*, NP_OPENNEURO_TOKEN, NP_ZENODO_TOKEN, NP_MENDELEY_*, NP_MIDRC_*, NP_FIGSHARE_TOKEN	the matching provider, see Providers
NB_*, COMPOSE_*, LOCAL_GRAPH_DATA	the development Neurobagel node, see Neurobagel deployment

To get NeuroData credentials, follow the user guide docs/get-access.md.

Running the tests

Always use the project environment (system pytest lacks the dependencies):

.venv/bin/python -m pytest -q
.venv/bin/python -m pytest -q test/test_download.py
.venv/bin/python -m pytest -q -k midrc

The test suite runs offline. Do not add tests that contact NeuroData or other remote services; mock the managers instead (see test/test_download.py for examples patching npdb.cli.cli.load_dotenv and the managers).