Development

t to

Development

This section is for people who develop npdb, run a Neurobagel node, or curate datasets for NeuroPoly. If you only want to find and download data, read the user guide in the repository README instead.

Architecture at a glance

flowchart LR
    subgraph Sources
        NG[NeuroData<br/>data.neuro.polymtl.ca]
        LOC[Local BIDS dataset]
        EXT[External providers<br/>OpenNeuro, Zenodo, Kaggle,<br/>Mendeley, Figshare, MIDRC, git]
    end
    subgraph npdb
        FETCH[Provider managers<br/>fetch / unpack / prepare]
        REPAIR[BIDS validation<br/>and repair]
        STD[Participants<br/>standardization]
        CONV[Neurobagel annotation<br/>and JSON-LD export]
        DL[npdb download]
    end
    subgraph Neurobagel node
        SEED[seed-datasets<br/>JSON-LD files]
        GRAPH[GraphDB + API]
        QUERY[Query tool<br/>data.neuro.polymtl.ca/bagel]
    end
    NG --> FETCH
    LOC --> FETCH
    EXT --> FETCH
    FETCH --> REPAIR --> STD --> CONV --> SEED --> GRAPH --> QUERY
    QUERY -- query-results TSV --> DL
    DL -- git / git-annex / HTTP --> NG

Loading

Repository layout

Path	Content
src/npdb/cli/	Typer command line (npdb), display helpers.
src/npdb/managers/	Provider managers (NeuroData, git, Kaggle, Mendeley, MIDRC, OpenNeuro, Zenodo, Figshare).
src/npdb/factories.py	Builds managers and AI clients from environment variables.
src/npdb/bids_repair.py	BIDS validation and repair workflow.
src/npdb/annotation/, src/npdb/automation/	Annotation modes and Playwright browser automation.
src/npdb/external/	Wrappers around external tools (NeuroData client, Neurobagelbagel, Rust BIDS validator, DataLad).
src/npdb/report/	Run ledger, progress observers and provenance files.
src/npdb/resources/	Default phenotype dictionary (phenotype_mappings.json).
config/	NeuroPoly vocabularies (imaging modalities, categorical terms).
scripts/	Neurobagel API patches and native tool installer.
tools/bids-validator-rust/	Pinned Rust BIDS validator runner.
docker-compose.yml, neurobagel-recipes/	Development Neurobagel node.
seed-datasets/	JSON-LD files served by the NeuroPoly Neurobagel node (private submodule).
docs/	User documentation only. Developer content lives in this wiki.

Pages

Getting started

• Environment setup: clone, dev container, dependencies, native tools, tests.
• CLI reference: every npdb command and option, with examples.

Infrastructure

Neurobagel node:

• Neurobagel deployment: run a development node with Docker Compose.
• Neurobagel node management: update datasets, hot-reload, NeuroPoly vocabulary extensions.

Curation pipeline:

• Ingestion into Neurobagel: npdb convert bagel local|gitea, annotation modes, imaging modality extensions.
• Participants standardization: npdb standardize bids, phenotype dictionaries, header maps.
• BIDS validation and repair: npdb repair bids|recover, journals and manifests.
• Provider framework: provider managers, unpacking, preparation, cache rule, adding a provider.
• Download internals: how npdb download picks HTTP, git or git-annex.

Providers

Providers overview: comparison table and common setup.

• NeuroData
• Git
• Kaggle
• Mendeley Data
• MIDRC
• OpenNeuro
• Zenodo
• Figshare

Naming

• NeuroData is NeuroPoly's data server at https://data.neuro.polymtl.ca. Use this name whenever the text is about that instance.
• Forgejo is the software NeuroData runs on. Use it only for statements that hold for any Forgejo (or Gitea) server.

Rules for contributors and coding agents

• Run tests offline with the project environment: .venv/bin/python -m pytest -q.
• Coding agents working in this repository must not run commands that contact NeuroData (see .github/copilot-instructions.md).
• Keep docs/ for end users: short, plain language, no deployment or internals. Put everything else here.