Development Download Internals

Development: download internals

← Development

How npdb download turns a Neurobagel query-results TSV into files on disk. User instructions: docs/download-data.md.

sequenceDiagram
	actor User
	participant CLI as npdb download
	participant TSV as Query-results TSV
	participant HTTP as HTTP server
	participant Git as NeuroData / Git
	participant Annex as git-annex
	participant Disk as Output folder
	User->>CLI: query.tsv + options
	CLI->>TSV: Read and group rows
	loop Each dataset with AccessLink
		CLI->>HTTP: Download each unique URL (parallel)
		HTTP-->>Disk: File under dataset/subject
	end
	loop Each dataset without HTTP links
		CLI->>Git: Sparse checkout selected paths
		Git-->>Disk: Repository metadata and selected files
		opt git-annex branch exists
			CLI->>Annex: Retrieve annexed content
			Annex-->>Disk: Imaging file contents
		end
	end
	CLI-->>User: Summary and exit status

Loading

• Inputs
• Backend selection
• Credentials and SSL
• Options
• Examples

Inputs

The TSV exported from the Neurobagel query tool has one row per subject and session. npdb download uses:

Column	Used for
DatasetName	Output sub-folder and grouping.
SubjectID	Output sub-folder (HTTP) and sparse checkout.
AccessLink	Direct HTTP(S) download when it is a valid URL.
RepositoryURL	Git clone URL when no HTTP link exists for the dataset.
ImagingSessionPath	Path of the session inside the repository, used for sparse checkout.

Backend selection

The backend is chosen per dataset, so one TSV can mix backends:

1. HTTP. If AccessLink holds a valid HTTP(S) URL, the dataset is downloaded with httpx. Each unique URL is fetched once into <output>/<DatasetName>/<SubjectID>/<filename>, in parallel (--max-workers, default 4).
2. Git. Otherwise, the repository at RepositoryURL is cloned with sparse checkout, limited to the ImagingSessionPath of the selected rows. Derivatives linked to those subjects are included by default (--no-derivatives to skip).
3. Git-annex. If the repository has a git-annex branch, annexed file content is retrieved after the git checkout (requires git-annex on PATH).

Datasets with neither a valid AccessLink nor RepositoryURL + ImagingSessionPath are skipped.

Credentials and SSL

Git and git-annex downloads go through the NeuroData manager and need, in .env (current folder first, then the repository root) or the environment:

NP_GITEA_APP_URL=https://data.neuro.polymtl.ca
NP_GITEA_APP_USER=<username>
NP_GITEA_APP_TOKEN=<token>

HTTP downloads need no credentials.

--no-verify-ssl is currently required for NeuroData because the server's certificate chain is not trusted by default clients. The risk is mitigated by the mandatory VPN. NP_GITEA_APP_VERIFY_SSL in template.env is not read by the CLI; only the flag counts. Remove this requirement from the user docs once the chain of trust is fixed.

Options

Option	Default	Effect
--output-dir, -o	current directory	Where files are written.
--derivatives / --no-derivatives	derivatives	Include derivatives (git mode).
--max-workers	4	Parallel HTTP downloads.
--verify-ssl / --no-verify-ssl	verify	SSL verification towards NeuroData (git mode).
--verbose, -v	off	Print each git command before it runs.

Examples

# Typical user run
npdb download --no-verify-ssl results.tsv -o ./my-cohort

# Raw data only, show git commands
npdb download --no-verify-ssl --no-derivatives -v results.tsv -o ./my-cohort

# HTTP-only TSV, more parallelism
npdb download --max-workers 8 results.tsv -o ./downloads

Tests for this command live in test/test_download.py and mock the NeuroData manager; they must stay offline.