Development Provider Framework

Development: provider framework

← Development · Providers

Infrastructure shared by every npdb convert bagel <provider> command: how a provider manager is built, what happens to downloaded files before conversion, the cache rule, credential loading, and how to add a provider. Per-provider usage lives in Providers.

• Flow
• Code map
• Unpacking
• Preparation
• Shared cache rule
• Credentials and environment
• Adding a provider

Flow

sequenceDiagram
    actor User
    participant CLI as npdb convert bagel
    participant Factory as ProviderManagerFactory
    participant Provider as Provider manager
    participant Prep as Shared preparation
    participant Pipeline as Local conversion pipeline
    participant Output as Output directory
    User->>CLI: provider + identifier + options
    CLI->>Factory: create(provider, credentials, cache)
    Factory-->>CLI: manager
    CLI->>Provider: fetch(identifier, staging path)
    Provider->>Provider: Download provider files
    Provider->>Prep: prepare_fetched(staging root)
    Prep->>Prep: unpack archives, normalize BIDS layout
    Prep-->>Provider: prepared BIDS root
    Provider-->>CLI: prepared dataset path
    CLI->>Pipeline: local2bagel(dataset, options)
    Pipeline->>Output: validation report, ledger, JSON-LD
    Output-->>User: generated dataset files

Loading

• The fetch directory is <output>/<provider>_<dataset_id>; dataset_id is derived from the identifier (/ and . replaced by _; git and MIDRC compute their own).
• fetch returns the prepared dataset path. The CLI no longer unpacks or prepares.
• Providers that collect several components (Figshare collection articles, MIDRC jobs, Mendeley files) accumulate them into one dataset root and prepare it once, after the whole loop.
• MIDRC and Mendeley downloads show a live progress display (download observers).
• After fetch, the pipeline is identical to npdb convert bagel local.

Code map

Path	Role
src/npdb/managers/model.py	ProviderName enum, ProviderManager base (fetch, unpack, prepare, prepare_fetched, describe, ensure_cache_dir), GitManager (sparse clone, auth headers).
src/npdb/managers/preparation.py	unpack_provider_archives, extract_archive, prepare_provider_dataset.
src/npdb/managers/<provider>.py	One manager per provider.
src/npdb/factories.py	ProviderManagerFactory.create (reads credentials from the environment), GiteaManagerFactory.
src/npdb/cli/cli.py	One @bagel.command per provider, all calling _provider_call.

Unpacking

ProviderManager.unpack scans the download for ZIP and TAR archives (including gzip, bzip2 and xz compressed TAR):

• Each archive is extracted into the directory containing it and deleted only after successful extraction.
• Member paths are validated; unsafe paths fail extraction.
• Ordinary files are unchanged.
• Archives found inside newly extracted content are not recursively unpacked.

Preparation

ProviderManager.prepare first selects the dataset root, then normalizes common layouts into a BIDS root:

0. If the staging root has no dataset_description.json and no sub-* entry, the shallowest nested folder that looks like a dataset (participants.tsv, rawdata/ or sub-* folders) is used instead. This handles archives wrapped in a folder such as <repo>-<commit>/ (Zenodo GitHub releases, Mendeley). Hidden folders, __MACOSX and the auxiliary folders below are ignored; several candidates raise an error.
1. If no derivatives/ exists, non-hidden top-level directories other than rawdata, sub-*, sourcedata, stimuli, code and phenotype are moved into a new derivatives/.
2. The contents of rawdata/ are moved to the dataset root and the empty rawdata/ is removed.
3. If participants.tsv is missing, it is generated with participant_id, age and sex, from the unique sub-<label> entities in raw-data file and directory names. Age and sex are N/A. Existing tables are preserved.

Providers can override unpack/prepare (MIDRC converts DICOMs and builds derivatives, see MIDRC).

Shared cache rule

Providers that download large archives or bundles use a persistent local cache:

• --cache-dir on the CLI, otherwise NP_NPDB_CACHE_DIR.
• A manager with requires_cache = True (Kaggle), or one that calls ensure_cache_dir(required=True) (MIDRC Discovery), fails with: A cache directory is required for this provider. Pass --cache-dir or set NP_NPDB_CACHE_DIR.
• This avoids silently downloading very large archives into a temporary directory that may be cleaned up, or that fills the system disk.

export NP_NPDB_CACHE_DIR="$HOME/.cache/npdb-downloads"

Credentials and environment

ProviderManagerFactory.create reads provider credentials from process environment variables (NP_GIT_*, NP_MENDELEY_ACCESS_TOKEN, NP_MIDRC_*, NP_ZENODO_TOKEN, NP_FIGSHARE_TOKEN); CLI options (--token, --credentials-path, --endpoint) take precedence.

Important

.env is loaded automatically only by npdb download and npdb convert bagel gitea. For the other providers, export it first:

set -a; source .env; set +a

Variable template: template.env.

Adding a provider

Add one provider end to end: its command appears in help without the optional library installed, fetches into the standard staging directory, returns a prepared BIDS root, and writes Neurobagel JSON-LD. Keep tests offline by mocking only the provider's network/client boundary.

1. Define the provider contract

Before coding, write down the backend contract. This determines which shared APIs fit and where provider-specific work is needed.

Question	Decide
Identifier	What exact ID, URL, DOI, or manifest does identifies the dataset?
Access	Which data is public/private? What credential is needed, and how is it supplied?
Payload	Does the source return files, archives, a manifest, or multiple parts?
BIDS readiness	Is the payload already BIDS? If not, what transformation produces a valid BIDS root?
Storage	Can a download require a persistent cache?
Provenance	What stable URL and access type should appear in the Neurobagel output?

Shared preparation handles common archives and layouts; it cannot infer subject metadata or convert arbitrary data into BIDS. If the source is not BIDS, implement and test a provider-specific transformation or clearly reject that layout.

Use existing providers as references for techniques, not templates to duplicate wholesale:

Technique	Reference
Lazy optional-client import, one download call	OpenNeuro manager
HTTP API, redirects, collection files	Figshare manager
Integrity checks, cache, authentication fallback	Mendeley manager
Authenticated manifests and specialized BIDS assembly	MIDRC manager
Git repository	Git manager

2. Register the provider name

ProviderName is the canonical internal key used by the factory and CLI. Add the backend's lowercase key there:

class ProviderName(str, Enum):
    # existing names ...
    EXAMPLE = "example"

Use that key for the CLI subcommand, manager provider_name, and dependency extra. The wiki page may use a readable display name, but follow the Development-Provider-<Name> filename convention.

3. Implement the manager

Create src/npdb/managers/example.py as a subclass of ProviderManager. Its contract is: accept an identifier and staging path, fetch source files there, and return the prepared BIDS root. The base prepare_fetched() runs unpacking and common layout preparation exactly once.

from pathlib import Path
from typing import Any, ClassVar

from npdb.extras import require_module
from npdb.managers.model import ProviderManager, ProviderName


class ExampleProviderManager(ProviderManager):
    provider_name: ClassVar[ProviderName] = ProviderName.EXAMPLE

    def __init__(self, token: str | None = None, cache_dir: str | Path | None = None):
        super().__init__(cache_dir=cache_dir)
        self.token = token

    def fetch(self, identifier: str, output_dir: str | Path, **_: Any) -> Path:
        client = require_module(
            "example_client", extra="example", feature="The Example provider"
        )
        output = Path(output_dir)
        output.mkdir(parents=True, exist_ok=True)
        client.download(
            identifier=identifier,
            output_dir=str(output),
            token=self.token,
        )
        return self.prepare_fetched(output)

• The module, token, and method names above are placeholders; match the real client.
• Keep optional imports inside fetch() using require_module() so npdb --help and other backends work in limited installs.
• Use require_executable() for required system programs.
• If every fetch requires a cache, set requires_cache = True; if only some requests do, call self.ensure_cache_dir(required=True) for those inputs.
• Use the base unpack() and prepare() unless the provider needs a documented exception.
• For collections or APIs returning several archives, download all components into one root, then call prepare_fetched() once after the loop. Do not prepare each component separately.

4. Add dependencies and configuration

Add an optional extra in pyproject.toml:

[project.optional-dependencies]
example = ["example-client>=1.0"]

all = [
  "npdb[git,gitea,kaggle,mendeley,figshare,midrc,openneuro,zenodo,example]",
]

If the backend uses only core dependencies or system programs, use example = [] so npdb[example] is valid. Update the existing all list, then regenerate the lockfile:

uv lock
uv sync --active --extra example

If the manager uses credentials, add their names and comments, not secrets, to template.env. ProviderManagerFactory.create reads environment variables and CLI values, then constructs the manager. Add a branch like this, adapting constructor fields to the manager:

if provider_name == ProviderName.EXAMPLE:
    return ExampleProviderManager(
        token=token or os.environ.get("NP_EXAMPLE_TOKEN"),
        cache_dir=cache_dir,
    )

The token value above is already a factory argument used by Zenodo and Figshare. If the new provider needs a distinct CLI option, add it to create() and _provider_call(). Keep secrets out of command echo, logs, exception text, and test output.

5. Add the CLI command

In src/npdb/cli/cli.py, add a @bagel.command("example") beside the provider commands. The OpenNeuro command shows the exact relationship: Typer arguments/options become values passed to _provider_call(), which performs shared validation, repair, annotation, ledger, and JSON-LD export.

After declaring the command's Typer arguments and options (see the complete OpenNeuro command), pass their values into _provider_call():

_provider_call(
    "example",
    dataset,
    output=output,
    online_url=f"https://example.org/datasets/{dataset}",
    access_type="public",
    mode=mode,
    phenotype_dict=phenotype_dict,
    headless=headless,
    timeout=timeout,
    artifacts_dir=artifacts_dir,
    ai_provider=ai_provider,
    ai_model=ai_model,
    header_map=header_map,
    extend_modalities=extend_modalities,
    bids_version=bids_version,
    suffix_alias=suffix_alias,
    bids_severity_threshold=bids_severity_threshold,
    cache_dir=cache_dir,
    token=token,
)

In this example, those names are the values declared by Typer in the command signature. Use the provider's canonical URL and access type, not the placeholder values above. If the factory needs a new provider-specific value, pass it through _provider_call() to ProviderManagerFactory.create() as well.

6. Test the complete path offline

Use monkeypatch or unittest.mock.patch at the network/client boundary. Never call the live provider, and never require credentials in CI. A provider is not finished when a mocked manager unit test passes: the shared acceptance test must take a tiny downloaded BIDS fixture through preparation and the real JSON-LD conversion path.

1. Manager unit tests: mock the client; assert identifier, credentials, and output path; test success and anticipated failures. For multi-file providers, assert all files share one staging root and prepare_fetched() runs once after all downloads.
2. Preparation tests: use tiny synthetic archives/layouts. Cover wrapped roots, missing participants.tsv, path collisions, unsafe archive members, and any provider-specific transformation. The shared archive rules are tested in test/test_provider_archives.py.
3. CLI conversion test: invoke npdb convert bagel example ...; mock the remote fetch and browser annotation, but assert the prepared BIDS root reaches the conversion pipeline and a JSON-LD file is written. See the synthetic end-to-end test in test/test_midrc.py.
4. Limited-install test: add the module/extra/manager case to test/test_extras.py. Confirm the package is imported lazily, missing-extra guidance names npdb[example], and npdb --help still works.
5. Provider adequacy test: add a row to the PROVIDERS table in test/test_provider_adequacy.py. Its parametrized checks cover ingestability, missing configuration, HTTP/network failures, and non-BIDS payloads. Only the fetch boundary is mocked; shared unpacking, preparation, validation, and JSON-LD conversion run for real.

Useful existing patterns: shared preparation, Figshare client mocking, Mendeley integrity/cache cases, and MIDRC download + conversion.

Put provider-specific unit cases in a new test/test_example.py. After adding the provider to the shared tests, run the focused checks first, then the offline full suite:

.venv/bin/python -m pytest -q test/test_example.py test/test_provider_adequacy.py test/test_extras.py
.venv/bin/python -m pytest -q

7. Make it discoverable

Add Development-Provider-Example.md using the same Install → Access → Download → Details structure as the other provider pages. Then add the new backend to each index:

• Development-Providers.md: comparison-table row with identifier, extra, credentials, and cache requirements.
• Development.md: provider link in its provider list.
• _Sidebar.md: link in the provider section.
• Development-CLI-Reference.md: command, argument names, and provider-specific options.

Keep README.md and user docs/ focused on common user workflows. A conversion backend alone does not add support to npdb download; that command has its own TSV-driven backend selection and requires a separate implementation and test change.