Development Provider MIDRC

Provider: MIDRC

← Providers

Imaging datasets from MIDRC, a Gen3 data commons. Portal labels and access policies below were verified on 2026-10-06 and can change.

sequenceDiagram
  actor User
  participant Portal as MIDRC portal
  participant CLI as npdb convert bagel midrc
  participant MDS as Discovery metadata API
  participant Gen3 as Gen3 auth / index / file APIs
  participant Cache as Required Discovery cache
  participant Prep as MIDRC preparation
  participant Pipeline as Shared conversion pipeline
  User->>Portal: Select cohort / files
  Portal-->>User: Discovery ID, file GUID, or manifest
  User->>CLI: Input + credentials.json + cache
  alt Discovery ID or URL
    CLI->>MDS: Resolve data_download_links
    MDS-->>CLI: File GUIDs
  else GUID or manifest
    CLI->>CLI: Parse and deduplicate GUIDs
  end
  CLI->>Gen3: Authenticate and request signed file URLs
  Gen3-->>CLI: File metadata, sizes, checksums, URLs
  loop Each file
    CLI->>Gen3: Stream file from signed URL
    Gen3-->>Cache: Verified archive / file
  end
  CLI->>Prep: Extract roles, map metadata, convert DICOM
  Prep-->>CLI: Prepared BIDS dataset + reports
  CLI->>Pipeline: Validate, annotate, export
  Pipeline-->>User: JSON-LD output

Loading

Install

uv sync --active --extra midrc

Native tools (see Environment setup):

• dcm2niix on PATH: required to convert DICOM series.
• bids-validator-rust: bash scripts/install_bids_validator.sh.

Optional, outside npdb: the official Gen3 Data Client (gen3-client) for manual downloads.

Access

MIDRC calls its data open access, but resources are open to registered users under a Data Use Agreement (DUA). Browsing works anonymously; downloading files does not.

1. Sign in at MIDRC Login (InCommon or ORCID on the verification date).

2. Complete registration and accept the DUA when prompted (DUA information).

3. Open Profile, select Create API key, then Download json to save credentials.json. Treat it as a password: never commit or share it.

4. Point npdb at it:

   export NP_MIDRC_CREDENTIALS=/absolute/path/to/credentials.json
   export NP_MIDRC_ENDPOINT=https://data.midrc.org   # default

   or pass --credentials-path (and --endpoint) on the command line.

Registration or access problems: midrc-support@gen3.org.

Download

npdb convert bagel midrc accepts three kinds of input:

Input	Example	Where to get it
Discovery dataset URL or ID	https://data.midrc.org/discovery/H6K0-A61V/, H6K0-A61V	Discovery
File GUID	dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad	Exploration > Data Files > Object Id (details)
Manifest JSON	./midrc-manifest.json	Exploration > Download file manifest (details)

npdb convert bagel midrc <input> <output> [--credentials-path <json>] [--cache-dir <cache>] [conversion options]

npdb convert bagel midrc H6K0-A61V ./out \
  --credentials-path /absolute/path/to/credentials.json \
  --cache-dir /absolute/path/to/midrc-cache
npdb convert bagel midrc 'dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad' ./out
npdb convert bagel midrc ./midrc-manifest.json ./out

• Discovery inputs require a cache: --cache-dir or NP_NPDB_CACHE_DIR. Allow space for cache, staged archives and extracted content (Duke CSpineSeg: four ZIPs, about 13.56 GB compressed).
• A Discovery URL must belong to the configured endpoint.

Manual download with gen3-client instead (documented commands, not run during verification):

gen3-client configure --profile=midrc --cred=/absolute/path/to/credentials.json --apiendpoint=https://data.midrc.org
gen3-client download-single --profile=midrc --guid=dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad --download-path=./midrc-downloads
gen3-client download-multiple --profile=midrc --manifest=./midrc-manifest.json --download-path=./midrc-downloads

Details

File GUIDs

Gen3 assigns a globally unique identifier (GUID) to each uploaded file. The same identifier appears as:

Context	Field or label
MIDRC Exploration, Data Files tab	Object Id
Gen3 file metadata / download manifest	object_id
Gen3 file index (indexd)	did
gen3-client download-single	--guid

To find one: open Exploration, apply filters, select Data Files, scroll to Object Id, and copy the complete value including dg.MD1R/. The example GUID above identifies one ZIP file (not the whole Open-R1 project); its index record is at https://data.midrc.org/index/index/dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad.

Do not use Patient ID, Case ID, Study UID, Series Uid, Submitter Id or a metadata node id: they identify other entities.

Manifests

A download manifest is a local JSON list of files generated from your portal selection.

1. Sign in, open Exploration, filter to your cohort.
2. Use Download file manifest on the Data Files tab (or Download file manifest for cases on Cases). Download table exports metadata, not a manifest.
3. If the portal says cohort too large, narrow the filters.

Accepted formats:

[{ "object_id": "dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad" }]

The standard Gen3 top-level array (portal exports may add file_name, file_size, subject_id). For backward compatibility, an object with records or files, identifiers under object_id, guid or did, is also accepted:

{ "records": [{ "did": "dg.MD1R/00001e99-cdd5-43d1-871a-09b6f9df5dad" }] }

Duplicate GUIDs are downloaded once. Empty manifests, malformed entries and missing identifiers fail explicitly. Provider code: src/npdb/managers/midrc.py.

Anonymous access checks

Observed without an account for the example GUID:

Operation	Result
Browse Exploration and Data Files	works
Read /index/index/<GUID>	returns metadata (storage s3://open-data-midrc/zip/..., authz /programs/Open/projects/R1)
Export a manifest	button reads Login to download file manifest
/user/data/download/<GUID>	HTTP 401
Unsigned HEAD on the S3 object	HTTP 403

A readable index record does not grant access to the bytes. These checks cover this route and file only.

Discovery resolution and transfers

• Discovery inputs read /mds/metadata/<dataset-id> and resolve every GUID in gen3_discovery.data_download_links (not files_count: the Duke example has files_count=1 but four links). Datasets without such links need an Exploration manifest.
• The Gen3 Python SDK obtains signed HTTPS URLs; files stream to disk. Indexed nested filenames are preserved; unsafe paths and conflicting destinations are rejected.
• Size and a supported checksum (SHA-256, SHA-1 or MD5) are checked before a file is complete. Authorization failures stop the download; a recognized expired storage URL is renewed once.
• Disk space is checked for downloads and staging before transfer.
• Validated cached files are reused on retry, with separate staged copies. Cache and fetch directories must not overlap. Interrupted .part files must be removed before retrying; there is no byte-range resume.

Preparation

MIDRC specializes the shared preparation:

1. <dataset>_structured.zip is extracted first into sourcedata/midrc/structured. Case, study, series and annotation foreign keys build participants.tsv and its sidecar.
2. <dataset>_imaging_files.zip is extracted separately. Nested series ZIPs are unpacked, DICOM study/series UIDs verified against metadata, and each mapped series converted with dcm2niix (compressed NIfTI + anonymized JSON).
3. Other archives go to isolated role directories; linked NIfTI images go under <dataset>/derivatives/<archive-suffix>/sub-*/ses-*/anat. Discrete _SEG images use desc-SEG_dseg; original bytes are preserved.

Prepared data is at <fetch-directory>/<dataset>; source archives, TSVs, intermediate conversions and reports stay outside it. Example: 593973-000001_Study-MR-1_Series-22.nii.gz becomes sub-593973000001_ses-MR1_run-22_T2w.nii.gz when series metadata establishes T2 weighting.

• Supported today: structural MR and linked NIfTI derivatives. Skull-stripping status of derivatives is taken from explicit metadata or established by voxel/affine equivalence to raw; otherwise the image is excluded.
• sourcedata/midrc/conversion_report.json and derivative_report.json list mapped outputs and quarantined sources with reasons (unknown acquisitions, unmapped series, unsupported annotations including DICOM-SEG, ambiguous converter output).
• Conversion errors, unsafe extraction, conflicting metadata and output collisions fail explicitly. Empty imaging conversion is not success.
• Completed extraction and preparation are reused on retry when the archive manifest matches and outputs are intact, so annotation can be retried without re-converting. Use a fresh staging destination after an interrupted preparation.
• Source layouts that don't fit fail explicitly: prepare a local dataset instead and use local ingestion.

Offline preparation, without credentials or network:

from npdb.managers.midrc import MIDRCProviderManager

provider = MIDRCProviderManager()
staged = provider.unpack("/path/to/local/archive-directory", output_dir="/path/to/new/staging")
bids = provider.prepare(staged)

References

• MIDRC overview, login, Exploration, DUA
• Gen3 portal guide, Gen3 client guide, Gen3 access