Development Provider Mendeley

Provider: Mendeley Data

← Providers

Datasets from Mendeley Data. Core dependencies only, nothing to install (npdb[mendeley] is accepted and adds nothing).

sequenceDiagram
  actor User
  participant CLI as npdb convert bagel mendeley
  participant Public as Public Mendeley API
  participant Private as Authenticated API
  participant Cache as Persistent verified cache
  participant Stage as Output staging
  participant Pipeline as Conversion pipeline
  User->>CLI: Dataset ID / DOI + optional token
  CLI->>Public: Fetch metadata
  alt Public endpoint denies access and token supplied
    CLI->>Private: Retry with bearer token
    Private-->>CLI: Metadata and file URLs
  else Public dataset
    Public-->>CLI: Metadata and file URLs
  end
  loop Each file
    CLI->>Cache: Verify matching size and SHA-256
    alt Verified hit
      Cache-->>Stage: Atomic staged copy
    else Miss or invalid entry
      CLI->>Public: Stream file / follow redirect
      Public-->>Cache: Verify and publish download
      Cache-->>Stage: Stage verified copy
    end
  end
  CLI->>Pipeline: Extract, select BIDS root, convert
  Pipeline-->>User: JSON-LD output

Loading

Access

• Public datasets: nothing to do (no authentication is sent, even if a token is set).

• Restricted data: an OAuth access token, passed with --token or exported:

  export NP_MENDELEY_ACCESS_TOKEN=<token>

  Client ID/secret alone are not enough: npdb does not exchange app credentials for a token.

Download

npdb convert bagel mendeley <dataset_id_or_doi> <output> [--cache-dir <cache>] [conversion options]

npdb convert bagel mendeley yd695ynbdz ./out --mode auto --cache-dir "$HOME/.cache/npdb-downloads"
npdb convert bagel mendeley yd695ynbdz ./out --mode full-auto --headless --bids-version latest

Files are staged under <output>/mendeley_<dataset_id>. Verified downloads are cached automatically, including when no cache option is supplied. Cache location precedence is --cache-dir, then NP_NPDB_CACHE_DIR, then $XDG_CACHE_HOME/npdb/downloads (or ~/.cache/npdb/downloads when XDG_CACHE_HOME is unset).

Details

Authorization

• Public metadata comes from https://data.mendeley.com/public-api/datasets/<dataset_id>. On HTTP 401, 403 or 404 with a token available, the authenticated API is tried.
• HTTP 401/403 from metadata or downloads produce an explicit authorization error; other HTTP failures are preserved.

Verified downloads

• Redirects to storage are followed and files stream to disk.
• Advertised size and SHA-256 are checked when present; a file is installed only after successful verification.
• Failed downloads remove their partial file and leave any previous file unchanged.
• Missing download URLs and duplicate or unsafe filenames stop the fetch.

Cache

• Layout: <cache-dir>/mendeley/<identity-hash>/<filename>. The identity covers dataset/version, file/content identifiers, size and SHA-256, not the storage URL. Metadata (and authorization) is fetched on every run.
• Entries are reused only when size and SHA-256 match current metadata. Invalid entries are replaced only after a successful verified download or copy from an existing staged file verified against the same metadata. Without size or SHA-256 metadata, the file bypasses the cache (the progress display says so).
• Partial downloads are never published. An abruptly killed process may leave a hidden .part file; it is never reused and can be deleted when no download runs.
• Verified archives are copied atomically into staging and checked before extraction. Extraction deletes only the staged copy, so the cache survives a later conversion failure.
• If an earlier run left a complete archive in staging, it is verified and copied into the cache without downloading it again. Archives deleted by earlier uncached runs cannot be recovered from the extracted dataset; convert the existing local dataset to avoid fetching it again.
• Cache and staging must not overlap or contain symlink destinations. Free-space checks cover advertised download and staging sizes (combined on one filesystem), not extracted size.
• No resume, no automatic eviction. Remove old entries under <cache-dir>/mendeley/ manually when no fetch is running. Protect the cache like the data.

To retry conversion without fetching an already extracted dataset:

npdb convert bagel local ./out/mendeley_yd695ynbdz/Spinal_fMRI_UpperLimb \
  https://data.mendeley.com/datasets/yd695ynbdz ./out \
  --access-type public --mode full-auto --headless --bids-version latest

Dataset root

After extraction, a single wrapped BIDS dataset directory is selected (ignoring __MACOSX, hidden paths, derivatives and BIDS auxiliary directories) and passed to conversion. Multiple dataset roots fail explicitly.

Phenotype conversion

Conversion removes surplus empty TSV cells beyond the header width before parsing, and trims leading/trailing whitespace from headers and cells (for example, a padded sub-06 would otherwise not match the sub-06 imaging subject). Non-empty extra cells stop conversion with the row number rather than silently discarding data. Categorical annotation generation and preprocessing retain both BIDS Levels descriptions and Neurobagel semantic level mappings. Unexpected Bagel exceptions are included in the reported failure, not just preceding warnings.