Add HiGlass-compatible tileset endpoints for Hi-C contact maps

[conradbzura]

Description

Add two endpoints that serve Hi-C contact maps as HiGlass tiles, so Gosling can render them as matrix tracks:

• GET /tileset_info/{dcc}/{local_id} — tileset metadata. Cheap and side-effect-free, mirroring the existing /data/{dcc}/{local_id}/status probes.
• GET /tiles/{dcc}/{local_id} — individual tiles.

Implement the HiGlass tile contract directly in FastAPI, using clodius and hicstraw as libraries. Do not deploy higlass-server itself: it is a Django application with its own database and tileset registry, which would duplicate what the files collection already is.

Format handling splits by source:

• mcool (773 files, all 4DN) — clodius.tiles.cooler handles both tileset_info and tiles directly.
• hic (3,581 files, all ENCODE) — no clodius hic tile module exists (all 34 modules in clodius.tiles were checked), so use hicstraw.

clodius.tiles.cooler.tileset_info supplies resolutions, max_pos, min_pos, and chromsizes. The endpoint must add the fields HiGlass requires but clodius does not emit: datatype: "matrix", name, and coordSystem (derived from genomeAssembly).

Motivation

Gosling's MatrixData type compiles to HiGlass's native heatmap track, not to gosling-track — see gosling-to-higlass.ts:182:

if (IsHiGlassMatrix(firstResolvedSpec)) {
    // By changing the track type, HiGlass uses its native heatmap track
    hgTrack.type = 'heatmap';

The accompanying comment explains why: HiGlass is better optimized for matrix rendering, and doing it in Gosling makes zooming slow. The consequence is that MatrixData.url must point at a live HiGlass tile API, not at a file. Neither /data nor /index can satisfy a matrix track, so contact maps are currently unreachable from Gosling despite 4,364 of them being indexed:

Extension	Count	DCC
.hic	3,581	ENCODE
.mcool	773	4DN
.cool	10	4DN

This endpoint pair is architecturally distinct from /data and /index. Those produce one artifact per source file and stream bytes; a tileset endpoint answers thousands of small, latency-sensitive requests per file. The 202-plus-poll workflow pattern does not apply — a heatmap track cannot be handed a job ID.

Notably, no pre-aggregation is required. Both formats are already multi-resolution pyramids built at creation time. Measured on real files: a 4DN mcool carries 13 resolutions (1kb–10Mb) and an ENCODE hic carries 17 (10bp–2.5Mb). clodius reads a tile from a local mcool in 16ms with no ingest step.

Expected Outcome

• GET /tileset_info/{dcc}/{local_id} returns a HiGlass-compatible tileset info document for .hic, .mcool, and .cool files, and mirrors the error codes of the existing /status probes for invalid DCC, missing file, and access-controlled files.
• GET /tiles/{dcc}/{local_id} returns tile data for the same set.
• A Gosling spec with data: { type: "matrix", url: "<cfdb>/tileset_info/..." } renders a contact map.
• Contact maps are routable. This is blocked today: services/ontology_mappings.py maps hic, cool, mcool, and h5ad all to file_format.name == "HDF5", and ProcessorRegistry.lookup_for keys on format name alone, so it cannot distinguish a contact map from an AnnData file. Resolve with filename-extension discrimination or a new registry predicate.
• Tile latency is acceptable for interactive panning, via a tile cache.
• Limitations are documented: interchromosomal tile handling, and that 21 of the 73 4DN hdf5 files carry no genomeAssembly, leaving coordSystem underivable for those.

Open design decision

Whether to read .hic in place via HTTP range requests, or convert once to .mcool with hic2cool to get a single code path.

Measured evidence favours reading in place. ENCODE .hic files run 6–51 GB, so whole-file caching is infeasible; but hicstraw opened a 6.2 GB .hic over HTTP range requests through cfdb's own /data endpoint in 0.88s and read its full header. Tile reads measured 2–8s from a laptop over the public internet — expected to be substantially faster from inside AWS, and tiles are small and immutable, so an LRU tile cache should absorb the remainder.

Conversion is the only part of this work that would warrant the wool worker pool, and it is expensive on a 51 GB input. Recommend starting with in-place range reads plus a tile cache, and revisiting conversion only if latency proves unacceptable from inside AWS.