Ingest ENCODE annotation files during sync

[conradbzura]

Description

fetch_encode_metadata requests one URL:

metadata_url = f"{api_base}/metadata/?type=Experiment&status=released"

Everything ENCODE publishes as an Annotation — candidate cis-regulatory elements, element gene regulatory interaction predictions, chromatin state, footprints, imputations — is absent from cfdb. Add a second ingest path for type=Annotation&status=released and map its rows onto the C2M2 model in transform_to_c2m2.

The two TSVs are not the same shape. Experiment carries 59 columns, Annotation 32, and the overlap is partial:

Renamed	Experiment accession → Dataset accession, Assay → Assay term name, File assembly → Assembly, Experiment date released → Dataset date released, s3_uri → S3 URL, Biosample organism → Organism
Annotation-only, no home in the current model	Annotation type, Software used, Encyclopedia Version, Targets, Life stage, Age, Age units
Experiment-only, absent from Annotation	All eight Library * fields, all six Biosample genetic modifications *, Biosample treatments *, Biological/Technical replicate(s), Donor(s), Experiment target, File analysis title/status, File format type, Genome annotation, Index of

The renames are mechanical. The second row is the actual design work: Annotation type is the field that gives an annotation file its meaning, and there is nowhere to put it. EnrichedEncodeFile and EnrichedEncodeCollection need new fields, and Life stage/Age/Age units map onto Subject (age_at_sampling) rather than onto the file. The third row means the annotation path must leave those enrichment fields unset rather than deriving them — _extract_donor_ids has no input at all, since Donor(s) does not exist in the annotation TSV.

Motivation

Annotation files are the interpretive layer over the raw experiments cfdb already serves. A cCRE track or an element-gene interaction set is what a Gosling visualization usually wants alongside signal, and today none of it is reachable through cfdb.

Scope — start with two annotation types

Confirmed starting set: 7,773 datasets.

annotation_type	Datasets	Formats	Assemblies
candidate Cis-Regulatory Elements	6,230	bed bed9+, bigBed bed9+, bed bed3+, bigBed bed3+	GRCh38 (4,046), mm10 (1,111), hg19 (1,073)
element gene regulatory interaction predictions	1,543	bed bed3+, bedpe, bigInteract, bed bed3	GRCh38 only

Against 27,043 experiments and a 295,697-file corpus, 7,773 datasets is proportionate. The full annotation space is 580,910 datasets, 86% of it footprints (499,790), so annotation_type must be an allowlist from the outset rather than a filter retrofitted once volume becomes a problem. Chromatin state (1,815) is the natural third.

Two differences between the chosen types matter for implementation.

cCREs are format-clean. Every file is BED or bigBed, both already routed correctly by the tabix processor. Nothing new is needed to serve them.

cCREs are multi-organism and multi-assembly. Mus musculus (1,111) and hg19 (1,073) both appear, where the experiment path has so far assumed a single assembly per query. Organism is one of the annotation-only columns with no current home; Assembly maps onto the existing EnrichedEncodeFile.assembly. A client asking for cCREs without qualifying assembly will get GRCh38, mm10, and hg19 in one result set, so both fields should be queryable rather than left to filename inspection.

A format gap ships with the interaction predictions

Two of that type's four formats have no real processor:

# src/cfdb/services/ontology_mappings.py
"bedpe":       {"id": "format:3003", "name": "BED"},      # BED paired-end
"biginteract": {"id": "format:3004", "name": "bigBed"},   # bigInteract is a bigBed variant

# src/cfdb/workflows/processors/tabix.py
_TABIX_PRESET = {"BED": "bed", ..., "bigBed": "bed"}
_SORT_ARGS    = {"bed": ["-k1,1", "-k2,2n"]}

A bedpe file therefore enters the BED pipeline and gets sort -k1,1 -k2,2n then tabix -p bed. BEDPE columns are chrom1/start1/end1/chrom2/start2/end2, so the result is an index over the first mate with the second silently unindexed — a cached artifact that looks successful and is wrong. The byte-sniff guard added in #71 does not catch this: it rejects non-gzip binary magics (starch's ca5cade5 and similar), and a bedpe file is plaintext or gzip, so it passes.

bigInteract is milder. bigBedToBed extracts it and the leading three columns are the interaction's bounding span, so a bed-preset index over them is at least range-coherent. What is lost is the structure: source and target endpoints collapse into opaque trailing columns, and the file is served as intervals rather than as interactions.

This is the same shape as the .starch misrouting — a format aliased to BED in the ontology mapping, then mishandled by the BED processor — and #72 is the template for fixing it. That issue stops conflating starch with BED by minting its own file_format CV entry, adds a converter branch to _stage_prepare, installs the tool in Dockerfile.wool, and lists the format in _SELF_VALIDATING_FORMATS. A bedpe fix follows the same four steps, differing only in the converter and in whether tabix -p bed is the right preset at all. Whoever picks up either should read both.

Note the bedpe problem applies to any .bedpe already in the corpus, independent of this feature.

Pick one before implementing: ingest only the bed bed3/bed bed3+ files of this type and exclude bedpe/bigInteract until a processor exists, or treat the processor as part of this work. The first gets interaction data in front of users sooner; the second makes it usable as interactions rather than as intervals.

Suggested sequencing

cCREs first. They exercise the whole annotation ingest path — new TSV shape, new columns, Annotation type queryability, multi-organism, multi-assembly — without also requiring a format decision. The interaction predictions then land on proven ingest code with only the format question outstanding.

Expected Outcome

• Sync fetches both type=Experiment and type=Annotation, and a failure in one does not abort the other.
• Annotation rows produce FileMetadataModel documents whose dcc, file_format, data_type, access_url, md5, and size_in_bytes are populated on the same code paths experiments use.
• Annotation type is queryable — a client can ask for cCRE files without string-matching filenames.
• Organism and Assembly are queryable, so a multi-assembly result set can be narrowed.
• Experiment-only enrichment fields are None on annotation documents rather than fabricated.
• Which annotation_type values are ingested is configuration, not a hardcoded assumption.
• The field mapping is documented in encode.py's module docstring in the same form the experiment mapping already uses — that docstring is the reference for this service and would otherwise describe half of it.

Open questions

• Does an annotation dataset become a Collection the way an experiment does, given Dataset accession has no replicate structure?
• Targets is a list where Experiment target was a scalar. Reuse EnrichedEncodeCollection.experiment_target, or add a field?
• Does the mm10 subset need anything the human path does not, or is organism purely a metadata field here?