Sharded dataset generation

[PatrickRMiles]

Summary

Adds v3 physical sharded dataset generation and loading, with shard files written to match the configured DistConv layout. This lets large samples be generated directly as per-rank physical shards instead of always writing full-volume samples, and should allow us to generate scale 11 data.

Changes

• Generate one set of voxelized fractal indices per logical volume, then render all requested physical shards from that shared data.
• Write v3 dataset metadata containing the physical shard layout and shard-suffixed volume/mask files.
• Load v3 physical shard files directly when the dataset layout matches the requested DistConv layout.
• Preserve v2/full-volume dataset loading for unsharded cache reuse and runtime slicing.
• Canonicalize v3 cache IDs for equivalent shard layouts so reordered dc_num_shards / dc_shard_dims pairs reuse the same dataset cache.

Validation

• Ran sharded dataset equivalence comparisons across problem scales 6, 7, and 8 for layouts 1x1x1, 1x1x2, 1x2x2, and 1x1x4; all 18 pairwise comparisons passed.

[michaelmckinsey1]

I don't understand when unsharded "V2" datasets would be used? When would those be used? When under scale 11?

In general I think we should discuss the use of keeping around V1 and V2 support. Is it even feasible to fully migrate to V3 and deprecate?

[PatrickRMiles]

I don't understand when unsharded "V2" datasets would be used? When would those be used? When under scale 11?

In general I think we should discuss the use of keeping around V1 and V2 support. Is it even feasible to fully migrate to V3 and deprecate?

@michaelmckinsey1 I figured leaving V2 support in would be good so we can reuse existing datasets. It does add complexity though, so I'm happy to remove it if you're cool with regenerating datasets for existing problem configs. I suppose since we already have good data up to scale 10, there won't be much opportunity to reuse existing V2 datasets anyway...

[michaelmckinsey1]

I don't understand when unsharded "V2" datasets would be used? When would those be used? When under scale 11?
In general I think we should discuss the use of keeping around V1 and V2 support. Is it even feasible to fully migrate to V3 and deprecate?

@michaelmckinsey1 I figured leaving V2 support in would be good so we can reuse existing datasets. It does add complexity though, so I'm happy to remove it if you're cool with regenerating datasets for existing problem configs. I suppose since we already have good data up to scale 10, there won't be much opportunity to reuse existing V2 datasets anyway...

I am ok with regenerating them if it reduces complexity.