Balance the transfer load between the workers

[gregorjerse]

Stacked on #1257 and #1256. Until they merge, review the last commit only.

Problem

The transfer helpers split the files into one fixed slice per worker (files[i::n]), regardless of size. A worker that drew the large files ran alone long after the others were idle, so the download took as long as the unluckiest slice. The init container also derived the worker count from the file count, one per 100 files, so any input set under 200 files was downloaded by a single worker.

Fix

• The workers share one thread-safe iterator over the files sorted largest first and each takes the next file when it finishes the previous one (SharedIterator, largest_first). Starting the big files first and assigning dynamically keeps every worker busy until the queue drains, so they finish at about the same time, also when a file transfers slower than its size suggests. This is longest-first list scheduling. It lives in paralelize and async_paralelize, so Transfer.transfer_objects (uploads, shared-storage downloads) and StorageLocation.verify_data gain the same balance.
• The init container always uses GENESIS_MAX_DOWNLOAD_THREADS workers (default 3), capped by the number of files.
• Empty input returns no workers instead of raising from asyncio.wait.
• chunks is removed; nothing else used it.

Considered and not done

A static split of the bytes into equal shares per worker balances only on paper: the throughput of a file depends on its size (small files are latency bound), on the S3 side and on the moment, so fixed shares still finish apart. The shared queue balances on actual progress. One file larger than everything else still bounds the download by its own transfer time; that is set by boto's per-file concurrency (10 ranged GETs) and the volume throughput, not by scheduling.

Verification

• New tests: the shared iterator hands out every item exactly once across four threads; largest_first ordering; a balancing test with one slow and nine fast items on two workers, where the old static split gives 5 and 5 items per worker and the shared queue 1 and 9.
• resolwe.storage.tests.test_utils, test_transfer, test_models and resolwe.flow.tests.test_transfer pass (50 tests).

[dblenkus]

Best of the stack — the shared queue is the right fix and the diff is smaller than what it replaces. Dropping chunks also removes an accidental step-slice of a QuerySet in verify_data. Reviewing the last commit only.

1. paralelize now always returns already-done futures (the with joins on the way out). The docstring says so, which is honest, but the "collection of futures, caller checks them" shape is vestigial — both callers immediately .result(). list[bool] would be simpler while the function is being rewritten anyway.
2. verify_data gains the shared queue but not largest-first — largest_first is applied only in transfer_objects. ReferencedPath has a size, so it could be ordered too.
3. paralelize/async_paralelize do objects = list(objects) on the list largest_first just built.

One interaction worth noting: the shared queue means a failing transfer_objects now retries every object rather than one worker's slice, which multiplies the retry-budget concern I raised on #1257.