Give the init container resource requests and limits

[gregorjerse]

Independent of #1256, #1257, #1258 and #1260; branches off master.

Problem

The init container had no resource requests or limits. Kubernetes then gives it the minimal CPU share (cpu.shares 2, cpu.weight 1), so on a busy node the download of the inputs is starved by every pod that has a request, and its oom_score_adj is the highest in the pod, so it is the first process the kernel kills under memory pressure.

Fix

The init container requests 1 CPU and 1Gi of memory, with limits of 2 CPUs and 2Gi. The download runs a few Python threads and needs far less than the processing container: the memory covers the three download workers with their ten in-flight 8 MiB parts each plus the s3transfer io queues. The settings FLOW_KUBERNETES_INIT_CONTAINER_LIMITS and FLOW_KUBERNETES_INIT_CONTAINER_REQUESTS override the defaults, in the shape of the communicator settings. The pod's effective request is the maximum of the init containers and the sum of the app containers, so these values never raise it.

Verification

• Unit test of the helper: the defaults and the settings override. resolwe.flow.tests.test_utils.KubernetesTestCase passes; the new test fails against master.
• The wiring into the pod spec is a one-line reference, verified by reading. No test builds the job description today, since that needs the Kubernetes API, Redis and a located Data object.

[dblenkus]

Right problem — an init container with no requests really does get cpu.weight 1 and the worst oom_score_adj in the pod. Two things about the numbers.

1. "these values never raise it" does not hold for the default process. cores=1, memory=4096 with the default overcommit {"cpu": 0.8, "memory": 0.8} gives 0.8 CPU for processing plus 0.1 for the communicator = 0.9. The init container's 1 CPU raises the pod's effective CPU request for every single-core job, and by more wherever FLOW_KUBERNETES_OVERCOMMIT lowers the factor for a scheduling class. Memory is fine (3276 MiB + 256 M ≫ 1 Gi). Requesting 0.5 CPU would keep the claim true.

2. The 2 Gi limit is not derived from the knobs it depends on. Downloader memory ≈ GENESIS_MAX_DOWNLOAD_THREADS × 10 (boto max_concurrency; use_threads = True is hardcoded in S3Connector) × chunk_size, and chunk_size = max(8 MiB, size/10000). The defaults give the ~240 MiB in the docstring, but an input above ~84 GB pushes chunk_size past 8 MiB (200 GB → 600 MiB, 500 GB → 1.5 GiB), and raising the thread count multiplies it. Going from no limit to 2 Gi converts "slow because starved" into OOMKilled, which backoffLimit: 0 will not retry. Worth scaling the limit from the thread count, or at least stating the assumption.

3. _init_container_resources does not use self.

Note this and #1260 both open an Unreleased/Changed section — CHANGELOG conflict for whichever lands second.