Reduce the worker-side discovery to an inert stub instead of rebuilding it

[conradbzura]

Description

Have the workflow discovery's pickle reduce target an inert stub on the worker side instead of faithfully reconstructing a functional EcsDiscovery/_EcsSubscriber.

wool serializes the dispatched task's WorkerProxy (task.to_protobuf → proxy_dumps → cloudpickle), and WorkerProxy.__wool_reduce__ drags the caller's discovery across the boundary into the worker. cfdb does single-level dispatch only — there is exactly one @wool.routine (src/cfdb/workflows/executor.py:182), and its body runs the preprocessing pipeline (processors shelling out to samtools/bgzip/tabix); it constructs no WorkerPool, holds no WorkerProxy, and calls no other routine. WorkerPool/discovery are constructed only dispatch-side (the API in src/cfdb/api/main.py, the LAN pool host in src/cfdb/workflows/worker_lan.py). So the discovery wool ships to the worker is never used worker-side.

Today's pickle fixes (#54 for EcsDiscovery, #60 for _EcsSubscriber) faithfully rebuild that unused object on unpickle: EcsDiscovery.__setstate__ calls build_ecs_client(...) to mint a fresh boto3 ECS client and fresh asyncio.Locks; _EcsSubscriber.__setstate__ mints a fresh queue. This is dead baggage — and worse, it forces the worker to be able to construct a boto3 ECS client (region resolution, credential chain) purely to reconstruct an object it never touches: a latent worker-side failure surface for zero functional benefit.

Proposed end state: the discovery reduces (via __reduce__) to an inert stub that satisfies wool's structural Discovery protocol but whose activation methods (__aenter__/__aexit__, subscribe, subscriber, publisher, poll_once) raise NotImplementedError("discovery is inert worker-side; nested dispatch is unsupported"). The stub needs no boto3 client, no region, no credentials, and no transient-field stripping. This is the natural terminus of the __reduce__ path and the cfdb-side counterpart of the durable factory-callable-discovery direction.

Motivation

• Remove a latent worker-side failure surface. The worker currently must be able to build a boto3 ECS client (region/credentials) just to reconstruct dead baggage; any future change to client construction can break unpickling on the worker for no functional reason.
• Stop hand-maintaining pickle state on a transient-heavy object. Fix ECS workflow dispatch: unpicklable EcsDiscovery in proxy, cold-start TimeoutError, and exhausted subscriber #54 and Make _EcsSubscriber cloudpickle-safe so ECS workflow dispatch survives the wool reduce boundary #60 each added __getstate__/__setstate__ strip-and-rebuild logic; a third such object — or a new transient field — re-opens the same drift hazard. An inert stub eliminates the rebuild path entirely and supersedes both interim fixes.
• Make the "never used worker-side" assumption explicit and fail-fast. Today it's an unenforced invariant; a raising stub converts it into a loud failure if nested dispatch is ever added without revisiting this.

Expected Outcome

• The object wool drags to the worker deserializes into an inert stub; the worker constructs no boto3 ECS client and needs no AWS region/credentials to unpickle it.
• Activating the stub worker-side (entering it / subscribe / poll_once / publisher) raises NotImplementedError naming nested dispatch as the unsupported case.
• The interim __getstate__/__setstate__ hand-stripping on EcsDiscovery (Fix ECS workflow dispatch: unpicklable EcsDiscovery in proxy, cold-start TimeoutError, and exhausted subscriber #54) and _EcsSubscriber (Make _EcsSubscriber cloudpickle-safe so ECS workflow dispatch survives the wool reduce boundary #60) is removed in favor of the reduce-to-stub path (or kept only where still load-bearing).
• A regression test asserts the reduced object is the inert stub (not a rebuilt discovery) and that its activation methods raise.
• Verified precondition: wool does not enter/activate the proxy's discovery during normal single-level dispatch (only on nested dispatch). Confirm via the live GET /index/{dcc}/{local_id} E2E completing with the stub in place; if wool does touch it during normal dispatch, narrow which methods the stub must implement as functional no-ops rather than raisers.

[conradbzura]

Two pieces of evidence gathered while working on #85/#90, carried over from #93 (closed as a duplicate of this issue).

The rebuild now has a visible cost in the test suite. tests/integration/conftest.py grew an offline_aws_env fixture that sets static AWS credentials and clears AWS_PROFILE before the worker spawns, solely so EcsDiscovery.__setstate__ can construct a client the worker never calls. Without it, dispatch fails with MissingDependencyException for the botocore[crt] extra when botocore's chain falls through to the login provider; with a stray AWS_PROFILE, ProfileNotFound. Both surface as a dispatch failure, indistinguishable from an mTLS or capacity fault, which is why the fixture's docstring has to explain the whole chain.

That makes a clean acceptance signal for this issue: when the reduce-to-stub path lands, offline_aws_env can be deleted along with tests/integration/test_ecs_discovery_dispatch.py's dependency on it.

The contrast with S3Cache is worth stating explicitly. EcsDiscovery.__setstate__'s comment says the guard mirrors S3Cache.__setstate__, and the pattern was copied from there — but S3Cache's worker-side client is genuinely used (head_object, upload_file, delete_object all run in the worker). EcsDiscovery's is not. The shape was copied without the justification, so S3Cache should keep its rebuild while this one goes away.