Test the shard manager's service layer against SQLite, Postgres and etcd, and add a distributed-mode compose example

.agents/skills/understanding-durable-execution/SKILL.md

@@ -34,9 +34,16 @@ and entity bodies), `retries.md` (in-function versus trap-based retries), and
 
 2. **The resident runtime is disposable.** The Wasmtime `Store`, the worker task, the executor
    process, sockets, channels and subscriptions can vanish while work is pending. Suspension,
-   eviction, resharding, restart and crash must all be recoverable the same way: throw the
-   instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`,
+   eviction, restart and crash must all be recoverable the same way, on the same executor: throw
+   the instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`,
    `Interrupted`, `Restart`) change status and scheduling policy, not the recovery mechanism.
+   Losing the shard does not reconstruct in place: a durable agent's oplog asserts this
+   executor's shard epoch whenever it appends or deletes entries in storage, and once another
+   executor holds the shard, the write is refused (`OplogError::Fenced`) instead of accepted. The agent is then *given up*
+   (`InterruptKind::ShardLost`): that generation is stopped, dropped from this executor and never
+   restarted, and the executor that now holds the shard builds a new `Store` and replays — the
+   new owner, or this executor again if the shard came back to it at a higher epoch. See
+   "Resharding, revocation and the oplog epoch fence" below and `crash-matrix.md`.
    Owner: `worker/invocation_loop.rs::run` (outer loop: create instance → recover → run →
    suspend/retry) and `durable_host/mod.rs::prepare_instance`.
 
@@ -114,6 +121,30 @@ and the `RunningWorkers` recovery index remains synchronously flushed by the sta
 The unchanged synchronous `Create` write preserves an ephemeral agent's identity before execution;
 after executor loss, that identity reconstructs an observation-only owner, never a fresh execution.
 
+A commit can also be *refused*. A durable agent's primary oplog, opened while this executor holds
+the agent's shard, asserts that shard epoch whenever it appends or deletes entries in storage — an
+explicit commit, a threshold flush, a deletion — inside the storage transaction. Storage refuses the write with
+`OplogError::Fenced` unless the record it holds names that epoch *and* was written by this
+process (`WriterId`, one per executor process): a newer epoch means another executor took over,
+the same epoch from another process means the epoch was issued twice, and no record at all
+refuses too. An append only buffers, so it meets the check when the buffer is committed.
+`commit_oplog_and_update_state` and `add_and_commit_oplog` surface the refusal rather than
+swallowing it, so a refused `PendingAgentInvocation` commit is not acknowledged as accepted and a
+refused `AgentInvocationFinished` commit is not published to waiters. The first refusal latches:
+every later add or commit on that handle is refused locally, without reaching storage, and the
+agent is given up instead of retried.
+
+The check sits at the commit, not at the effect. A call whose `Start` is only buffered when its
+effect runs — an idempotent `WriteRemote`, which opens no committed scope — can still run on an
+executor that has just lost the shard; its commit is then refused and the new owner runs it
+again, the same window as a crash before that commit. Non-idempotent, batched and transactional
+calls commit their scope `Start` first, so a refusal stops them before the effect. Oplogs that
+assert no epoch are never refused: ephemeral agents' oplogs (their archive layer appends without
+one), a handle opened before this executor has an assignment, fork stages and their publication,
+and compressed archive chunks. Nor do two writes a primary oplog makes outside its entries: the
+prefix it drops once the archive transfer has copied it (`drop_prefix` takes no epoch, and a
+latched fence does not stop the transfer), and blob uploads of large payloads.
+
 `worker/state_actor.rs::commit_and_update_state` samples the appended tip before its explicit
 commit and ignores receipt entries already folded into the published status. Primary/ephemeral
 threshold flushes and replica waits can commit outside the status actor, so even an empty receipt
@@ -170,8 +201,13 @@ tool owners instantiate the deployed component but never queue or replay an agen
 Replay starts from the chosen snapshot baseline
 (see Snapshots and updates), not necessarily from `OplogIndex::INITIAL`. Interruption kinds
 (`Worker::set_interrupting`): `Interrupt` stays interrupted, `Restart` is a simulated crash with
-automatic recovery, `Suspend` unloads and resumes on demand; all three end in the same
-reconstruction path. Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a
+automatic recovery, `Suspend` unloads and resumes on demand — each of these three reconstructs on
+this same executor. `ShardLost` does not: it means this executor lost the agent's shard (a
+revoked/reassigned shard, or an oplog write refused on the shard epoch), and instead of
+reconstructing, that generation is given up — stopped without writing its status, dropped from
+this executor and never restarted — and the executor that now holds the shard reconstructs it.
+
+Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a
 worker that is executing or holds non-durable in-memory work. Ephemeral agents are fail-stop:
 `reconstructed_ephemeral` rebuilds only for observation and result lookup, "but the instance must
 never be started again" (`worker/mod.rs`, `INACTIVE_EPHEMERAL_AGENT_ERROR`).
@@ -301,6 +337,54 @@ in a pending p3 wait). `Resumed` does not enqueue an invocation: the existing
 does not use this marker and retains its normal `Idle`/automatic-recovery semantics; ephemeral
 agents retain their clean fail-stop lifecycle and never append it.
 
+### Resharding, revocation and the oplog epoch fence
+
+Two triggers give an agent up rather than reconstructing it here, and both end in the same place,
+`GiveUpReason` and `InterruptKind::ShardLost`:
+
+- **Assignment change.** `GiveUpReason::ShardRevoked` is a `RevokeShards` push
+  (`grpc/mod.rs::revoke_shards_internal`). `GiveUpReason::ShardNotAssigned` is any delivered
+  assignment — an `AssignShards` push, the set returned at registration, or a lease renewal that
+  corrects it, all through `apply_shard_assignment_effects` — that no longer holds the agent's
+  shard or holds it at a higher epoch than the agent's oplog was opened at (another executor may
+  have written to it meanwhile); a late check after construction uses it too. The reason names the
+  trigger, not the condition. `apply_shard_assignment_effects` then calls the *other*
+  `on_shard_assignment_changed` (`durable_host/mod.rs`, the `WorkerCtx` hook) to recover agents on
+  shards held now, the opposite direction from a give-up. An agent given up for an epoch bump is
+  reopened on this executor at the new epoch: by that recovery if it was in the running-workers
+  index, otherwise by its next invocation. The recovery waits for a live lease.
+- **Oplog epoch fence.** A durable agent's primary oplog asserts the epoch it was opened at
+  whenever it appends or deletes entries, inside the storage transaction, on every indexed-storage backend (a Lua
+  script on Redis, `FOR UPDATE` on Postgres, the single-connection write pool on SQLite, a held
+  entry lock in memory; `storage/indexed/*.rs`, surfaced through `services/oplog/primary.rs`). The
+  shard manager mints a new, higher epoch for each new owner, including a restarted executor
+  process at registration, so a write from an executor that has lost the shard is refused rather
+  than written (`OplogError::Fenced` / `OplogFence`, carrying the asserted and, when known, the
+  stored epoch, and whether the refusal was another writer at the same epoch). This protects an
+  assignment change this executor has not yet heard about, and a revoked lease it is still trying
+  to renew: `GiveUpReason::Fenced`. Once one write is refused the fence *latches*: every later add
+  or commit on that handle is refused without a second round trip to storage, and any trap on the
+  agent classifies as `ShardLost` however the refusal reached it (`TrapType::under_latched_fence`).
+
+Either way, `Worker::give_up` (`worker/mod.rs`) stops the agent, drops it from this executor's
+`ActiveAgents`, and hands its invocation waiters a retriable error rather than an in-place restart:
+`ShardingNotReady`, or `OplogFenced` when the fence carries the epochs, which reaches the client as
+`ShardingNotReady` too, so the worker service refreshes its routing table and retries on the
+owner. Nothing writes a given-up generation's status blob, status checkpoint or recovery-index row
+any more: `mark_given_up` stops its `AgentStatusFlusher` and `StatusCheckpointer` at once, because
+those are unfenced key-value writes that belong to the new owner and a stale one could overwrite
+its status or drop the row its crash recovery relies on. Entries still buffered are committed only if
+storage still accepts this executor's epoch, so the oplog holds nothing the new owner has not
+seen. An invocation still pending in this executor's queue when it gives up is failed the same
+way: with a retriable error and no cached result, never with a result the queue happened to
+already hold, so a client retry runs it exactly once, on the owner. One exception keeps writing:
+a deletion already under way owns the agent's retirement and keeps going. Its stream-cleanup
+commits to the oplog and its storage remove both assert the epoch; its calls to dependent agents
+do not. Each succeeds while the key is still this executor's, and once another executor has taken
+it, the first refused step hands the delete to the new owner. See
+`crash-matrix.md` for the fence's failure modes and `services/active_agents/mod.rs` for the sweep
+that gives agents up on an assignment change.
+
 ## Oplog model
 
 Entries are positional or hints (`OplogEntry::is_hint()`). Replay consumes positional entries in
@@ -745,6 +829,7 @@ satisfies one does not imply the others.
 | "Cursor reached the end, so I can do the live effect now." | Liveness is `store_is_live(...)`: the primary needs `switch_to_live` to publish after reconstruction fences; an entity Store needs its own `local_live_tail`. Cursor exhaustion is neither. | `pending_replay_to_live_is_fail_closed_until_finished`, `entity_store_liveness_is_scoped_to_its_invocation_mode` |
 | "The voluntary-suspension predicate gates interruption or recovery." | It only defers proactive yielding while live work progresses; explicit interruption and arbitrary Store loss still use ordinary reconstruction. | Simulated-crash tests at arbitrary points (`simulated_crash`, `interrupt`) |
 | "Restart differs from suspend." | Both discard the `Store` and reconstruct. | `counter_resource_test_2_with_restart` (state continues across an executor restart), `reacquire_permits_restart_preserves_accepted_queued_live_invocation` |
+| "Losing a shard reconstructs the agent, like a restart." | It is given up instead: that generation is stopped without writing its status, dropped and never restarted. Only the executor now holding the shard reconstructs it — the new owner, or this one if the shard came back at a higher epoch. | `worker/mod.rs::a_failure_that_is_a_lost_shard_is_given_up_on_every_path`, `services/oplog/tests.rs::a_fenced_oplog_refuses_new_adds_and_keeps_the_indices_it_handed_out_readable`, the two-writer tests in `tests/indexed_storage.rs` |
 | "A retried RPC attempt executed the target again." | Same key ⇒ same target invocation; count target mutations, not attempts. | Provider-side counter tests in `tests/rpc.rs` |
 | "Atomic rollback should generate a fresh RPC key." | Logical counter is owned by the outermost atomic region; keys survive `Jump`. | `tests/transactions.rs`, `tests/revert.rs` |
 | "Equal return values prove deduplication." | Deterministic echoes are equal even with duplicate execution; count side effects. | Counter-based RPC tests |

.agents/skills/understanding-durable-execution/reference/crash-matrix.md

@@ -1,10 +1,17 @@
 # Crash-window matrix
 
-"Crash" here means any loss of the resident runtime: process death, `Restart` (simulated crash),
-`Suspend`, eviction, resharding (`on_shard_assignment_changed`), or an executor drop in a test.
-Reconstruction is identical in every case: new `Store`, `prepare_instance`, `resume_replay`,
-publish Live. The matrix says what the next incarnation does for a crash inside each window and
-which durable fact makes that safe.
+"Crash" here means any loss of the resident runtime *on the same executor*: process death,
+`Restart` (simulated crash), `Suspend`, eviction, or an executor drop in a test. Reconstruction is
+identical in every case: new `Store`, `prepare_instance`, `resume_replay`, publish Live. The
+matrix says what the next incarnation does for a crash inside each window and which durable fact
+makes that safe.
+
+Resharding and the oplog epoch fence are different: the generation running here does not
+reconstruct at all. It is given up (`InterruptKind::ShardLost`) — stopped without writing its
+status, dropped here, never restarted — and the executor that now holds the shard runs
+`prepare_instance` / `resume_replay` from the committed oplog: the new owner, or this executor
+again if the shard came back to it at a higher epoch. See "Resharding and the oplog epoch fence"
+below for what that leaves behind.
 
 ## Durable host call (`concurrent/call.rs`, `concurrent/delivery.rs`)
 
@@ -94,6 +101,27 @@ which durable fact makes that safe.
 | Body traps | no entity terminal | Owner invocation fails; owner group drains; siblings blocked on the lane are fenced | `guest_trap_fences_a_blocked_sibling_and_drains_the_owner_group` |
 | Owner reaches replay tail while a body is still reconstructing | — | `HistoricalReconstruction` fences keep `PendingReplayToLive` closed until every active body validates | `completed_reconstruction_claim_blocks_concurrent_replay_to_live` |
 
+## Resharding and the oplog epoch fence (`worker/mod.rs::give_up`, `services/oplog/primary.rs`)
+
+Two triggers give up an agent instead of reconstructing it here: the shard manager revoking or
+reassigning the shard (`GiveUpReason::ShardRevoked` for a `RevokeShards` push,
+`GiveUpReason::ShardNotAssigned` for any delivered assignment that drops the shard or raises its
+epoch), and a write refused because the epoch this executor asserted no longer matches storage
+(`OplogError::Fenced`, `GiveUpReason::Fenced`). Every indexed-storage backend refuses such a write.
+Only a durable agent's primary oplog asserts an epoch; ephemeral oplogs, fork stages and archive
+layers do not, so an ephemeral agent is given up only by an assignment change.
+
+| Crash window | Oplog shape left behind | What happens here | Durable fact relied on |
+|---|---|---|---|
+| Assignment revoked/reassigned, before any write is attempted | whatever was already committed, plus any buffered entries the stop commits while storage still accepts this executor's epoch | `give_up_matching` stops matching agents directly; no status blob, checkpoint or recovery-index row is written (`mark_given_up` stops the flusher and checkpointer) | `ShardService::check_worker` / the delivered assignment, not the oplog |
+| The shard moves while a live call's `Start` is only buffered | nothing from this call | Its effect has already run here (an idempotent `WriteRemote` opens no committed scope); the next commit is refused and the agent gives up; the owner runs the call again | Idempotence mode, as for a crash before the commit; non-idempotent, batched and transactional calls commit their scope `Start` first |
+| A write is attempted after the shard actually moved | nothing new; the attempted batch is refused, not partially written | The refusal is returned (`OplogError::Fenced`), not retried or swallowed; the agent gives up | Epoch asserted inside the storage transaction |
+| An earlier attempt of the refused batch ended indeterminate | that attempt's entries, if it landed before the takeover | The refusal is still returned, so the batch is never acknowledged here; the owner replays it like any committed entry | Nothing is acknowledged that the owner cannot see |
+| Any later write on the same oplog handle | still nothing new | The fence latches: every later add/commit is refused immediately, without a second storage round trip | The oplog's own latched `OplogFence` |
+| An invocation still queued when the give-up runs | unaffected; its `PendingAgentInvocation` stays pending | Failed in memory with a retriable error (`fail_pending_invocations` / `give_up_error`: `ShardingNotReady`, or `OplogFenced`), never a cached result | The `PendingAgentInvocation` left pending in the oplog, for the owner to run |
+| A deletion is under way when the give-up runs | whatever the deletion had committed | The deletion keeps going: its stream-cleanup commits and its storage remove run with the epoch asserted (its calls to dependent agents do not). Each succeeds while the key is still this executor's; once another executor holds it, the first refused step hands the delete to that owner | Epoch asserted by the cleanup commits and by the remove |
+| The owner opens the same agent | the fenced executor's last accepted entries | Ordinary `prepare_instance` / `resume_replay`, from committed history exactly as it was left | Nothing is acknowledged after the fence latched, and no entry is appended or deleted without the asserted epoch |
+
 ## Oplog-processor plugins (`services/oplog/plugin.rs`)
 
 | Crash window | Recovery |

Makefile.toml

@@ -724,6 +724,7 @@ cargo-test-r run --package golem-service-base --test '*' -- --nocapture --report
 cargo-test-r run --package golem-registry-service --test '*' -- --nocapture --report-time $JUNIT_OPTS
 cargo-test-r run --package golem-worker-service --test '*' -- --nocapture --report-time $JUNIT_OPTS
 cargo-test-r run --package golem-shard-manager --test integration -- --nocapture --report-time $JUNIT_OPTS
+cargo-test-r run --package golem-test-framework --test signal_unreaped_child -- --nocapture --report-time $JUNIT_OPTS
 '''
 
 [tasks.integration-tests-group6]

docker-examples/distributed-etcd/.env

@@ -0,0 +1,24 @@
+GOLEM_IMAGES_VERSION=v1.5.0
+POSTGRES_IMAGE_VERSION=17
+REDIS_IMAGE_VERSION=8
+ETCD_IMAGE_VERSION=v3.5.17
+NGINX_IMAGE_VERSION=1.29
+
+GOLEM_ROUTER_PORT=9881 # Golem APIs proxy
+CORS_ORIGIN_REGEX="http://localhost:9881"
+ADMIN_TOKEN="n0AstqJ6cBgh9ob2BBT_h39t7B00I-8wSnBBix2lo-I"
+MARKETING_TOKEN="evJ-8NPn7hIPBJ-htGgfZkk36PeA-UGAmT2iKasTFM4"
+GOLEM_ROUTER_COMPONENT_MAX_SIZE_ALLOWED=1g
+
+REGISTRY_SERVICE_HTTP_PORT=8083
+REGISTRY_SERVICE_GRPC_PORT=9008
+COMPONENT_COMPILATION_SERVICE_HTTP_PORT=8084
+COMPONENT_COMPILATION_SERVICE_GRPC_PORT=9010
+SHARD_MANAGER_HTTP_PORT=8081
+SHARD_MANAGER_GRPC_PORT=9002
+WORKER_EXECUTOR_HTTP_PORT=8082
+WORKER_EXECUTOR_GRPC_PORT=9000
+WORKER_SERVICE_HTTP_PORT=9005
+WORKER_SERVICE_CUSTOM_REQUEST_PORT=9006 # worker API Gateway
+WORKER_SERVICE_MCP_PORT=9007
+WORKER_SERVICE_GRPC_PORT=9094