diff --git a/.agents/skills/understanding-durable-execution/SKILL.md b/.agents/skills/understanding-durable-execution/SKILL.md index 895cf29f86..c50fad4d1c 100644 --- a/.agents/skills/understanding-durable-execution/SKILL.md +++ b/.agents/skills/understanding-durable-execution/SKILL.md @@ -34,9 +34,16 @@ and entity bodies), `retries.md` (in-function versus trap-based retries), and 2. **The resident runtime is disposable.** The Wasmtime `Store`, the worker task, the executor process, sockets, channels and subscriptions can vanish while work is pending. Suspension, - eviction, resharding, restart and crash must all be recoverable the same way: throw the - instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`, + eviction, restart and crash must all be recoverable the same way, on the same executor: throw + the instance away, build a new `Store`, replay the oplog, continue. Lifecycle hints (`Suspend`, `Interrupted`, `Restart`) change status and scheduling policy, not the recovery mechanism. + Losing the shard does not reconstruct in place: a durable agent's oplog asserts this + executor's shard epoch whenever it appends or deletes entries in storage, and once another + executor holds the shard, the write is refused (`OplogError::Fenced`) instead of accepted. The agent is then *given up* + (`InterruptKind::ShardLost`): that generation is stopped, dropped from this executor and never + restarted, and the executor that now holds the shard builds a new `Store` and replays — the + new owner, or this executor again if the shard came back to it at a higher epoch. See + "Resharding, revocation and the oplog epoch fence" below and `crash-matrix.md`. Owner: `worker/invocation_loop.rs::run` (outer loop: create instance → recover → run → suspend/retry) and `durable_host/mod.rs::prepare_instance`. @@ -114,6 +121,30 @@ and the `RunningWorkers` recovery index remains synchronously flushed by the sta The unchanged synchronous `Create` write preserves an ephemeral agent's identity before execution; after executor loss, that identity reconstructs an observation-only owner, never a fresh execution. +A commit can also be *refused*. A durable agent's primary oplog, opened while this executor holds +the agent's shard, asserts that shard epoch whenever it appends or deletes entries in storage — an +explicit commit, a threshold flush, a deletion — inside the storage transaction. Storage refuses the write with +`OplogError::Fenced` unless the record it holds names that epoch *and* was written by this +process (`WriterId`, one per executor process): a newer epoch means another executor took over, +the same epoch from another process means the epoch was issued twice, and no record at all +refuses too. An append only buffers, so it meets the check when the buffer is committed. +`commit_oplog_and_update_state` and `add_and_commit_oplog` surface the refusal rather than +swallowing it, so a refused `PendingAgentInvocation` commit is not acknowledged as accepted and a +refused `AgentInvocationFinished` commit is not published to waiters. The first refusal latches: +every later add or commit on that handle is refused locally, without reaching storage, and the +agent is given up instead of retried. + +The check sits at the commit, not at the effect. A call whose `Start` is only buffered when its +effect runs — an idempotent `WriteRemote`, which opens no committed scope — can still run on an +executor that has just lost the shard; its commit is then refused and the new owner runs it +again, the same window as a crash before that commit. Non-idempotent, batched and transactional +calls commit their scope `Start` first, so a refusal stops them before the effect. Oplogs that +assert no epoch are never refused: ephemeral agents' oplogs (their archive layer appends without +one), a handle opened before this executor has an assignment, fork stages and their publication, +and compressed archive chunks. Nor do two writes a primary oplog makes outside its entries: the +prefix it drops once the archive transfer has copied it (`drop_prefix` takes no epoch, and a +latched fence does not stop the transfer), and blob uploads of large payloads. + `worker/state_actor.rs::commit_and_update_state` samples the appended tip before its explicit commit and ignores receipt entries already folded into the published status. Primary/ephemeral threshold flushes and replica waits can commit outside the status actor, so even an empty receipt @@ -170,8 +201,13 @@ tool owners instantiate the deployed component but never queue or replay an agen Replay starts from the chosen snapshot baseline (see Snapshots and updates), not necessarily from `OplogIndex::INITIAL`. Interruption kinds (`Worker::set_interrupting`): `Interrupt` stays interrupted, `Restart` is a simulated crash with -automatic recovery, `Suspend` unloads and resumes on demand; all three end in the same -reconstruction path. Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a +automatic recovery, `Suspend` unloads and resumes on demand — each of these three reconstructs on +this same executor. `ShardLost` does not: it means this executor lost the agent's shard (a +revoked/reassigned shard, or an oplog write refused on the shard epoch), and instead of +reconstructing, that generation is given up — stopped without writing its status, dropped from +this executor and never restarted — and the executor that now holds the shard reconstructs it. + +Eviction (`EvictionClass::{LoadedIdle, WarmRunnable}`) never unloads a worker that is executing or holds non-durable in-memory work. Ephemeral agents are fail-stop: `reconstructed_ephemeral` rebuilds only for observation and result lookup, "but the instance must never be started again" (`worker/mod.rs`, `INACTIVE_EPHEMERAL_AGENT_ERROR`). @@ -301,6 +337,54 @@ in a pending p3 wait). `Resumed` does not enqueue an invocation: the existing does not use this marker and retains its normal `Idle`/automatic-recovery semantics; ephemeral agents retain their clean fail-stop lifecycle and never append it. +### Resharding, revocation and the oplog epoch fence + +Two triggers give an agent up rather than reconstructing it here, and both end in the same place, +`GiveUpReason` and `InterruptKind::ShardLost`: + +- **Assignment change.** `GiveUpReason::ShardRevoked` is a `RevokeShards` push + (`grpc/mod.rs::revoke_shards_internal`). `GiveUpReason::ShardNotAssigned` is any delivered + assignment — an `AssignShards` push, the set returned at registration, or a lease renewal that + corrects it, all through `apply_shard_assignment_effects` — that no longer holds the agent's + shard or holds it at a higher epoch than the agent's oplog was opened at (another executor may + have written to it meanwhile); a late check after construction uses it too. The reason names the + trigger, not the condition. `apply_shard_assignment_effects` then calls the *other* + `on_shard_assignment_changed` (`durable_host/mod.rs`, the `WorkerCtx` hook) to recover agents on + shards held now, the opposite direction from a give-up. An agent given up for an epoch bump is + reopened on this executor at the new epoch: by that recovery if it was in the running-workers + index, otherwise by its next invocation. The recovery waits for a live lease. +- **Oplog epoch fence.** A durable agent's primary oplog asserts the epoch it was opened at + whenever it appends or deletes entries, inside the storage transaction, on every indexed-storage backend (a Lua + script on Redis, `FOR UPDATE` on Postgres, the single-connection write pool on SQLite, a held + entry lock in memory; `storage/indexed/*.rs`, surfaced through `services/oplog/primary.rs`). The + shard manager mints a new, higher epoch for each new owner, including a restarted executor + process at registration, so a write from an executor that has lost the shard is refused rather + than written (`OplogError::Fenced` / `OplogFence`, carrying the asserted and, when known, the + stored epoch, and whether the refusal was another writer at the same epoch). This protects an + assignment change this executor has not yet heard about, and a revoked lease it is still trying + to renew: `GiveUpReason::Fenced`. Once one write is refused the fence *latches*: every later add + or commit on that handle is refused without a second round trip to storage, and any trap on the + agent classifies as `ShardLost` however the refusal reached it (`TrapType::under_latched_fence`). + +Either way, `Worker::give_up` (`worker/mod.rs`) stops the agent, drops it from this executor's +`ActiveAgents`, and hands its invocation waiters a retriable error rather than an in-place restart: +`ShardingNotReady`, or `OplogFenced` when the fence carries the epochs, which reaches the client as +`ShardingNotReady` too, so the worker service refreshes its routing table and retries on the +owner. Nothing writes a given-up generation's status blob, status checkpoint or recovery-index row +any more: `mark_given_up` stops its `AgentStatusFlusher` and `StatusCheckpointer` at once, because +those are unfenced key-value writes that belong to the new owner and a stale one could overwrite +its status or drop the row its crash recovery relies on. Entries still buffered are committed only if +storage still accepts this executor's epoch, so the oplog holds nothing the new owner has not +seen. An invocation still pending in this executor's queue when it gives up is failed the same +way: with a retriable error and no cached result, never with a result the queue happened to +already hold, so a client retry runs it exactly once, on the owner. One exception keeps writing: +a deletion already under way owns the agent's retirement and keeps going. Its stream-cleanup +commits to the oplog and its storage remove both assert the epoch; its calls to dependent agents +do not. Each succeeds while the key is still this executor's, and once another executor has taken +it, the first refused step hands the delete to the new owner. See +`crash-matrix.md` for the fence's failure modes and `services/active_agents/mod.rs` for the sweep +that gives agents up on an assignment change. + ## Oplog model Entries are positional or hints (`OplogEntry::is_hint()`). Replay consumes positional entries in @@ -745,6 +829,7 @@ satisfies one does not imply the others. | "Cursor reached the end, so I can do the live effect now." | Liveness is `store_is_live(...)`: the primary needs `switch_to_live` to publish after reconstruction fences; an entity Store needs its own `local_live_tail`. Cursor exhaustion is neither. | `pending_replay_to_live_is_fail_closed_until_finished`, `entity_store_liveness_is_scoped_to_its_invocation_mode` | | "The voluntary-suspension predicate gates interruption or recovery." | It only defers proactive yielding while live work progresses; explicit interruption and arbitrary Store loss still use ordinary reconstruction. | Simulated-crash tests at arbitrary points (`simulated_crash`, `interrupt`) | | "Restart differs from suspend." | Both discard the `Store` and reconstruct. | `counter_resource_test_2_with_restart` (state continues across an executor restart), `reacquire_permits_restart_preserves_accepted_queued_live_invocation` | +| "Losing a shard reconstructs the agent, like a restart." | It is given up instead: that generation is stopped without writing its status, dropped and never restarted. Only the executor now holding the shard reconstructs it — the new owner, or this one if the shard came back at a higher epoch. | `worker/mod.rs::a_failure_that_is_a_lost_shard_is_given_up_on_every_path`, `services/oplog/tests.rs::a_fenced_oplog_refuses_new_adds_and_keeps_the_indices_it_handed_out_readable`, the two-writer tests in `tests/indexed_storage.rs` | | "A retried RPC attempt executed the target again." | Same key ⇒ same target invocation; count target mutations, not attempts. | Provider-side counter tests in `tests/rpc.rs` | | "Atomic rollback should generate a fresh RPC key." | Logical counter is owned by the outermost atomic region; keys survive `Jump`. | `tests/transactions.rs`, `tests/revert.rs` | | "Equal return values prove deduplication." | Deterministic echoes are equal even with duplicate execution; count side effects. | Counter-based RPC tests | diff --git a/.agents/skills/understanding-durable-execution/reference/crash-matrix.md b/.agents/skills/understanding-durable-execution/reference/crash-matrix.md index 975b034e26..f748831a52 100644 --- a/.agents/skills/understanding-durable-execution/reference/crash-matrix.md +++ b/.agents/skills/understanding-durable-execution/reference/crash-matrix.md @@ -1,10 +1,17 @@ # Crash-window matrix -"Crash" here means any loss of the resident runtime: process death, `Restart` (simulated crash), -`Suspend`, eviction, resharding (`on_shard_assignment_changed`), or an executor drop in a test. -Reconstruction is identical in every case: new `Store`, `prepare_instance`, `resume_replay`, -publish Live. The matrix says what the next incarnation does for a crash inside each window and -which durable fact makes that safe. +"Crash" here means any loss of the resident runtime *on the same executor*: process death, +`Restart` (simulated crash), `Suspend`, eviction, or an executor drop in a test. Reconstruction is +identical in every case: new `Store`, `prepare_instance`, `resume_replay`, publish Live. The +matrix says what the next incarnation does for a crash inside each window and which durable fact +makes that safe. + +Resharding and the oplog epoch fence are different: the generation running here does not +reconstruct at all. It is given up (`InterruptKind::ShardLost`) — stopped without writing its +status, dropped here, never restarted — and the executor that now holds the shard runs +`prepare_instance` / `resume_replay` from the committed oplog: the new owner, or this executor +again if the shard came back to it at a higher epoch. See "Resharding and the oplog epoch fence" +below for what that leaves behind. ## Durable host call (`concurrent/call.rs`, `concurrent/delivery.rs`) @@ -94,6 +101,27 @@ which durable fact makes that safe. | Body traps | no entity terminal | Owner invocation fails; owner group drains; siblings blocked on the lane are fenced | `guest_trap_fences_a_blocked_sibling_and_drains_the_owner_group` | | Owner reaches replay tail while a body is still reconstructing | — | `HistoricalReconstruction` fences keep `PendingReplayToLive` closed until every active body validates | `completed_reconstruction_claim_blocks_concurrent_replay_to_live` | +## Resharding and the oplog epoch fence (`worker/mod.rs::give_up`, `services/oplog/primary.rs`) + +Two triggers give up an agent instead of reconstructing it here: the shard manager revoking or +reassigning the shard (`GiveUpReason::ShardRevoked` for a `RevokeShards` push, +`GiveUpReason::ShardNotAssigned` for any delivered assignment that drops the shard or raises its +epoch), and a write refused because the epoch this executor asserted no longer matches storage +(`OplogError::Fenced`, `GiveUpReason::Fenced`). Every indexed-storage backend refuses such a write. +Only a durable agent's primary oplog asserts an epoch; ephemeral oplogs, fork stages and archive +layers do not, so an ephemeral agent is given up only by an assignment change. + +| Crash window | Oplog shape left behind | What happens here | Durable fact relied on | +|---|---|---|---| +| Assignment revoked/reassigned, before any write is attempted | whatever was already committed, plus any buffered entries the stop commits while storage still accepts this executor's epoch | `give_up_matching` stops matching agents directly; no status blob, checkpoint or recovery-index row is written (`mark_given_up` stops the flusher and checkpointer) | `ShardService::check_worker` / the delivered assignment, not the oplog | +| The shard moves while a live call's `Start` is only buffered | nothing from this call | Its effect has already run here (an idempotent `WriteRemote` opens no committed scope); the next commit is refused and the agent gives up; the owner runs the call again | Idempotence mode, as for a crash before the commit; non-idempotent, batched and transactional calls commit their scope `Start` first | +| A write is attempted after the shard actually moved | nothing new; the attempted batch is refused, not partially written | The refusal is returned (`OplogError::Fenced`), not retried or swallowed; the agent gives up | Epoch asserted inside the storage transaction | +| An earlier attempt of the refused batch ended indeterminate | that attempt's entries, if it landed before the takeover | The refusal is still returned, so the batch is never acknowledged here; the owner replays it like any committed entry | Nothing is acknowledged that the owner cannot see | +| Any later write on the same oplog handle | still nothing new | The fence latches: every later add/commit is refused immediately, without a second storage round trip | The oplog's own latched `OplogFence` | +| An invocation still queued when the give-up runs | unaffected; its `PendingAgentInvocation` stays pending | Failed in memory with a retriable error (`fail_pending_invocations` / `give_up_error`: `ShardingNotReady`, or `OplogFenced`), never a cached result | The `PendingAgentInvocation` left pending in the oplog, for the owner to run | +| A deletion is under way when the give-up runs | whatever the deletion had committed | The deletion keeps going: its stream-cleanup commits and its storage remove run with the epoch asserted (its calls to dependent agents do not). Each succeeds while the key is still this executor's; once another executor holds it, the first refused step hands the delete to that owner | Epoch asserted by the cleanup commits and by the remove | +| The owner opens the same agent | the fenced executor's last accepted entries | Ordinary `prepare_instance` / `resume_replay`, from committed history exactly as it was left | Nothing is acknowledged after the fence latched, and no entry is appended or deleted without the asserted epoch | + ## Oplog-processor plugins (`services/oplog/plugin.rs`) | Crash window | Recovery | diff --git a/Cargo.lock b/Cargo.lock index fa1e64c272..2e137e6957 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4336,6 +4336,7 @@ dependencies = [ "heck", "humantime-serde", "itertools 0.14.0", + "libc", "log", "opentelemetry 0.30.0", "opentelemetry_sdk 0.30.0", diff --git a/Makefile.toml b/Makefile.toml index e460a6aeb4..931f5f80f3 100644 --- a/Makefile.toml +++ b/Makefile.toml @@ -724,6 +724,7 @@ cargo-test-r run --package golem-service-base --test '*' -- --nocapture --report cargo-test-r run --package golem-registry-service --test '*' -- --nocapture --report-time $JUNIT_OPTS cargo-test-r run --package golem-worker-service --test '*' -- --nocapture --report-time $JUNIT_OPTS cargo-test-r run --package golem-shard-manager --test integration -- --nocapture --report-time $JUNIT_OPTS +cargo-test-r run --package golem-test-framework --test signal_unreaped_child -- --nocapture --report-time $JUNIT_OPTS ''' [tasks.integration-tests-group6] diff --git a/docker-examples/distributed-etcd/.env b/docker-examples/distributed-etcd/.env new file mode 100644 index 0000000000..a96f07e550 --- /dev/null +++ b/docker-examples/distributed-etcd/.env @@ -0,0 +1,24 @@ +GOLEM_IMAGES_VERSION=v1.5.0 +POSTGRES_IMAGE_VERSION=17 +REDIS_IMAGE_VERSION=8 +ETCD_IMAGE_VERSION=v3.5.17 +NGINX_IMAGE_VERSION=1.29 + +GOLEM_ROUTER_PORT=9881 # Golem APIs proxy +CORS_ORIGIN_REGEX="http://localhost:9881" +ADMIN_TOKEN="n0AstqJ6cBgh9ob2BBT_h39t7B00I-8wSnBBix2lo-I" +MARKETING_TOKEN="evJ-8NPn7hIPBJ-htGgfZkk36PeA-UGAmT2iKasTFM4" +GOLEM_ROUTER_COMPONENT_MAX_SIZE_ALLOWED=1g + +REGISTRY_SERVICE_HTTP_PORT=8083 +REGISTRY_SERVICE_GRPC_PORT=9008 +COMPONENT_COMPILATION_SERVICE_HTTP_PORT=8084 +COMPONENT_COMPILATION_SERVICE_GRPC_PORT=9010 +SHARD_MANAGER_HTTP_PORT=8081 +SHARD_MANAGER_GRPC_PORT=9002 +WORKER_EXECUTOR_HTTP_PORT=8082 +WORKER_EXECUTOR_GRPC_PORT=9000 +WORKER_SERVICE_HTTP_PORT=9005 +WORKER_SERVICE_CUSTOM_REQUEST_PORT=9006 # worker API Gateway +WORKER_SERVICE_MCP_PORT=9007 +WORKER_SERVICE_GRPC_PORT=9094 diff --git a/docker-examples/distributed-etcd/README.md b/docker-examples/distributed-etcd/README.md new file mode 100644 index 0000000000..56aa13ddbc --- /dev/null +++ b/docker-examples/distributed-etcd/README.md @@ -0,0 +1,90 @@ +# Golem with a distributed shard manager (etcd) + +Same stack as `../published-postgres`, with one difference: the shard manager stores its shard +lease state in **etcd** instead of a SQL database, which is what allows more than one shard manager +replica to run. Every other service still uses Postgres. + +```sh +docker compose up +``` + +The Golem APIs are then on `http://localhost:9881`, as in the other example. + +Distributed mode is newer than the image tag in `.env` (`GOLEM_IMAGES_VERSION`), which is shared with +`published-postgres`. Point that variable at a release that includes it, or at an image built from +this repository, before bringing the stack up. + +### What has actually been verified + +The compose file parses (`docker compose config`), and the shard manager's half of it was checked +directly: an etcd container started with the flags below, and a shard manager built from this +repository run against it with this file's environment variables. It elected a leader and served: + +``` +INFO golem_shard_manager: Configured the etcd client for shard lease state persistence + endpoints="http://127.0.0.1:12379" state_key="/golem/shard-manager/state" +INFO ...leader_election: Elected as the shard manager leader + leader_key="/golem/shard-manager/leader/..." create_revision=2 granted_ttl=10s +INFO golem_shard_manager: Started shard manager on ports: grpc: 19002 +``` + +with `shard_manager_is_leader 1` on the HTTP port's `/metrics`, and on `SIGTERM` it logged +`Released the shard manager leadership` before exiting. (The ports differ from this file's only +because that run was on the host.) Both misconfigurations this file warns about stop the shard +manager at startup: + +- `https://` endpoint → exits 1 with `Error: Internal error: etcd endpoint https://... must start + with http:// (TLS is not supported)` +- unbracketed endpoint list → `Failed to load config: invalid type: found string "...", expected a + sequence for key "PERSISTENCE"`. This one exits with status **0**, as every Golem service does on a + config it cannot load, so `restart: on-failure` leaves the container stopped and it looks like a + clean exit; check its log. + +The rest of the stack — a worker executor registering, quota degrading — has **not** been run, +because the published images predate this mode. + +## What changes, and why + +| | `published-postgres` | here | +|---|---|---| +| `GOLEM__PERSISTENCE__TYPE` | `Postgres` | `Etcd` | +| Shard manager replicas | exactly one | any number; one is elected, the rest stand by | +| Quota service | available | **unavailable** | + +The endpoint list must be bracketed — `'["http://etcd:2379"]'`. Unbracketed it is read as a single +string and fails to deserialize. Only `http://` endpoints are accepted: TLS is not configurable and +the shard manager refuses to start on `https://`. + +## Limitations of this example + +**One etcd node.** It is enough to show the mode, not to survive losing the container — etcd is now +the shard manager's durable state, so a real deployment runs a cluster. + +**One shard manager replica.** Compose has no readiness gating, so scaling this service up here +would not behave like a real deployment. See below. + +**Quota is not enforced in this mode.** In distributed mode the quota repository is wired to an +unavailable implementation; quota state has not moved into etcd. Use local (SQL) mode if you need it. + +## Running more than one replica for real + +A standby **does not open its gRPC port until it is elected** — that is what routes traffic to the +leader. It does bind its HTTP port while campaigning, so: + +- **liveness** probe → the HTTP port (`GOLEM__HTTP_PORT`) +- **readiness** probe → the gRPC port (`GOLEM__GRPC__PORT`) + +Wiring readiness to HTTP would put every standby into the service's endpoint list, and clients would +get connection-refused on every request that did not land on the leader. Clients hold a single shard +manager address and do not fail over between replicas themselves. + +During a failover the gRPC port is closed on every replica. A graceful handover takes milliseconds; +an ungraceful one (kill or partition) takes between two thirds and one times `leader_lease_ttl` +(10s by default), plus the new leader's own startup, whose initial executor health check is capped +at 15s. A worker executor that starts inside that window retries its registration for +about 10s and then exits. + +`/metrics` on the HTTP port distinguishes the roles: `shard_manager_is_leader` is 1 on the leader and +0 on a standby. + +See `docs/src/content/next/deploy.mdx` for the full deployment guide. diff --git a/docker-examples/distributed-etcd/compose.yaml b/docker-examples/distributed-etcd/compose.yaml new file mode 100644 index 0000000000..4bf4ee3373 --- /dev/null +++ b/docker-examples/distributed-etcd/compose.yaml @@ -0,0 +1,282 @@ +x-aliases: + blob-storage-env-vars: &blob-storage-env-vars + GOLEM__BLOB_STORAGE__TYPE: "LocalFileSystem" + GOLEM__BLOB_STORAGE__CONFIG__ROOT: "/blob_storage" + +services: + router: + image: nginx:${NGINX_IMAGE_VERSION} + pull_policy: always + ports: + - "${GOLEM_ROUTER_PORT}:80" + environment: + GOLEM_COMPONENT_MAX_SIZE_ALLOWED: ${GOLEM_ROUTER_COMPONENT_MAX_SIZE_ALLOWED} + GOLEM_REGISTRY_SERVICE_HOST: golem-registry-service + GOLEM_REGISTRY_SERVICE_PORT: ${REGISTRY_SERVICE_HTTP_PORT} + GOLEM_WORKER_SERVICE_HOST: golem-worker-service + GOLEM_WORKER_SERVICE_PORT: ${WORKER_SERVICE_HTTP_PORT} + depends_on: + - golem-worker-service + - golem-shard-manager + - golem-component-compilation-service + - golem-registry-service + volumes: + - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro + + redis: + image: redis:${REDIS_IMAGE_VERSION} + volumes: + - redis_data:/data + + postgres: + image: postgres:${POSTGRES_IMAGE_VERSION} + environment: + POSTGRES_DB: golem_db + POSTGRES_USER: golem_user + POSTGRES_PASSWORD: golem_password + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U golem_user -d golem_db"] + interval: 5s + timeout: 5s + retries: 5 + + # Holds the shard manager's lease state and its leader-election key. Only the shard + # manager talks to it; every other service still uses Postgres. + # + # Single node, which is enough to demonstrate the mode but not to survive losing it: + # a real deployment runs an etcd cluster, since this container is now the shard + # manager's durable state. + etcd: + image: gcr.io/etcd-development/etcd:${ETCD_IMAGE_VERSION} + restart: on-failure + # The image's default entrypoint binds the client port to the container's own + # localhost, so other services could not reach it. Bind 0.0.0.0 and advertise the + # compose service name. + command: + - /usr/local/bin/etcd + - --name=golem-etcd + - --data-dir=/etcd-data + - --listen-client-urls=http://0.0.0.0:2379 + - --advertise-client-urls=http://etcd:2379 + - --listen-peer-urls=http://0.0.0.0:2380 + - --initial-advertise-peer-urls=http://etcd:2380 + - --initial-cluster=golem-etcd=http://etcd:2380 + - --initial-cluster-token=golem-etcd + - --initial-cluster-state=new + - --log-level=info + volumes: + - etcd_data:/etcd-data + healthcheck: + test: ["CMD", "/usr/local/bin/etcdctl", "--endpoints=http://127.0.0.1:2379", "endpoint", "health"] + interval: 5s + timeout: 5s + retries: 5 + + golem-registry-service: + image: golemservices/registry-service:${GOLEM_IMAGES_VERSION} + pull_policy: always + restart: on-failure + environment: + <<: [*blob-storage-env-vars] + GOLEM_ENVIRONMENT: local + RUST_BACKTRACE: 1 + RUST_LOG=info,h2=warn,hyper=warn,tower: warn + + GOLEM__HTTP_PORT: ${REGISTRY_SERVICE_HTTP_PORT} + GOLEM__GRPC__PORT: ${REGISTRY_SERVICE_GRPC_PORT} + + GOLEM__CORS_ORIGIN_REGEX: ${CORS_ORIGIN_REGEX} + GOLEM__INITIAL_ACCOUNTS__ROOT__TOKEN: ${ADMIN_TOKEN} + GOLEM__INITIAL_ACCOUNTS__MARKETING__TOKEN: ${MARKETING_TOKEN} + GOLEM__LOGIN__TYPE: Disabled + + GOLEM__DB__TYPE: Postgres + GOLEM__DB__CONFIG__DATABASE: golem_db + GOLEM__DB__CONFIG__SCHEMA: golem_registry_service + GOLEM__DB__CONFIG__MAX_CONNECTIONS: 10 + GOLEM__DB__CONFIG__HOST: postgres + GOLEM__DB__CONFIG__PORT: 5432 + GOLEM__DB__CONFIG__USERNAME: golem_user + GOLEM__DB__CONFIG__PASSWORD: golem_password + + GOLEM__COMPILATION__CONFIG__HOST: golem-component-compilation-service + GOLEM__COMPILATION__CONFIG__PORT: ${COMPONENT_COMPILATION_SERVICE_GRPC_PORT} + volumes: + - blob_storage:/blob_storage + depends_on: + postgres: + condition: service_healthy + + golem-shard-manager: + image: golemservices/golem-shard-manager:${GOLEM_IMAGES_VERSION} + pull_policy: always + restart: on-failure + environment: + RUST_BACKTRACE: 1 + RUST_LOG=info,h2=warn,hyper=warn,tower: warn + GOLEM__ENVIRONMENT: local + + GOLEM__HTTP_PORT: ${SHARD_MANAGER_HTTP_PORT} + GOLEM__GRPC__PORT: ${SHARD_MANAGER_GRPC_PORT} + + GOLEM__REGISTRY_SERVICE__HOST: golem-registry-service + GOLEM__REGISTRY_SERVICE__PORT: ${REGISTRY_SERVICE_GRPC_PORT} + + # Distributed mode. The shard lease state lives in etcd behind a compare-and-swap, + # and one replica wins an etcd lease campaign and drives all topology decisions. + # The endpoint list must be bracketed: unbracketed, it is read as a single string + # and fails to deserialize. Only http:// endpoints are accepted - the shard + # manager refuses to start on https://, because TLS is not configurable. + GOLEM__PERSISTENCE__TYPE: Etcd + GOLEM__PERSISTENCE__CONFIG__ENDPOINTS: '["http://etcd:2379"]' + # connect_timeout (10s), request_timeout (5s), leader_lease_ttl (10s) and + # compaction_retention_revisions (1000) keep their defaults here. + depends_on: + etcd: + condition: service_healthy + golem-registry-service: + condition: service_started + + golem-worker-service: + image: golemservices/golem-worker-service:${GOLEM_IMAGES_VERSION} + pull_policy: always + restart: on-failure + environment: + <<: [*blob-storage-env-vars] + RUST_BACKTRACE: 1 + RUST_LOG=info,h2=warn,hyper=warn,tower: warn + GOLEM__ENVIRONMENT: local + + GOLEM__CUSTOM_REQUEST_PORT: ${WORKER_SERVICE_CUSTOM_REQUEST_PORT} + GOLEM__MCP_PORT: ${WORKER_SERVICE_MCP_PORT} + GOLEM__PORT: ${WORKER_SERVICE_HTTP_PORT} + GOLEM__GRPC__PORT: ${WORKER_SERVICE_GRPC_PORT} + + GOLEM__GATEWAY_SESSION_STORAGE__TYPE: "Redis" + GOLEM__GATEWAY_SESSION_STORAGE__CONFIG__HOST: redis + GOLEM__GATEWAY_SESSION_STORAGE__CONFIG__PORT: 6379 + + GOLEM__REGISTRY_SERVICE__HOST: golem-registry-service + GOLEM__REGISTRY_SERVICE__PORT: ${REGISTRY_SERVICE_GRPC_PORT} + + GOLEM__SHARD_MANAGER__HOST: golem-shard-manager + GOLEM__SHARD_MANAGER__PORT: ${SHARD_MANAGER_GRPC_PORT} + volumes: + - blob_storage:/blob_storage + ports: + - "${WORKER_SERVICE_CUSTOM_REQUEST_PORT}:${WORKER_SERVICE_CUSTOM_REQUEST_PORT}" + - "${WORKER_SERVICE_MCP_PORT}:${WORKER_SERVICE_MCP_PORT}" + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_started + golem-worker-executor: + condition: service_started + golem-shard-manager: + condition: service_started + golem-registry-service: + condition: service_started + + golem-component-compilation-service: + image: golemservices/golem-component-compilation-service:${GOLEM_IMAGES_VERSION} + pull_policy: always + restart: on-failure + environment: + <<: [*blob-storage-env-vars] + RUST_BACKTRACE: 1 + RUST_LOG=info,h2=warn,hyper=warn,tower: warn + + GOLEM__HTTP_PORT: ${COMPONENT_COMPILATION_SERVICE_HTTP_PORT} + GOLEM__GRPC__PORT: ${COMPONENT_COMPILATION_SERVICE_GRPC_PORT} + + GOLEM__REGISTRY_SERVICE__TYPE: "Static" + GOLEM__REGISTRY_SERVICE__CONFIG__HOST: golem-registry-service + GOLEM__REGISTRY_SERVICE__CONFIG__PORT: ${REGISTRY_SERVICE_GRPC_PORT} + volumes: + - blob_storage:/blob_storage + depends_on: + - golem-registry-service + + golem-worker-executor: + image: golemservices/golem-worker-executor:${GOLEM_IMAGES_VERSION} + pull_policy: always + restart: on-failure + environment: + <<: [*blob-storage-env-vars] + WASMTIME_BACKTRACE_DETAILS: 1 + RUST_BACKTRACE: 1 + RUST_LOG: info + + GOLEM__HTTP_PORT: ${WORKER_EXECUTOR_HTTP_PORT} + GOLEM__GRPC__PORT: ${WORKER_EXECUTOR_GRPC_PORT} + + GOLEM__KEY_VALUE_STORAGE__TYPE: "Postgres" + GOLEM__KEY_VALUE_STORAGE__CONFIG__DATABASE: golem_db + GOLEM__KEY_VALUE_STORAGE__CONFIG__MAX_CONNECTIONS: 10 + GOLEM__KEY_VALUE_STORAGE__CONFIG__HOST: postgres + GOLEM__KEY_VALUE_STORAGE__CONFIG__PORT: 5432 + GOLEM__KEY_VALUE_STORAGE__CONFIG__USERNAME: golem_user + GOLEM__KEY_VALUE_STORAGE__CONFIG__PASSWORD: golem_password + GOLEM__KEY_VALUE_STORAGE__CONFIG__SCHEMA: golem_worker_executor + + GOLEM__INDEXED_STORAGE__TYPE: "Postgres" + GOLEM__INDEXED_STORAGE__CONFIG__DATABASE: golem_db + GOLEM__INDEXED_STORAGE__CONFIG__MAX_CONNECTIONS: 10 + GOLEM__INDEXED_STORAGE__CONFIG__HOST: postgres + GOLEM__INDEXED_STORAGE__CONFIG__PORT: 5432 + GOLEM__INDEXED_STORAGE__CONFIG__USERNAME: golem_user + GOLEM__INDEXED_STORAGE__CONFIG__PASSWORD: golem_password + GOLEM__INDEXED_STORAGE__CONFIG__SCHEMA: golem_worker_executor_indexed + + GOLEM__SCHEDULER_STORAGE__TYPE: "Postgres" + GOLEM__SCHEDULER_STORAGE__CONFIG__DATABASE: golem_db + GOLEM__SCHEDULER_STORAGE__CONFIG__MAX_CONNECTIONS: 10 + GOLEM__SCHEDULER_STORAGE__CONFIG__HOST: postgres + GOLEM__SCHEDULER_STORAGE__CONFIG__PORT: 5432 + GOLEM__SCHEDULER_STORAGE__CONFIG__USERNAME: golem_user + GOLEM__SCHEDULER_STORAGE__CONFIG__PASSWORD: golem_password + GOLEM__SCHEDULER_STORAGE__CONFIG__SCHEMA: golem_worker_executor_scheduler + + GOLEM__REGISTRY_SERVICE__HOST: golem-registry-service + GOLEM__REGISTRY_SERVICE__PORT: ${REGISTRY_SERVICE_GRPC_PORT} + + GOLEM__SHARD_MANAGER__TYPE: "Grpc" + GOLEM__SHARD_MANAGER__HOST: golem-shard-manager + GOLEM__SHARD_MANAGER__CONFIG__PORT: ${SHARD_MANAGER_GRPC_PORT} + GOLEM__SHARD_MANAGER__CONFIG__RETRIES__MAX_ATTEMPTS: 5 + GOLEM__SHARD_MANAGER__CONFIG__RETRIES__MIN_DELAY: "100ms" + GOLEM__SHARD_MANAGER__CONFIG__RETRIES__MAX_DELAY: "2s" + GOLEM__SHARD_MANAGER__CONFIG__RETRIES__MULTIPLIER: 2 + + GOLEM__PUBLIC_WORKER_API__HOST: golem-worker-service + GOLEM__PUBLIC_WORKER_API__PORT: ${WORKER_SERVICE_GRPC_PORT} + + GOLEM__COMPILED_COMPONENT_SERVICE__TYPE: "Enabled" + ulimits: + nofile: + soft: 65536 + hard: 65536 + volumes: + - blob_storage:/blob_storage + depends_on: + postgres: + condition: service_healthy + golem-shard-manager: + condition: service_started + golem-registry-service: + condition: service_started + +volumes: + redis_data: + driver: local + etcd_data: + driver: local + component_store: + driver: local + postgres_data: + driver: local + blob_storage: + driver: local diff --git a/docker-examples/distributed-etcd/nginx.conf.template b/docker-examples/distributed-etcd/nginx.conf.template new file mode 100644 index 0000000000..fbec6348bb --- /dev/null +++ b/docker-examples/distributed-etcd/nginx.conf.template @@ -0,0 +1,55 @@ +client_max_body_size ${GOLEM_COMPONENT_MAX_SIZE_ALLOWED}; # Increase this especially if your component size is higher than this + +# For docker we need this for service discovery in docker network +resolver 127.0.0.11; + +upstream registry-service { + server ${GOLEM_REGISTRY_SERVICE_HOST}:${GOLEM_REGISTRY_SERVICE_PORT} fail_timeout=0 max_fails=0; +} + +upstream worker-service { + server ${GOLEM_WORKER_SERVICE_HOST}:${GOLEM_WORKER_SERVICE_PORT} fail_timeout=0 max_fails=0; +} + +server { + listen 80; + server_name localhost; + + # Common proxy headers + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + location ~ /v1/components/[^/]+/workers/[^/]+/connect$ { + proxy_pass http://worker-service; + proxy_http_version 1.1; + proxy_set_header Upgrade "websocket"; + proxy_set_header Connection "upgrade"; + + proxy_read_timeout 3600s; + proxy_send_timeout 3600s; + } + + location ~ /v1/components/[^/]+/workers(.*)$ { + proxy_pass http://worker-service; + } + + location = /v1/agents/invoke-agent-session { + proxy_pass http://worker-service; + proxy_http_version 1.1; + proxy_set_header Upgrade "websocket"; + proxy_set_header Connection "upgrade"; + + proxy_read_timeout 3600s; + proxy_send_timeout 3600s; + } + + location /v1/agents { + proxy_pass http://worker-service; + } + + location / { + proxy_pass http://registry-service; + } +} diff --git a/docs/src/content/next/deploy.mdx b/docs/src/content/next/deploy.mdx index 16cb739a19..d923b21446 100644 --- a/docs/src/content/next/deploy.mdx +++ b/docs/src/content/next/deploy.mdx @@ -38,7 +38,7 @@ See also: [configuration](https://github.com/golemcloud/golem/blob/main/golem-wo ### Worker Executor -[golem-worker-executor](https://github.com/golemcloud/golem/tree/main/golem-worker-executor) is responsible for running the [agents](/next/concepts/agents) that belong to assigned shards. The service uses Redis and Blob storage as data storage. +[golem-worker-executor](https://github.com/golemcloud/golem/tree/main/golem-worker-executor) is responsible for running the [agents](/next/concepts/agents) that belong to assigned shards. The service keeps each agent's oplog in **indexed storage** - PostgreSQL, SQLite or Redis - and uses key-value and blob storage alongside it. See also: [configuration](https://github.com/golemcloud/golem/blob/main/golem-worker-executor/config/worker-executor.toml), [environment variables](https://github.com/golemcloud/golem/blob/main/golem-worker-executor/config/worker-executor.sample.env), [docker image](https://hub.docker.com/r/golemservices/golem-worker-executor) @@ -49,7 +49,15 @@ See also: [configuration](https://github.com/golemcloud/golem/blob/main/golem-wo - **Local mode** (`persistence.type = "Postgres"` or `"Sqlite"`) stores the shard state and the quota state in one SQL database. There is no leader election, so **exactly one instance may run**. - **Distributed mode** (`persistence.type = "Etcd"`) stores the shard state in etcd. Any number of replicas may run: one wins an etcd lease campaign and drives all topology decisions while the rest stand by. Quota enforcement is not available in this mode yet: the quota state has no store outside the SQL database, so every quota lease request fails. -Every worker executor holds a **lease** on its shard assignment and renews it with the shard manager. A lease lasts `shard_lease_duration` (1m by default) and an executor renews at a third of the time it has left, so roughly every 20s. The shard manager refuses to start with a lease shorter than 30s and warns about one shorter than 1m: the deadline it allows each renewal call never drops below 10s, so a shorter lease cannot fit the three renewal attempts a lease is meant to get. Each renewal carries the epoch of every shard the executor holds; a shard's epoch is its ownership generation, advanced only when the shard moves to another executor and never by a renewal. A renewal whose claim does not match the shard manager's view is renewed all the same, and the response carries the shard manager's set, which the executor adopts exactly as it would an assignment push: pushes deliver a change at once, and renewals guarantee it arrives within a third of the lease even if the push was lost. The shard manager records a shard's new owner before it tells anybody about it, so a rebalance whose write is refused changes nothing at all, and every delivery carries the revision of the state it was read from. An executor ignores a delivery older than one it has already applied, so a push and a renewal that cross on the network cannot leave the older set in place. An executor that stops renewing loses its lease, and the shard manager reclaims and redistributes those shards on its next pass, which runs at least every third of `shard_lease_duration` even in a cluster where nothing else is happening; a crashed or unreachable executor's shards are therefore re-homed within about one lease duration plus one pass. The lease reaches the executor as the time remaining on it rather than as an absolute time, so the two machines' clocks are never compared and skew between them can neither lengthen nor shorten it. The executor anchors that time to the moment it sent the request the lease answers, so time the reply spends in flight comes off its copy of the lease and is never added to it, and its copy lapses no later than the shard manager's. An assignment push carries the shard set and its revision only: it never extends a lease. The lease is the one place the two machines' clocks must agree on a rate rather than on a value: an executor whose clock runs slow relative to the shard manager's keeps admitting for that fraction of a lease past the shard manager reclaiming it, which is milliseconds under ordinary `ntp` or `chrony` discipline and seconds while such a daemon is slewing off a large offset, so an executor should not be started until its clock is synchronised. An executor whose own copy of the lease has expired stops **admitting** new work for its shards and reports that sharding is not ready, which a worker service retries after refreshing its routing table - invocations already running are not interrupted. A graceful stop - including the `SIGTERM` an orchestrator sends to stop a pod - deregisters the executor instead of waiting for its lease to expire, so its shards move on the next pass. A shard manager restart or failover re-grants the lease of every executor that still answers the startup health check, so a restart never evicts a healthy cluster: only executors that fail that check are removed. An executor whose own copy of the lease lapsed during the outage resumes admitting work on its next successful renewal rather than on the push that follows the re-grant, and neither a re-grant nor a renewal ever moves an executor's deadline earlier, so a `shard_lease_duration` reduced across a restart takes effect only once the new length outruns the deadline each executor was last told. +Every worker executor holds a **lease** on its shard assignment and renews it with the shard manager. A lease lasts `shard_lease_duration` (1m by default) and an executor renews at a third of the time it has left, so roughly every 20s. The shard manager refuses to start with a lease shorter than 30s and warns about one shorter than 1m: the deadline it allows each renewal call never drops below 10s, so a shorter lease cannot fit the three renewal attempts a lease is meant to get. Each renewal carries the epoch of every shard the executor holds; a shard's epoch is its ownership generation, advanced when the shard moves to another executor and otherwise only to repair a shard manager state that lost history, as described under oplog fencing below. A renewal whose held epochs do not match the shard manager's view is renewed all the same, and the response carries the shard manager's set, which the executor adopts exactly as it would an assignment push: pushes deliver a change at once, and renewals guarantee it arrives within a third of the lease even if the push was lost. The shard manager records a shard's new owner before it tells anybody about it, so a rebalance whose write is refused changes nothing at all, and every delivery carries the revision of the state it was read from. An executor ignores a delivery older than one it has already applied, so a push and a renewal that cross on the network cannot leave the older set in place. Revisions are only comparable within one shard manager process, so every delivery also names the process that sent it: an executor follows the process that answered its latest registration or renewal and starts its revisions over when that changes - which is how a shard manager that failed over, or came back on a wiped or restored store, gets its sets applied although its revisions start lower - and it ignores a push from any other process, renewing at once to hear from the one in charge. An executor that stops renewing loses its lease, and the shard manager reclaims and redistributes those shards on its next pass, which runs at least every third of `shard_lease_duration` even in a cluster where nothing else is happening; a crashed or unreachable executor's shards are therefore re-homed within about one lease duration plus one pass. The lease reaches the executor as the time remaining on it rather than as an absolute time, so the two machines' clocks are never compared and skew between them can neither lengthen nor shorten it. The executor anchors that time to the moment it sent the request the lease answers, so time the reply spends in flight comes off its copy of the lease and is never added to it, and its copy lapses no later than the shard manager's. An assignment push carries the shard set and its revision only: it never extends a lease. The lease is the one place the two machines' clocks must agree on a rate rather than on a value: an executor whose clock runs slow relative to the shard manager's keeps admitting for that fraction of a lease past the shard manager reclaiming it, which is milliseconds under ordinary `ntp` or `chrony` discipline and seconds while such a daemon is slewing off a large offset, so an executor should not be started until its clock is synchronised. An executor whose own copy of the lease has expired stops **admitting** new work for its shards and reports that sharding is not ready, which a worker service retries after refreshing its routing table - invocations already running are not interrupted. A graceful stop - including the `SIGTERM` an orchestrator sends to stop a pod - deregisters the executor instead of waiting for its lease to expire, so its shards move on the next pass. A shard manager restart or failover re-grants the lease of every executor that still answers the startup health check, so a restart never evicts a healthy cluster: only executors that fail that check are removed. An executor whose own copy of the lease lapsed during the outage resumes admitting work on its next successful renewal rather than on the push that follows the re-grant, and neither a re-grant nor a renewal ever moves an executor's deadline earlier, so a `shard_lease_duration` reduced across a restart takes effect only once the new length outruns the deadline each executor was last told. + +The lease bounds how long a lost shard keeps being served, but it cannot stop an executor that has already lost one from finishing a write it had started. That is what **oplog fencing** is for. Each agent's oplog records the shard epoch allowed to write it, and every batch of entries is checked against that record atomically with the insert - in the same database transaction, or on Redis in the same script - so a refused write leaves nothing behind. An executor whose epoch is behind is turned away at the storage rather than discovering the problem later: that one agent stops there, and the worker service resumes it on the shard's new owner. The record only ever moves forward - an executor re-granted the shard at a higher epoch takes over, and one holding a stale epoch cannot claim it back - and an absent record fences an open oplog too, because the record is written before an oplog's first entry and removed before its last. That protection covers an oplog whose record exists. Deleting an agent removes its record and forgets the epoch with it: an executor that still has the oplog open is refused, but an oplog without a record - a new agent, a deleted agent created again, or one last written before this release - is claimed by whichever epoch opens it first, so for those only the lease stops an executor that has lost the shard. Ephemeral agents are not fenced: their oplogs are never replayed, so a duplicate there is a duplicated observability record rather than duplicated state. Fencing covers the oplog and nothing else. An executor that has lost a shard can still write that agent's key-value records, its status and its blob payloads, none of which carry an epoch: a stale status blob can overwrite a newer one until the agent's next status change on its new owner, while the oplog - the only record replay reads - stays correct. Nor does the fence undo what has already left the machine: an outgoing call an agent made before its first refused write has happened, and the shard's new owner, having no record of it, makes it again, which is the same at-least-once exposure a crash between the call and its oplog entry already has. A call the executor has not started yet is refused once the fence has latched, so the window is the call in flight rather than every call after the shard moved. + +Fencing is enforced by every indexed storage backend, for deleting an agent as much as for writing to it: a deletion that finds another executor has taken the agent over removes nothing, and fails with the answer that sends it to the new owner. A shard revoked with no new owner yet does not stop a deletion already under way. The shipped configuration defaults `indexed_storage` to a SQLite file of its own, which serves a single executor; executors that share shards need storage they can all reach, which means PostgreSQL or Redis, because a SQLite file is written by one process. Redis is only as durable as the last write the server kept: a failover that loses the most recent writes can lose a raised epoch with them, and the previous owner is then accepted again until the record is next raised, so prefer PostgreSQL or SQLite where that window matters. Redis Cluster is not supported. `KVStoreSqlite` and `KVStoreMultiSqlite` derive their location from the key-value storage, so they require `key_value_storage.type` to be `Sqlite` or `MultiSqlite` respectively; any other combination fails at startup. + +A shard epoch only means something against the shard manager state that minted it. A state that is wiped, replaced or restored from a backup mints epochs below the ones the oplogs already record, and every write at those would be refused, so the shard manager repairs its record from what the executors tell it, without an operator. An executor that kept running sends the epochs it holds on its next renewal, or - when the state no longer lists the executor and refuses that renewal - on the re-registration that follows. The shard manager raises its record to them on the shards that executor still owns and on unassigned ones. It does not take an executor's word about a shard it has meanwhile given to another one: that owner's writes are refused by the oplogs the earlier executor wrote, it reports the epoch they record on its next renewal, and the shard manager gives it a new epoch one past that, so two executors never write a shard's oplogs at the same epoch. Executors are otherwise trusted peers: an epoch reported from a refused write is believed as sent, because it names what the storage holds. The one exception is an epoch more than 2^32 above the one recorded: epochs climb by one per owner change, so no restore is ever that far behind, and such a value is ignored and logged rather than allowed to use up the shard's epochs. When every executor restarted as well, none of them holds the old epochs, and the oplogs supply them instead: an executor refused a write reports the epoch the oplog recorded on its next renewal, and the shard manager gives whichever executor owns that shard a new epoch one past it. Either way the affected agents resume at the new epoch within about one renewal interval, while the worker service retries their requests. A wiped state also mints from the beginning again, so it can hand an executor the very epoch another one is still writing at. The record names the writing process as well as the epoch, and a second process arriving at an epoch the record already holds is refused rather than sharing it; that refusal is reported on the next renewal like any other, and the shard manager mints its owner one past the collision, so the takeover happens at a generation nobody shares. An oplog nobody opens keeps its old record until it is opened, and that first refused open repairs it the same way. To clear the old records up front instead, or if an agent is still refused after several renewals, stop every executor, run `DELETE FROM indexed_key_epoch` on the indexed storage - for `MultiSqlite` and `KVStoreMultiSqlite`, on every `*-oplog-*.db` file under the root directory; on Redis, delete the keys matching `*oplog-epoch:*` instead - and start the executors again; each oplog records its owner's epoch when it is next opened. Run the delete only while no executor is running, because an oplog without a record accepts whichever executor opens it first. + +Upgrade the shard manager and the worker services before the worker executors. A shard manager from an earlier release names no process on its deliveries, and an executor then orders them by revision alone, as before. An upgraded executor reports a request that reached it without its shard as a routing miss, which only an upgraded worker service retries, and it sends the epochs the repair above relies on in fields an earlier shard manager ignores. Once any executor runs this release, only roll forward: it adds the table that records these epochs to the indexed storage, and an executor from an earlier release refuses to start against indexed storage that has applied that migration, so an older executor restarted mid-rollout, or a rollback of the executor image, fails at startup. During a rolling upgrade the fence is complete only once every executor runs the new code and each agent has been opened once under it. Distributed mode talks to etcd over **plaintext HTTP** and without authentication. Every entry in `persistence.config.endpoints` must start with `http://`; a replica configured with anything else refuses to start. etcd therefore has to be reachable unauthenticated, either on a trusted network or through a TLS-terminating proxy running alongside the shard manager, with the endpoints pointing at that proxy. diff --git a/docs/src/content/next/invoke/stream-session-public-protocol-v1.mdx b/docs/src/content/next/invoke/stream-session-public-protocol-v1.mdx index d45a68afde..b3ca0b0449 100644 --- a/docs/src/content/next/invoke/stream-session-public-protocol-v1.mdx +++ b/docs/src/content/next/invoke/stream-session-public-protocol-v1.mdx @@ -731,6 +731,7 @@ The v1 code set is frozen: | `resource-exhausted` | a frozen protocol or configured account/session resource limit was exceeded | | `producer-error` | output producer terminated one stream with an application error | | `invocation-failed` | invocation failed after acceptance | +| `routing-miss` | the agent's shard is moving between executors; `retryable` is `true` and a new attempt reaches its owner | | `internal-error` | safe, non-sensitive internal failure | Handshake failures use the existing public HTTP error envelope. Before diff --git a/docs/src/content/next/operate/persistence.mdx b/docs/src/content/next/operate/persistence.mdx index 3c4aae5c4b..edc8fb7908 100644 --- a/docs/src/content/next/operate/persistence.mdx +++ b/docs/src/content/next/operate/persistence.mdx @@ -18,7 +18,7 @@ Currently Golem provides the following implementations, configurable through the | ------------------ | -------------------------- | | Blob storage | S3, file system, in-memory | | Key-values storage | Redis, in-memory | -| Indexed storage | Redis (streams), in-memory | +| Indexed storage | PostgreSQL, SQLite, Redis (streams), in-memory | ## Compilation cache diff --git a/golem-api-grpc/proto/golem/shardmanager/v1/shard_manager_service.proto b/golem-api-grpc/proto/golem/shardmanager/v1/shard_manager_service.proto index 486bc4cc4d..80a51de447 100644 --- a/golem-api-grpc/proto/golem/shardmanager/v1/shard_manager_service.proto +++ b/golem-api-grpc/proto/golem/shardmanager/v1/shard_manager_service.proto @@ -50,6 +50,18 @@ message RegisterRequest { // the same address refreshes the existing lease rather than creating a second // one. An empty or non-UUID value is rejected before any state is touched. string executor_id = 3; + // The shards, with their epochs, this executor held under an earlier + // executor_id that the manager answered ShardLeaseNotFound for; empty on a + // process's first registration. Evidence only: it never assigns a shard. An + // epoch ahead of the manager's record means the manager's state lost history + // - a wiped or replaced store no longer lists the earlier executor_id, so the + // renewal that would have repaired it was refused - and the manager raises + // its record so that the epochs it mints next clear the oplog rows written at + // the old ones. Believed only for shards that are this executor's own or + // unassigned: on a shard another executor owns it moves nothing, and that + // owner's fenced_shard_epochs repair it instead. A malformed entry is + // rejected before any state is touched. + repeated golem.shardmanager.ShardEpochEntry previous_shard_epochs = 4; } message RegisterResponse { @@ -78,6 +90,16 @@ message RegisterSuccess { // least the last one it applied, so two deliveries that cross on the network // cannot leave the older set in place. uint64 revision = 4; + // Identifies the shard manager process that sent this, minted when it + // started. A revision orders the deliveries of one process only: a manager + // that failed over, or came back on a wiped or restored store, counts from a + // state the executor's last applied revision says nothing about. The executor + // follows the process that answered its latest request - a different id here + // or on ShardLease starts its revisions over - and ignores an AssignShards or + // RevokeShards from any other, renewing at once to hear from the one in + // charge. Empty from a manager that does not send one, and then deliveries + // are ordered by revision alone. + string incarnation_id = 5; } // An executor's shard lease: the complete set of shards it owns with the epoch @@ -89,18 +111,29 @@ message ShardLease { google.protobuf.Duration lease_ttl = 2; // See RegisterSuccess.revision. uint64 revision = 3; + // See RegisterSuccess.incarnation_id. + string incarnation_id = 4; } // Extends the executor's shard lease before it expires. shard_epochs is the // set the executor believes it holds. It is what the executor is proving it is -// alive for, not a condition of the renewal: a claim that does not match the +// alive for, not a condition of the renewal: a set that does not match the // manager's view is renewed all the same, and the response carries the set the // manager holds for this executor, which the executor adopts. That makes the // renewal a guaranteed second delivery path for a push that was lost, and it -// means a renewal never advances an epoch. +// means a renewal never advances an epoch against a record that kept its +// history. message RenewShardLeaseRequest { string executor_id = 1; repeated golem.shardmanager.ShardEpochEntry shard_epochs = 2; + // Epochs recorded on oplogs this executor was refused writes to, keyed by + // the shard each agent routes to. Evidence only: never an assignment. Above + // the manager's record they mean its state lost history, and every owner of + // the shard, this executor included, is minted one past them; at or below + // the record they move nothing. Applied before this request's shard_epochs. + // Reported on every renewal until one is granted. A malformed entry is + // rejected before any state is touched. + repeated golem.shardmanager.ShardEpochEntry fenced_shard_epochs = 3; } message RenewShardLeaseResponse { diff --git a/golem-api-grpc/proto/golem/workerexecutor/v1/worker_executor.proto b/golem-api-grpc/proto/golem/workerexecutor/v1/worker_executor.proto index c9fcb26ca3..363ac15f93 100644 --- a/golem-api-grpc/proto/golem/workerexecutor/v1/worker_executor.proto +++ b/golem-api-grpc/proto/golem/workerexecutor/v1/worker_executor.proto @@ -183,6 +183,8 @@ message RevokeShardsRequest { // The revision of the shard manager's persisted state the shards were moved // away in; see golem.shardmanager.v1.RegisterSuccess.revision. uint64 revision = 2; + // See golem.shardmanager.v1.RegisterSuccess.incarnation_id. + string incarnation_id = 3; } message RevokeShardsResponse { @@ -206,6 +208,8 @@ message AssignShardsRequest { // The revision of the shard manager's persisted state this set was read // from; see golem.shardmanager.v1.RegisterSuccess.revision. uint64 revision = 4; + // See golem.shardmanager.v1.RegisterSuccess.incarnation_id. + string incarnation_id = 5; } message AssignShardsResponse { diff --git a/golem-common/src/model/invocation_session_public.rs b/golem-common/src/model/invocation_session_public.rs index 1129c6cd8b..61fe5311c7 100644 --- a/golem-common/src/model/invocation_session_public.rs +++ b/golem-common/src/model/invocation_session_public.rs @@ -81,6 +81,11 @@ pub enum PublicErrorCode { ResourceExhausted, ProducerError, InvocationFailed, + /// The executor that answered does not own this agent's shard right now: the assignment is + /// moving, or one of its writes was fenced by the shard's new owner. Nothing is wrong with the + /// request, and a client that retries reaches the new owner - which is why this is not + /// `InternalError`. + RoutingMiss, InternalError, } @@ -112,6 +117,7 @@ impl PublicErrorCode { Self::ResourceExhausted => "resource-exhausted", Self::ProducerError => "producer-error", Self::InvocationFailed => "invocation-failed", + Self::RoutingMiss => "routing-miss", Self::InternalError => "internal-error", } } @@ -166,6 +172,7 @@ const ALL_ERROR_CODES: &[PublicErrorCode] = &[ PublicErrorCode::ResourceExhausted, PublicErrorCode::ProducerError, PublicErrorCode::InvocationFailed, + PublicErrorCode::RoutingMiss, PublicErrorCode::InternalError, ]; diff --git a/golem-common/src/model/mod.rs b/golem-common/src/model/mod.rs index 714ff172ba..b73c9358b0 100644 --- a/golem-common/src/model/mod.rs +++ b/golem-common/src/model/mod.rs @@ -562,21 +562,63 @@ impl Display for ShardEpoch { } } -/// The revision of the shard manager's persisted state that a delivered shard -/// set was read from. Every delivery carries one - a registration, a push, a -/// renewal - and an executor applies a delivery only if its revision is at -/// least the last one it applied, so two deliveries that cross on the network -/// cannot leave the older set in place. `0` is "nothing applied yet". The -/// executor's own newtype; it never imports the shard manager's. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct ShardLeaseRevision(pub u64); +/// Where a delivered shard set sits in the shard manager's order: the manager +/// process that sent it, and the revision of the persisted state the set was +/// read from. Every delivery carries one - a registration, a push, a renewal - +/// and an executor applies a delivery only if its number is at least the last +/// one it applied, so two deliveries that cross on the network cannot leave the +/// older set in place. `0` is "nothing applied yet". +/// +/// The numbers of two manager processes are not comparable: one that failed +/// over, or came back on a wiped or restored store, counts from a state this +/// executor's last applied number says nothing about. So this has no `Ord`, and +/// [`ShardAssignment`] is the one place the two halves are read together. +/// `incarnation` is `None` from a manager that does not send one. The +/// executor's own type; it never imports the shard manager's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct ShardLeaseRevision { + pub incarnation: Option, + pub number: u64, +} + +impl ShardLeaseRevision { + /// A revision from a manager that names no incarnation. + pub const fn of(number: u64) -> Self { + Self { + incarnation: None, + number, + } + } + + /// Off the wire. An id that is empty, or not a UUID, reads as no + /// incarnation, and the delivery is then ordered by its number alone. + pub fn from_wire(incarnation_id: &str, number: u64) -> Self { + Self { + incarnation: Uuid::parse_str(incarnation_id).ok(), + number, + } + } +} impl Display for ShardLeaseRevision { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - write!(f, "{}", self.0) + match self.incarnation { + Some(incarnation) => write!(f, "{}@{incarnation}", self.number), + None => write!(f, "{}", self.number), + } } } +/// Which way a delivery reached this executor, which decides what a change of +/// manager process means - see [`ShardAssignment::apply`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ShardDeliveryPath { + /// The answer to a registration or a renewal this executor sent. + Reply, + /// An `AssignShards` or `RevokeShards` the manager sent on its own. + Push, +} + /// What applying a delivered shard set did. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ShardDeliveryOutcome { @@ -589,6 +631,13 @@ pub enum ShardDeliveryOutcome { delivered: ShardLeaseRevision, applied: ShardLeaseRevision, }, + /// A push from a manager process other than the one this executor follows, + /// so ignored whole. The executor owes a renewal: its answer names the + /// process in charge and carries that process's set. + FromAnotherManager { + delivered: ShardLeaseRevision, + applied: ShardLeaseRevision, + }, } /// The shards this executor currently holds, with the epoch each was granted @@ -607,8 +656,9 @@ pub struct ShardAssignment { /// A push carries no lease. `None` means the lease never expires /// (single-shard mode and the pre-registration placeholder). pub expires_at: Option, - /// The revision of the delivery this set came from. A delivery older than - /// this is ignored; see [`ShardLeaseRevision`]. + /// The revision of the delivery this set came from, and the shard manager + /// process this executor follows. A delivery older than this, or pushed + /// by another process, is ignored; see [`ShardLeaseRevision`]. pub revision: ShardLeaseRevision, } @@ -655,9 +705,9 @@ impl ShardAssignment { self.shard_epochs.get(shard_id).copied() } - /// The claim sent on a lease renewal: exactly the set last received, in a + /// The held epochs sent on a lease renewal: exactly the set last received, in a /// deterministic order. - pub fn claim(&self) -> BTreeMap { + pub fn held_epochs(&self) -> BTreeMap { self.shard_epochs .iter() .map(|(shard_id, epoch)| (*shard_id, *epoch)) @@ -676,7 +726,12 @@ impl ShardAssignment { shard_epochs: &HashMap, revision: ShardLeaseRevision, ) -> ShardDeliveryOutcome { - self.apply(Some(number_of_shards), shard_epochs, revision) + self.apply( + Some(number_of_shards), + shard_epochs, + revision, + ShardDeliveryPath::Push, + ) } /// A grant: the answer to this executor's own registration @@ -692,7 +747,7 @@ impl ShardAssignment { /// set goes through [`Self::apply`]'s revision gate like every other /// delivery: the revision orders sets and the request time orders leases, /// and the two are independent. Normally the set is exactly what was - /// claimed, because a renewal never advances an epoch; when it is not, the + /// held, because a renewal never advances an epoch; when it is not, the /// manager is correcting a push this executor never received, and the /// caller sweeps and recovers agents exactly as it would for a push. /// @@ -709,7 +764,12 @@ impl ShardAssignment { revision: ShardLeaseRevision, ) -> ShardDeliveryOutcome { self.expires_at = Some(expires_at); - self.apply(number_of_shards, shard_epochs, revision) + self.apply( + number_of_shards, + shard_epochs, + revision, + ShardDeliveryPath::Reply, + ) } /// The one place any delivery's set is applied, including @@ -723,13 +783,39 @@ impl ShardAssignment { /// from the same persisted state and carry the same set, so they apply /// harmlessly. The lease is not this function's business: only /// [`Self::adopt_grant`] moves it, and it does so before coming here. + /// + /// That order holds within one manager process. A delivery from another + /// one is told apart by how it came. A reply answers a request this + /// executor has just made, so its sender is the manager in charge: the + /// executor follows it from here on and its numbers start over, which is + /// what lets a manager that lost its history deliver a set at all. A push + /// proves nothing of the kind - a deposed manager can still be sending - + /// so it is ignored, and the caller renews to hear from the one in charge. + /// Starting the numbers over cannot let an old manager's delayed push + /// back in, because that push no longer names the process followed. fn apply( &mut self, number_of_shards: Option, shard_epochs: &HashMap, revision: ShardLeaseRevision, + path: ShardDeliveryPath, ) -> ShardDeliveryOutcome { - if revision < self.revision { + let from_another_manager = matches!( + (revision.incarnation, self.revision.incarnation), + (Some(delivered), Some(followed)) if delivered != followed + ); + if from_another_manager { + match path { + ShardDeliveryPath::Reply => self.revision = ShardLeaseRevision::default(), + ShardDeliveryPath::Push => { + return ShardDeliveryOutcome::FromAnotherManager { + delivered: revision, + applied: self.revision, + }; + } + } + } + if revision.number < self.revision.number { return ShardDeliveryOutcome::Stale { delivered: revision, applied: self.revision, @@ -740,7 +826,16 @@ impl ShardAssignment { self.number_of_shards = number_of_shards; } self.shard_epochs = shard_epochs.clone(); - self.revision = revision; + // Only a reply names a manager to follow, and one that names none does not make this + // executor forget the one it follows. + let followed = self.revision.incarnation; + self.revision = ShardLeaseRevision { + incarnation: match path { + ShardDeliveryPath::Reply => revision.incarnation.or(followed), + ShardDeliveryPath::Push => followed, + }, + number: revision.number, + }; ShardDeliveryOutcome::Applied { set_changed } } @@ -760,7 +855,7 @@ impl ShardAssignment { ) -> ShardDeliveryOutcome { let mut remaining = self.shard_epochs.clone(); remaining.retain(|shard_id, _| !shard_ids.contains(shard_id)); - self.apply(None, &remaining, revision) + self.apply(None, &remaining, revision, ShardDeliveryPath::Push) } /// Drops every shard, keeping `number_of_shards`, and leaves the lease @@ -2931,6 +3026,7 @@ mod shard_assignment_tests { use std::collections::HashSet; use std::time::{Duration, Instant}; use test_r::test; + use uuid::Uuid; test_r::enable!(); @@ -2950,13 +3046,13 @@ mod shard_assignment_tests { fn set_shards_replaces_the_set_rather_than_merging_into_it() { let mut assignment = ShardAssignment::unexpiring(8, [ShardId::new(0), ShardId::new(1)]); - let outcome = assignment.set_shards(8, &epochs([(1, 4)]), ShardLeaseRevision(1)); + let outcome = assignment.set_shards(8, &epochs([(1, 4)]), ShardLeaseRevision::of(1)); assert_eq!(outcome, ShardDeliveryOutcome::Applied { set_changed: true }); assert!(!assignment.contains(&ShardId::new(0))); assert_eq!(assignment.epoch_of(&ShardId::new(1)), Some(ShardEpoch(4))); assert_eq!(assignment.len(), 1); - assert_eq!(assignment.revision, ShardLeaseRevision(1)); + assert_eq!(assignment.revision, ShardLeaseRevision::of(1)); } /// Two deliveries can cross on the network. A renewal reply read from an @@ -2968,17 +3064,17 @@ mod shard_assignment_tests { #[test] fn a_stale_grant_keeps_the_set_but_still_moves_the_lease_clock() { let mut assignment = ShardAssignment::unexpiring(8, [ShardId::new(0)]); - assignment.set_shards(8, &epochs([(0, 1), (5, 2)]), ShardLeaseRevision(7)); + assignment.set_shards(8, &epochs([(0, 1), (5, 2)]), ShardLeaseRevision::of(7)); let granted = in_secs(60); let outcome = - assignment.adopt_grant(None, &epochs([(0, 1)]), granted, ShardLeaseRevision(6)); + assignment.adopt_grant(None, &epochs([(0, 1)]), granted, ShardLeaseRevision::of(6)); assert_eq!( outcome, ShardDeliveryOutcome::Stale { - delivered: ShardLeaseRevision(6), - applied: ShardLeaseRevision(7), + delivered: ShardLeaseRevision::of(6), + applied: ShardLeaseRevision::of(7), } ); assert_eq!( @@ -2986,7 +3082,7 @@ mod shard_assignment_tests { epochs([(0, 1), (5, 2)]), "the older delivery narrowed the set the newer one had just widened" ); - assert_eq!(assignment.revision, ShardLeaseRevision(7)); + assert_eq!(assignment.revision, ShardLeaseRevision::of(7)); assert_eq!( assignment.expires_at, Some(granted), @@ -3000,11 +3096,15 @@ mod shard_assignment_tests { #[test] fn a_delivery_at_the_same_revision_is_applied() { let mut assignment = ShardAssignment::unexpiring(8, [ShardId::new(0)]); - assignment.set_shards(8, &epochs([(0, 1)]), ShardLeaseRevision(7)); + assignment.set_shards(8, &epochs([(0, 1)]), ShardLeaseRevision::of(7)); let refreshed = in_secs(60); - let outcome = - assignment.adopt_grant(None, &epochs([(0, 1)]), refreshed, ShardLeaseRevision(7)); + let outcome = assignment.adopt_grant( + None, + &epochs([(0, 1)]), + refreshed, + ShardLeaseRevision::of(7), + ); assert_eq!( outcome, @@ -3024,17 +3124,21 @@ mod shard_assignment_tests { Some(8), &epochs([(0, 1), (1, 1)]), granted, - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); - assignment.set_shards(8, &epochs([(0, 1), (1, 1), (2, 1)]), ShardLeaseRevision(2)); + assignment.set_shards( + 8, + &epochs([(0, 1), (1, 1), (2, 1)]), + ShardLeaseRevision::of(2), + ); assert_eq!( assignment.expires_at, Some(granted), "a full-replace push must leave the lease where the grant put it" ); - assignment.revoke_shards(&HashSet::from([ShardId::new(2)]), ShardLeaseRevision(3)); + assignment.revoke_shards(&HashSet::from([ShardId::new(2)]), ShardLeaseRevision::of(3)); assert_eq!( assignment.expires_at, Some(granted), @@ -3054,14 +3158,14 @@ mod shard_assignment_tests { Some(8), &epochs([(0, 1)]), now + Duration::from_secs(60), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); assignment.adopt_grant( None, &epochs([(0, 1)]), now + Duration::from_secs(30), - ShardLeaseRevision(2), + ShardLeaseRevision::of(2), ); assert_eq!(assignment.expires_at, Some(now + Duration::from_secs(30))); @@ -3078,16 +3182,16 @@ mod shard_assignment_tests { Some(8), &epochs([(0, 1), (1, 1)]), expiry, - ShardLeaseRevision(3), + ShardLeaseRevision::of(3), ); let revoked = HashSet::from([ShardId::new(0)]); - let stale = assignment.revoke_shards(&revoked, ShardLeaseRevision(2)); + let stale = assignment.revoke_shards(&revoked, ShardLeaseRevision::of(2)); assert_eq!( stale, ShardDeliveryOutcome::Stale { - delivered: ShardLeaseRevision(2), - applied: ShardLeaseRevision(3), + delivered: ShardLeaseRevision::of(2), + applied: ShardLeaseRevision::of(3), } ); assert!( @@ -3095,12 +3199,12 @@ mod shard_assignment_tests { "a revoke older than the last delivery applied must be ignored" ); - let applied = assignment.revoke_shards(&revoked, ShardLeaseRevision(5)); + let applied = assignment.revoke_shards(&revoked, ShardLeaseRevision::of(5)); assert_eq!(applied, ShardDeliveryOutcome::Applied { set_changed: true }); assert!(!assignment.contains(&ShardId::new(0))); assert_eq!( assignment.revision, - ShardLeaseRevision(5), + ShardLeaseRevision::of(5), "the revoke's revision is recorded like any other delivery's" ); assert_eq!( @@ -3113,7 +3217,7 @@ mod shard_assignment_tests { None, &epochs([(0, 1), (1, 1)]), in_secs(60), - ShardLeaseRevision(4), + ShardLeaseRevision::of(4), ); assert!(matches!(late_grant, ShardDeliveryOutcome::Stale { .. })); assert!( @@ -3123,18 +3227,18 @@ mod shard_assignment_tests { } /// The corrective delivery: a renewal that answers with a different set - /// than was claimed is applied like a push, and reports the set moved so + /// than was held is applied like a push, and reports the set moved so /// the caller sweeps and recovers. #[test] fn a_renewal_that_changes_the_set_reports_it() { let mut assignment = ShardAssignment::unexpiring(8, [ShardId::new(0), ShardId::new(1)]); - assignment.set_shards(8, &epochs([(0, 1), (1, 1)]), ShardLeaseRevision(3)); + assignment.set_shards(8, &epochs([(0, 1), (1, 1)]), ShardLeaseRevision::of(3)); let outcome = assignment.adopt_grant( None, &epochs([(1, 1), (2, 5)]), in_secs(60), - ShardLeaseRevision(4), + ShardLeaseRevision::of(4), ); assert_eq!(outcome, ShardDeliveryOutcome::Applied { set_changed: true }); @@ -3143,7 +3247,122 @@ mod shard_assignment_tests { "the dropped shard is gone" ); assert_eq!(assignment.epoch_of(&ShardId::new(2)), Some(ShardEpoch(5))); - assert_eq!(assignment.revision, ShardLeaseRevision(4)); + assert_eq!(assignment.revision, ShardLeaseRevision::of(4)); + } + + fn revision_of(incarnation: Uuid, number: u64) -> ShardLeaseRevision { + ShardLeaseRevision { + incarnation: Some(incarnation), + number, + } + } + + /// A manager that came back on a wiped or restored store counts its revisions from far below + /// the last one this executor applied. Its reply is still the manager in charge speaking. + #[test] + fn a_reply_from_another_manager_process_starts_the_revisions_over() { + let (old_manager, new_manager) = (Uuid::new_v4(), Uuid::new_v4()); + let mut assignment = ShardAssignment::default(); + assignment.adopt_grant( + Some(8), + &epochs([(0, 4)]), + in_secs(60), + revision_of(old_manager, 10_000), + ); + // what a lost lease does before the re-registration + assignment.clear(Instant::now()); + + let outcome = assignment.adopt_grant( + Some(8), + &epochs([(0, 5)]), + in_secs(60), + revision_of(new_manager, 3), + ); + + assert_eq!(outcome, ShardDeliveryOutcome::Applied { set_changed: true }); + assert_eq!(assignment.epoch_of(&ShardId::new(0)), Some(ShardEpoch(5))); + assert_eq!(assignment.revision, revision_of(new_manager, 3)); + } + + /// Starting the revisions over must not be a way back in for the manager that was left: its + /// delayed push carries a revision far above the new manager's, and would win on the number. + #[test] + fn a_push_from_a_manager_process_that_is_not_followed_is_ignored() { + let (old_manager, new_manager) = (Uuid::new_v4(), Uuid::new_v4()); + let mut assignment = ShardAssignment::default(); + assignment.adopt_grant( + Some(8), + &epochs([(0, 4)]), + in_secs(60), + revision_of(old_manager, 10_000), + ); + + // The new manager's push arrives before any reply from it: nothing says yet that it is + // the one in charge. + let early = assignment.set_shards(8, &epochs([(1, 1)]), revision_of(new_manager, 2)); + assert_eq!( + early, + ShardDeliveryOutcome::FromAnotherManager { + delivered: revision_of(new_manager, 2), + applied: revision_of(old_manager, 10_000), + } + ); + assert_eq!(assignment.shard_epochs, epochs([(0, 4)])); + + assignment.adopt_grant( + None, + &epochs([(0, 5)]), + in_secs(60), + revision_of(new_manager, 3), + ); + + let delayed_push = assignment.set_shards( + 8, + &epochs([(0, 4), (1, 4)]), + revision_of(old_manager, 10_001), + ); + let delayed_revoke = assignment.revoke_shards( + &HashSet::from([ShardId::new(0)]), + revision_of(old_manager, 10_002), + ); + for delayed in [delayed_push, delayed_revoke] { + assert!( + matches!(delayed, ShardDeliveryOutcome::FromAnotherManager { .. }), + "got {delayed:?}" + ); + } + assert_eq!(assignment.shard_epochs, epochs([(0, 5)])); + assert_eq!(assignment.revision, revision_of(new_manager, 3)); + + // The followed manager's own pushes are ordered by number as ever. + let push = assignment.set_shards(8, &epochs([(0, 5), (2, 1)]), revision_of(new_manager, 4)); + assert_eq!(push, ShardDeliveryOutcome::Applied { set_changed: true }); + } + + /// A manager from before incarnations sends none. Its deliveries are ordered by number alone, + /// and they do not make the executor forget the process it follows. + #[test] + fn a_delivery_that_names_no_incarnation_is_ordered_by_its_number_alone() { + let manager = Uuid::new_v4(); + let mut assignment = ShardAssignment::default(); + assignment.adopt_grant( + Some(8), + &epochs([(0, 1)]), + in_secs(60), + revision_of(manager, 5), + ); + + let stale = assignment.set_shards(8, &epochs([(1, 1)]), ShardLeaseRevision::of(4)); + assert!(matches!(stale, ShardDeliveryOutcome::Stale { .. })); + + let newer = assignment.adopt_grant( + None, + &epochs([(1, 1)]), + in_secs(60), + ShardLeaseRevision::of(6), + ); + assert_eq!(newer, ShardDeliveryOutcome::Applied { set_changed: true }); + assert_eq!(assignment.revision, revision_of(manager, 6)); } /// `clear()` lapses the lease as of `now`. `None` would mean diff --git a/golem-common/src/model/protobuf.rs b/golem-common/src/model/protobuf.rs index 104e283131..dda7944264 100644 --- a/golem-common/src/model/protobuf.rs +++ b/golem-common/src/model/protobuf.rs @@ -913,7 +913,7 @@ mod tests { assert_eq!(received, pushed); let mut assignment = ShardAssignment::default(); - assignment.set_shards(1024, &received, ShardLeaseRevision(1)); + assignment.set_shards(1024, &received, ShardLeaseRevision::of(1)); assert_eq!(assignment.epoch_of(&ShardId::new(0)), Some(ShardEpoch(1))); assert_eq!(assignment.epoch_of(&ShardId::new(7)), Some(ShardEpoch(42))); diff --git a/golem-common/src/model/quota.rs b/golem-common/src/model/quota.rs index e5104d8982..2a5811746c 100644 --- a/golem-common/src/model/quota.rs +++ b/golem-common/src/model/quota.rs @@ -55,8 +55,19 @@ impl LeaseEpoch { Self(0) } + /// The next epoch, or `None` when there is none left. + /// + /// The fallible one is the one to use where the counter is advanced: an epoch that cannot + /// advance is a stuck lease, and refusing that one operation keeps the process up, while + /// [`Self::next`]'s panic takes the whole shard manager down and does it again on every retry. + pub fn checked_next(self) -> Option { + self.0.checked_add(1).map(Self) + } + + /// Panics at the ceiling. Only for a caller that has no way to refuse - prefer + /// [`Self::checked_next`]. pub fn next(self) -> Self { - Self(self.0.checked_add(1).expect("LeaseEpoch overflow")) + self.checked_next().expect("LeaseEpoch overflow") } } diff --git a/golem-common/src/redis.rs b/golem-common/src/redis.rs index 5434e63640..862b200f95 100644 --- a/golem-common/src/redis.rs +++ b/golem-common/src/redis.rs @@ -431,6 +431,37 @@ return 1 self.record(start, "EVAL", result) } + /// Runs a Lua script, which Redis executes atomically. `keys` get the pool's key prefix. + pub async fn eval( + &self, + script: &'static str, + keys: &[K], + args: Vec, + options: Option<&Options>, + ) -> RedisResult + where + K: AsRef, + { + self.ensure_connected().await?; + let start = Instant::now(); + let options = options.cloned().unwrap_or_default(); + let mut command_args: Vec = Vec::with_capacity(2 + keys.len() + args.len()); + command_args.push(script.into()); + command_args.push((keys.len() as i64).into()); + for key in keys { + command_args.push(self.prefixed_key(key).into()); + } + command_args.extend(args); + let result = self + .pool + .next() + .with_options(&options) + .custom_raw(cmd!("EVAL"), command_args) + .await + .and_then(|frame| frame.try_into()); + self.record(start, "EVAL", result) + } + pub async fn compare_and_mutate_many_hash( &self, key: K, diff --git a/golem-service-base/src/clients/shard_manager.rs b/golem-service-base/src/clients/shard_manager.rs index 9b545460ea..95ae62ac3f 100644 --- a/golem-service-base/src/clients/shard_manager.rs +++ b/golem-service-base/src/clients/shard_manager.rs @@ -56,22 +56,37 @@ pub trait ShardManager: Send + Sync { /// UUID it generated at startup. Idempotent: the same `executor_id` at the /// same address refreshes the existing shard lease rather than creating a /// second one. + /// + /// `previous_shard_epochs` is the set this process held under an earlier + /// `executor_id` the manager answered `LeaseNotFound` for, and is empty on + /// a first registration. It is evidence, not a request: it never assigns a + /// shard, and only raises the manager's recorded epochs where its state has + /// lost history, so the epochs it mints next clear the oplog rows this + /// process wrote before. async fn register( &self, port: u16, pod_name: Option, executor_id: Uuid, + previous_shard_epochs: BTreeMap, ) -> Result; /// Extends this executor's shard lease. `shard_epochs` is the set the /// executor believes it holds; it is not a condition of the renewal. A - /// claim that does not match the manager's view is renewed all the same, + /// set that does not match the manager's view is renewed all the same, /// and the returned lease carries the manager's set, which the caller /// adopts exactly as it would an `AssignShards` push. + /// + /// `fenced_shard_epochs` are the epochs recorded on oplogs this executor + /// was refused writes to, keyed by shard. Evidence, like + /// `previous_shard_epochs` on `register`: above the manager's record they + /// mean its state lost history, and every owner of the shard is minted one + /// past them; at or below it they move nothing. async fn renew_shard_lease( &self, executor_id: Uuid, shard_epochs: BTreeMap, + fenced_shard_epochs: BTreeMap, ) -> Result; /// Releases the shard lease on a graceful shutdown. Lenient by contract: a @@ -158,7 +173,7 @@ fn shard_lease_from_wire( Ok(ShardLease { shard_epochs: shard_epochs_from_proto(lease.shard_epochs)?, expires_at: expires_at_from_ttl(lease.lease_ttl, sent_at)?, - revision: ShardLeaseRevision(lease.revision), + revision: ShardLeaseRevision::from_wire(&lease.incarnation_id, lease.revision), }) } @@ -285,14 +300,21 @@ impl ShardManager for GrpcShardManager { port: u16, pod_name: Option, executor_id: Uuid, + previous_shard_epochs: BTreeMap, ) -> Result { with_retries( "shard_manager", "register", Some(format!("{pod_name:?}")), &self.retries, - &(self.client.clone(), port, pod_name, executor_id), - |(client, port, pod_name, executor_id)| { + &( + self.client.clone(), + port, + pod_name, + executor_id, + previous_shard_epochs, + ), + |(client, port, pod_name, executor_id, previous_shard_epochs)| { Box::pin(async move { let (sent_at, response) = client .call("register", move |client| { @@ -300,6 +322,11 @@ impl ShardManager for GrpcShardManager { port: *port as i32, pod_name: pod_name.clone(), executor_id: executor_id.to_string(), + previous_shard_epochs: shard_epochs_to_proto( + previous_shard_epochs + .iter() + .map(|(shard_id, epoch)| (*shard_id, *epoch)), + ), }; Box::pin(async move { let (sent_at, response) = issued(client.register(request)).await; @@ -318,7 +345,10 @@ impl ShardManager for GrpcShardManager { .map_err(ShardManagerError::ConversionError)?, expires_at: expires_at_from_ttl(success.lease_ttl, sent_at) .map_err(ShardManagerError::ConversionError)?, - revision: ShardLeaseRevision(success.revision), + revision: ShardLeaseRevision::from_wire( + &success.incarnation_id, + success.revision, + ), }, }) } @@ -339,6 +369,7 @@ impl ShardManager for GrpcShardManager { &self, executor_id: Uuid, shard_epochs: BTreeMap, + fenced_shard_epochs: BTreeMap, ) -> Result { let (sent_at, response) = self .client @@ -350,6 +381,11 @@ impl ShardManager for GrpcShardManager { .iter() .map(|(shard_id, epoch)| (*shard_id, *epoch)), ), + fenced_shard_epochs: shard_epochs_to_proto( + fenced_shard_epochs + .iter() + .map(|(shard_id, epoch)| (*shard_id, *epoch)), + ), }; Box::pin(async move { let (sent_at, response) = issued(client.renew_shard_lease(request)).await; @@ -706,7 +742,7 @@ impl From<&'static str> for QuotaError { /// The failure arms of `RenewShardLease` and `Deregister`; the executor /// branches on the arm, never on the message string. There is no stale-epoch -/// arm: a claim that does not match the manager's view is renewed and +/// arm: a held set that does not match the manager's view is renewed and /// corrected in the response, not refused. #[derive(Debug, Clone, thiserror::Error)] pub enum ShardLeaseError { @@ -811,6 +847,7 @@ mod tests { /// earlier than one anchored on arrival would - by exactly the time it took. #[test] fn a_delayed_grant_is_anchored_where_the_request_was_sent_not_where_the_answer_arrived() { + let incarnation = Uuid::new_v4(); let on_the_wire = golem_api_grpc::proto::golem::shardmanager::v1::ShardLease { shard_epochs: vec![], lease_ttl: Some(prost_types::Duration { @@ -818,6 +855,7 @@ mod tests { nanos: 0, }), revision: 3, + incarnation_id: incarnation.to_string(), }; let sent_at = Instant::now(); // the answer took its time @@ -830,6 +868,12 @@ mod tests { lease.expires_at < Instant::now() + Duration::from_secs(60), "time the answer spent in flight must come off the lease, never on to it" ); - assert_eq!(lease.revision, ShardLeaseRevision(3)); + assert_eq!( + lease.revision, + ShardLeaseRevision { + incarnation: Some(incarnation), + number: 3 + } + ); } } diff --git a/golem-service-base/src/db/mod.rs b/golem-service-base/src/db/mod.rs index ea9e5cca16..6ecb2671d6 100644 --- a/golem-service-base/src/db/mod.rs +++ b/golem-service-base/src/db/mod.rs @@ -83,7 +83,7 @@ pub trait Pool: Debug + Sync + Clone { Err(err) => { warn!( svc_name, api_name, error = ?err, - "Rolling back, transaction failed with repo error", + "Rolling back transaction, closure returned an error", ); // If rollback fails, we still return the original error, but log the rollback error diff --git a/golem-service-base/src/error/worker_executor.rs b/golem-service-base/src/error/worker_executor.rs index f0f62f5d31..d6c3dca230 100644 --- a/golem-service-base/src/error/worker_executor.rs +++ b/golem-service-base/src/error/worker_executor.rs @@ -132,6 +132,16 @@ pub enum WorkerExecutorError { PermissionDenied { details: String, }, + /// A write to the agent's oplog was refused by the storage because the shard epoch this + /// executor asserted is behind the one recorded for the oplog: another executor owns the + /// shard now. Typed so the invocation loop can stop the agent cleanly instead of treating + /// it as a runtime failure to retry; crosses the wire as `ShardingNotReady`, which the + /// worker service already answers by refreshing its routing table and retrying. + OplogFenced { + agent_id: AgentId, + expected_epoch: u64, + actual_epoch: Option, + }, } impl WorkerExecutorError { @@ -185,6 +195,14 @@ impl WorkerExecutorError { } } + pub fn oplog_fenced(agent_id: AgentId, expected_epoch: u64, actual_epoch: Option) -> Self { + Self::OplogFenced { + agent_id, + expected_epoch, + actual_epoch, + } + } + pub fn invalid_shard_id(shard_id: ShardId, shard_ids: HashSet) -> Self { Self::InvalidShardId { shard_id, @@ -337,6 +355,22 @@ impl Display for WorkerExecutorError { Self::PermissionDenied { details } => { write!(f, "Permission denied: {details}") } + Self::OplogFenced { + agent_id, + expected_epoch, + actual_epoch, + } => match actual_epoch { + Some(actual) => write!( + f, + "Oplog write for {agent_id} fenced: this executor asserted shard epoch \ + {expected_epoch}, the stored epoch is {actual}" + ), + None => write!( + f, + "Oplog write for {agent_id} fenced: this executor asserted shard epoch \ + {expected_epoch}, but no epoch is stored for the oplog" + ), + }, } } } @@ -381,6 +415,7 @@ impl Error for WorkerExecutorError { Self::FileSystemError { .. } => "File system error", Self::ReadOnlyViolation { .. } => "Read-only agent method attempted a side effect", Self::PermissionDenied { .. } => "Permission denied", + Self::OplogFenced { .. } => "Oplog write fenced: the shard has a new owner", } } } @@ -417,6 +452,7 @@ impl ApiErrorDetails for WorkerExecutorError { Self::FileSystemError { .. } => "FileSystemError", Self::ReadOnlyViolation { .. } => "ReadOnlyViolation", Self::PermissionDenied { .. } => "PermissionDenied", + Self::OplogFenced { .. } => "OplogFenced", } } @@ -429,6 +465,7 @@ impl ApiErrorDetails for WorkerExecutorError { | Self::PromiseAlreadyCompleted { .. } | Self::Interrupted { .. } | Self::InvalidShardId { .. } + | Self::OplogFenced { .. } | Self::ComponentNotFound { .. } => true, Self::InvalidRequest { .. } | Self::AgentCreationFailed { .. } @@ -808,6 +845,18 @@ impl From for golem::worker::v1::WorkerExecutionError { ), ), }, + // The client cannot act on the epochs; what it can do is what it does for a lapsed + // lease - refresh its routing table and retry on the owner. A fence can land in the + // middle of an invocation, and the retry is still one invocation, not a second: the + // worker service sends it under the same idempotency key, the new owner finds the key + // in the oplog it took over if the invocation got that far, and answers from it. + WorkerExecutorError::OplogFenced { .. } => Self { + error: Some( + golem::worker::v1::worker_execution_error::Error::ShardingNotReady( + golem::worker::v1::ShardingNotReady {}, + ), + ), + }, } } } @@ -1102,6 +1151,11 @@ pub enum InterruptKind { Restart, Suspend(Timestamp), Jump, + /// This executor no longer owns the agent's shard. Terminal here: the agent is stopped + /// without writing to its oplog or its status, dropped from the executor, and left for the + /// worker service to resume on the shard's owner. Never a restart in place - that would + /// reopen the oplog with the same stale epoch. + ShardLost, } impl Display for InterruptKind { @@ -1111,6 +1165,9 @@ impl Display for InterruptKind { InterruptKind::Restart => write!(f, "Simulated crash via the Golem API"), InterruptKind::Suspend(_) => write!(f, "Suspended"), InterruptKind::Jump => write!(f, "Jumping back in time"), + InterruptKind::ShardLost => { + write!(f, "This executor no longer owns the agent's shard") + } } } } diff --git a/golem-shard-manager/src/grpc.rs b/golem-shard-manager/src/grpc.rs index 603174ea8d..c4e0c6b2b0 100644 --- a/golem-shard-manager/src/grpc.rs +++ b/golem-shard-manager/src/grpc.rs @@ -56,11 +56,22 @@ impl ShardManagerServiceImpl { executor_id: ExecutorId, pod: Pod, pod_name: Option, + previous_epochs: BTreeMap, ) -> Result { - debug!(executor_id = %executor_id, addr = %pod, "Received request to register executor"); + debug!( + executor_id = %executor_id, + addr = %pod, + previous_shards = previous_epochs.len(), + "Received request to register executor" + ); let ack = self .shard_management - .register_executor(executor_id, ExecutorAddr::from(pod), pod_name) + .register_executor_with_previous_epochs( + executor_id, + ExecutorAddr::from(pod), + pod_name, + previous_epochs, + ) .await?; debug!(executor_id = %executor_id, addr = %pod, "Registered executor"); Ok(ack) @@ -100,11 +111,14 @@ impl ShardManagerService for ShardManagerServiceImpl { .ok_or_else(|| tonic::Status::invalid_argument("missing source IP"))? .ip(); - let request = request.into_inner(); + let mut request = request.into_inner(); - // Before anything touches the state: an executor that cannot name itself has no identity to - // renew or deregister a lease with. + // Both before anything touches the state: an executor that cannot name itself has no + // identity to renew or deregister a lease with, and carried epochs the manager cannot + // decode is not evidence it can weigh. let executor_id = parse_executor_id(&request.executor_id)?; + let previous_epochs = + parse_shard_epochs(std::mem::take(&mut request.previous_shard_epochs))?; let record = recorded_grpc_api_request!( "register", @@ -117,7 +131,7 @@ impl ShardManagerService for ShardManagerServiceImpl { let pod = make_pod(source_ip, request.port)?; let response = self - .register_internal(executor_id, pod, request.pod_name) + .register_internal(executor_id, pod, request.pod_name, previous_epochs) .instrument(record.span.clone()) .await; @@ -128,6 +142,7 @@ impl ShardManagerService for ShardManagerServiceImpl { shard_epochs: shard_epoch_entries(&ack.grant.shard_epochs), lease_ttl: Some(lease_ttl_to_proto(ack.grant.expires_at, Utc::now())), revision: ack.grant.revision.0, + incarnation_id: crate::sharding::incarnation_id(), }, )), Err(error) => { @@ -150,14 +165,16 @@ impl ShardManagerService for ShardManagerServiceImpl { ) -> Result, tonic::Status> { let request = request.into_inner(); - // Both before any state is touched: an executor that cannot name itself has no lease to - // renew, and a claim the manager cannot decode is not one it can validate. + // All before any state is touched: an executor that cannot name itself has no lease to + // renew, and a held or fenced epoch the manager cannot decode is not evidence it can + // weigh. let executor_id = parse_executor_id(&request.executor_id)?; - let claimed = parse_shard_epochs(request.shard_epochs)?; + let held = parse_shard_epochs(request.shard_epochs)?; + let fenced = parse_shard_epochs(request.fenced_shard_epochs)?; let result = match self .shard_management - .renew_shard_lease(executor_id, claimed) + .renew_shard_lease_with_fenced_epochs(executor_id, held, fenced) .await { Ok(grant) => golem::shardmanager::v1::renew_shard_lease_response::Result::Success( @@ -165,6 +182,7 @@ impl ShardManagerService for ShardManagerServiceImpl { shard_epochs: shard_epoch_entries(&grant.shard_epochs), lease_ttl: Some(lease_ttl_to_proto(grant.expires_at, Utc::now())), revision: grant.revision.0, + incarnation_id: crate::sharding::incarnation_id(), }, ), Err(error) => { @@ -186,11 +204,11 @@ impl ShardManagerService for ShardManagerServiceImpl { let request = request.into_inner(); let executor_id = parse_executor_id(&request.executor_id)?; - let claimed = parse_shard_epochs(request.shard_epochs)?; + let held = parse_shard_epochs(request.shard_epochs)?; let result = match self .shard_management - .deregister_executor(executor_id, claimed) + .deregister_executor(executor_id, held) .await { Ok(()) => golem::shardmanager::v1::deregister_response::Result::Success( @@ -441,11 +459,11 @@ fn parse_executor_id(raw: &str) -> Result { .map_err(|err| tonic::Status::invalid_argument(format!("invalid executor_id: {err}"))) } -/// The shard set an executor claims, decoded from the wire. +/// The shard set an executor sends as held, decoded from the wire. /// /// A `ShardEpochEntry` without a shard id names no shard, so it cannot be validated against /// anything: that is a malformed request, refused before any state is touched rather than silently -/// dropped from a claim whose whole point is to be exact. +/// dropped from a set whose whole point is to be exact. fn parse_shard_epochs( entries: Vec, ) -> Result, tonic::Status> { diff --git a/golem-shard-manager/src/quota/quota_service.rs b/golem-shard-manager/src/quota/quota_service.rs index b7597471e4..1eaa920b70 100644 --- a/golem-shard-manager/src/quota/quota_service.rs +++ b/golem-shard-manager/src/quota/quota_service.rs @@ -214,7 +214,7 @@ impl QuotaService { })?; let snapshot = state.clone(); let prev_rev = state.current_revision(); - let result = state.acquire_lease(pod, self.lease_duration, self.min_executors); + let result = state.acquire_lease(pod, self.lease_duration, self.min_executors)?; if let Err(e) = state.bump_revision() { warn!(error = %e, "failed to bump revision, rolling back"); diff --git a/golem-shard-manager/src/quota/quota_service_tests.rs b/golem-shard-manager/src/quota/quota_service_tests.rs index e731f2ee74..56bf5c6861 100644 --- a/golem-shard-manager/src/quota/quota_service_tests.rs +++ b/golem-shard-manager/src/quota/quota_service_tests.rs @@ -379,6 +379,31 @@ async fn renew_lease_rejects_stale_epoch() { assert_eq!(l3.epoch(), l2.epoch().next()); } +#[test] +// `epoch` is a client-supplied argument straight off the wire (`grpc.rs` builds it as +// `LeaseEpoch(request.epoch)` with no validation), and `LeaseEpoch::next()` panics on overflow. +// A `u64::MAX` claim must be rejected as an ordinary stale epoch rather than aborting the +// process. +async fn renew_lease_rejects_u64_max_epoch_instead_of_panicking() { + let fetcher = Arc::new(InMemoryFetcher::new()); + let env = env_id(); + let def = make_definition(env, "tokens"); + let id = def.id; + fetcher.put(def).await; + + let svc = QuotaService::new(test_config(), fetcher, test_repo()); + let pod = test_pod(); + + svc.acquire_lease(env, ResourceName("tokens".into()), pod) + .await + .unwrap(); + + let result = svc + .renew_lease(id, pod, LeaseEpoch(u64::MAX), 0, vec![]) + .await; + assert!(matches!(result, Err(QuotaError::StaleEpoch { .. }))); +} + #[test] async fn renew_lease_fails_for_unknown_pod() { let fetcher = Arc::new(InMemoryFetcher::new()); @@ -565,6 +590,27 @@ async fn release_lease_rejects_stale_epoch() { svc.release_lease(id, pod, l2.epoch(), 0).await.unwrap(); } +#[test] +// Same defect class as `renew_lease_rejects_u64_max_epoch_instead_of_panicking`, on the release +// path's own `epoch.next()` comparison. +async fn release_lease_rejects_u64_max_epoch_instead_of_panicking() { + let fetcher = Arc::new(InMemoryFetcher::new()); + let env = env_id(); + let def = make_definition(env, "tokens"); + let id = def.id; + fetcher.put(def).await; + + let svc = QuotaService::new(test_config(), fetcher, test_repo()); + let pod = test_pod(); + + svc.acquire_lease(env, ResourceName("tokens".into()), pod) + .await + .unwrap(); + + let result = svc.release_lease(id, pod, LeaseEpoch(u64::MAX), 0).await; + assert!(matches!(result, Err(QuotaError::StaleEpoch { .. }))); +} + #[test] async fn release_lease_allows_re_acquire() { let fetcher = Arc::new(InMemoryFetcher::new()); diff --git a/golem-shard-manager/src/quota/quota_state.rs b/golem-shard-manager/src/quota/quota_state.rs index 66a055d855..7ad6409b0c 100644 --- a/golem-shard-manager/src/quota/quota_state.rs +++ b/golem-shard-manager/src/quota/quota_state.rs @@ -24,6 +24,19 @@ use std::collections::HashMap; use std::time::Duration; use tracing::debug; +/// Whether `epoch` immediately precedes `next`, without calling `LeaseEpoch::next()` on `epoch` +/// itself - which panics at `u64::MAX` (`checked_add(1).expect(..)`). +/// +/// `epoch` here is the caller's claimed epoch, taken straight off the wire (a `renew_lease` / +/// `release_lease` argument, itself `golem_common::model::quota::LeaseEpoch(request.epoch)` in +/// `grpc.rs` with nothing upstream bounding it) - unlike `pod_lease.epoch`, which only ever +/// advances by exactly one through this state's own `checked_next()` calls. A `u64::MAX` claim can never +/// legitimately precede a real stored epoch, so it simply fails this check like any other stale +/// one, rather than aborting the process. +fn precedes(epoch: LeaseEpoch, next: LeaseEpoch) -> bool { + epoch.0.checked_add(1) == Some(next.0) +} + pub(super) struct AcquireLeaseResult { pub epoch: LeaseEpoch, pub allocated_amount: u64, @@ -312,7 +325,7 @@ impl QuotaState { pod: Pod, lease_duration: Duration, min_executors: u64, - ) -> AcquireLeaseResult { + ) -> Result { let expired = self.housekeep(); if let Some(existing) = self.leases.get(&pod) { @@ -340,19 +353,23 @@ impl QuotaState { let pod_lease = self.leases.get_mut(&pod).expect("just inserted"); let epoch = pod_lease.epoch; - pod_lease.epoch = epoch.next(); + pod_lease.epoch = epoch.checked_next().ok_or_else(|| { + QuotaError::InternalError(anyhow::anyhow!( + "lease epoch for pod {pod} cannot advance past {epoch}" + )) + })?; self.remaining -= allocated_amount; pod_lease.allocated = allocated_amount; pod_lease.granted_at = now; pod_lease.expires_at = expires_at; - AcquireLeaseResult { + Ok(AcquireLeaseResult { epoch, allocated_amount, expires_at, expired, total_available_amount, - } + }) } pub fn renew_lease( @@ -367,7 +384,7 @@ impl QuotaState { let pod_lease = self.leases.get_mut(pod).ok_or(QuotaError::LeaseNotFound { resource_definition_id: self.definition.id, })?; - if epoch.next() != pod_lease.epoch { + if !precedes(epoch, pod_lease.epoch) { return Err(QuotaError::StaleEpoch { resource_definition_id: self.definition.id, provided: epoch, @@ -393,7 +410,11 @@ impl QuotaState { .get_mut(pod) .expect("just validated and refreshed"); let new_epoch = pod_lease.epoch; - pod_lease.epoch = new_epoch.next(); + pod_lease.epoch = new_epoch.checked_next().ok_or_else(|| { + QuotaError::InternalError(anyhow::anyhow!( + "lease epoch for pod {pod} cannot advance past {new_epoch}" + )) + })?; let allocated_amount = self.compute_allocation(pod, min_executors); let total_available_amount = self.total_available_amount(); @@ -424,7 +445,7 @@ impl QuotaState { let pod_lease = self.leases.get(pod).ok_or(QuotaError::LeaseNotFound { resource_definition_id: self.definition.id, })?; - if epoch.next() != pod_lease.epoch { + if !precedes(epoch, pod_lease.epoch) { return Err(QuotaError::StaleEpoch { resource_definition_id: self.definition.id, provided: epoch, diff --git a/golem-shard-manager/src/sharding/mod.rs b/golem-shard-manager/src/sharding/mod.rs index e7dc09cdf5..41cd059923 100644 --- a/golem-shard-manager/src/sharding/mod.rs +++ b/golem-shard-manager/src/sharding/mod.rs @@ -27,5 +27,5 @@ pub mod worker_executor; pub use model::{ ExecutorAddr, ExecutorAddrs, ExecutorId, ExecutorLease, ExecutorShards, RegisterAck, ShardAssignmentEntry, ShardAssignmentPush, ShardEpoch, ShardLeaseGrant, ShardLeaseRevision, - ShardLeaseState, + ShardLeaseState, incarnation_id, }; diff --git a/golem-shard-manager/src/sharding/model.rs b/golem-shard-manager/src/sharding/model.rs index 873f4bf1d2..5603a8ba2d 100644 --- a/golem-shard-manager/src/sharding/model.rs +++ b/golem-shard-manager/src/sharding/model.rs @@ -22,6 +22,7 @@ use std::collections::{BTreeMap, BTreeSet, HashSet}; use std::fmt; use std::fmt::{Debug, Display, Formatter}; use std::net::IpAddr; +use std::sync::LazyLock; use std::time::Duration; use tracing::warn; use uuid::Uuid; @@ -36,7 +37,15 @@ impl ShardEpoch { } pub fn next(self) -> Self { - Self(self.0.checked_add(1).expect("ShardEpoch overflow")) + self.checked_next().expect("ShardEpoch overflow") + } + + /// `None` at `u64::MAX`, rather than panicking. Callers deriving an epoch to *store* from + /// untrusted input (held or fenced epochs off the wire - see `raise_epoch_floor_for`) use this: the + /// value that ends up in `shard_epochs` must never be `u64::MAX` itself, or the next ordinary + /// reassignment's call to [`Self::next`] on it panics instead of this one. + pub fn checked_next(self) -> Option { + self.0.checked_add(1).map(Self) } } @@ -114,6 +123,19 @@ impl Display for ShardLeaseRevision { } } +/// Names this shard manager process on every shard set it delivers, beside the revision. +/// +/// A revision orders the deliveries of one store's history, and an executor keeps the last one +/// it applied. A process that starts on a wiped or restored store counts from below that, so its +/// sets would be dropped as stale. Minted per process and never stored, so a failover, a wipe +/// and a restore all present a new one: the executor follows the process that answered its +/// latest request, starts its revisions over when that changes, and ignores a push from any +/// other process. +pub fn incarnation_id() -> String { + static INCARNATION_ID: LazyLock = LazyLock::new(Uuid::new_v4); + INCARNATION_ID.to_string() +} + #[derive(Clone, Copy, Debug, PartialEq, Eq, BinaryCodec)] #[desert(evolution())] pub struct ShardAssignmentEntry { @@ -475,25 +497,209 @@ impl ShardLeaseState { .collect() } + /// Raises the recorded epoch of every shard `held` names to at least the held value, + /// and reports the shards that moved. + /// + /// A held epoch can only be ahead of what is recorded here if this state regressed, because an + /// executor is never told an epoch that was not written first: a rebalance is stored before + /// any of it is sent, and a grant is read off the state that is about to be persisted. So in + /// ordinary operation this is a no-op, and a non-empty result means the store lost history - + /// it was wiped, restored from a backup, or replaced. + /// + /// That case has to be repaired, because [`Self::shard_epochs`] is the only record of it. A + /// fresh state starts the epochs at zero while the executors, and the oplog rows their writes + /// are fenced against, still hold higher ones; since an epoch only climbs when a shard changes + /// owner, the fence would refuse those agents for good. An executor's whole set reaches the + /// manager at two moments, and each repairs one kind of loss. A renewal repairs a state + /// restored from a backup that still lists the renewing executor. A state that was wiped or + /// replaced does not list it, so the renewal is refused as a lease not found, and the + /// re-registration that follows carries the set the executor held under its earlier id. A + /// process that restarted holds no set to carry, so a store lost while every executor restarted + /// as well is repaired by the first executor refused a write to one of those oplogs, on its + /// next renewal - see [`Self::raise_epoch_floor_past`] - and an oplog nobody opens keeps its + /// rows until it is opened. + /// + /// `executor_id`'s own shards, and unassigned ones, take the held epoch. A held epoch on a shard + /// the manager has given to somebody else moves nothing: it is the holder's word alone, + /// and believing it there would let any executor push another one off its epoch. At or below + /// the record it is an executor that missed a push, and the grant corrects it. Above the record + /// the holder may well have held that epoch before the history was lost, and then the oplog + /// rows it wrote refuse the owner's writes - which the owner reports, and + /// [`Self::raise_epoch_floor_past`] repairs one renewal later on evidence the storage holds. + /// + /// The assignment moves with the high-water and never apart from it - [`Self::check_invariants`] + /// requires the two to agree. The value only ever climbs, so this cannot walk an epoch back to + /// one a stale writer still holds. Returns every shard whose epoch moved. + pub fn raise_epoch_floor( + &mut self, + executor_id: ExecutorId, + held: &BTreeMap, + ) -> Vec { + self.raise_epoch_floor_for(Some(executor_id), None, held) + } + + /// Raises the recorded epoch of every assigned shard `stored` names to one past the stored + /// value, and reports the shards that moved. + /// + /// `stored` is what refused oplog writes found on the rows. Ahead of the record it proves the + /// state lost history, exactly as a held epoch ahead of it does - see [`Self::raise_epoch_floor`] - + /// but nothing proves the reporter wrote those rows, so every assigned entry, the reporter's + /// own included, is minted one past it: a generation nobody has held. An unassigned shard's + /// high-water rises to the stored value, so its next mint lands one past it. At or below the + /// record it is the ordinary loser of a shard move - the new owner recorded its epoch and the + /// old one was refused - and moves nothing. + /// + /// It does not commute with [`Self::raise_epoch_floor`] when the reporter's held epoch on its own + /// shard equals the report. Held first records the held epoch, and the report then sits at the + /// record; report first mints one past it, and the held epoch is then below the record. Callers + /// that hold both apply the report first, so a report at or above the held epoch always ends one + /// past it. A held epoch that already reached the state in an earlier request is at the record when + /// an equal report arrives, and that tie is the equality this cannot tell from a shard move. + pub fn raise_epoch_floor_past( + &mut self, + reporter: ExecutorId, + stored: &BTreeMap, + ) -> Vec { + self.raise_epoch_floor_for(None, Some(reporter), stored) + } + + /// How far one held or fenced epoch may move a shard's recorded epoch. + /// + /// Epochs climb by one per owner change, so even a store restored from a very old backup is + /// nowhere near this far behind the oplogs. A value further ahead is a bad value - a bug, a + /// corrupted row, a misbehaving executor - and storing it would leave the shard one or two + /// owner changes before its epochs run out, for good. + const MAX_EPOCH_JUMP: u64 = 1 << 32; + + /// The one rule behind [`Self::raise_epoch_floor`] and [`Self::raise_epoch_floor_past`]: a + /// held epoch moves the entries `holder` owns to the epoch itself and leaves everybody else's + /// alone, a report mints every assigned entry one past it, and an unassigned shard's + /// high-water takes the epoch either way. + /// + /// `reporter` is the executor a fenced report came from, and only [`Self::raise_epoch_floor_past`] + /// has one. It settles the single case equality cannot: a report of the epoch this shard is + /// already recorded at, from the executor currently assigned that shard, means the storage + /// refused the assignee at its own generation - so somebody else holds it. Only a manager that + /// lost its state mints one generation twice, and the repair is to mint past it. Every other + /// report at or below the record is the ordinary loser of a shard move and moves nothing. + fn raise_epoch_floor_for( + &mut self, + holder: Option, + reporter: Option, + epochs: &BTreeMap, + ) -> Vec { + let mut raised = Vec::new(); + for (shard_id, sent_epoch) in epochs { + // An epoch naming a shard outside the current count is stale in a way this cannot + // repair, and recording it would break the invariants. + if !self.contains_shard(*shard_id) { + continue; + } + let assignee = self + .shard_assignments + .get(shard_id) + .map(|entry| entry.executor_id); + // Nothing corroborates a held epoch, so it moves only what is the holder's to move. + if holder.is_some() && assignee.is_some() && assignee != holder { + continue; + } + // The assignee reporting a fence at the epoch it was granted: the row refused the + // executor the manager believes owns it, so another writer holds that generation. + let collides_with_the_assignee = reporter.is_some_and(|reporter| { + self.shard_epochs + .get(shard_id) + .is_some_and(|recorded| recorded == sent_epoch) + && assignee == Some(reporter) + }); + // Against the high-water, so the floor only ever climbs: a value at or below one this + // shard has already reached is not evidence of anything. + if !collides_with_the_assignee + && self + .shard_epochs + .get(shard_id) + .is_some_and(|recorded| recorded >= sent_epoch) + { + continue; + } + // Nothing says who wrote the rows a report found, so stamping their epoch onto an + // entry could put two live executors on one `(shard, epoch)` - the pair the fence + // cannot tell apart. One past it is a generation nobody has held. An unassigned shard + // has no entry to corrupt, and raising its high-water only makes the next mint start + // above the epoch already in use. + let recorded = self.shard_epochs.get(shard_id).map_or(0, |epoch| epoch.0); + if sent_epoch.0 - recorded > Self::MAX_EPOCH_JUMP { + warn!( + shard_id = %shard_id, + recorded, + epoch = sent_epoch.0, + "Ignoring a shard epoch implausibly far above the recorded one" + ); + continue; + } + let mints_past_report = holder.is_none() && assignee.is_some(); + let candidate = if mints_past_report { + sent_epoch.checked_next() + } else { + Some(*sent_epoch) + }; + // The wire carries a raw `u64`, and a record near the top is reachable one bounded + // jump at a time. The guard is on what this would store, and it stops one short of the + // last epoch `next_epoch_for` mints, so a shard can still change owner once more after + // its floor was raised. + let Some(epoch) = candidate.filter(|epoch| epoch.0 < u64::MAX - 1) else { + warn!( + shard_id = %shard_id, + "Ignoring an out-of-range shard epoch; storing it would overflow a later mint" + ); + continue; + }; + self.shard_epochs.insert(*shard_id, epoch); + if let Some(entry) = self.shard_assignments.get_mut(shard_id) { + entry.epoch = epoch; + } + raised.push(*shard_id); + } + raised + } + /// The ownership epoch `shard_id` takes when it is assigned to `executor_id`: unchanged while /// the owner stays the same, one past the highest epoch ever recorded for that shard when the /// owner changes. /// + /// `None` when there is no epoch left above the recorded one: `u64::MAX - 1` is the last + /// epoch minted. [`Self::raise_epoch_floor_for`] stores nothing that high, so only ordinary + /// owner changes reach it, and a shard whose epoch cannot advance stays where it is rather + /// than abort the manager on every retry of the same plan. + /// /// Pure, and the single definition of the rule; [`Self::assign_shard`] mints with it. - pub fn next_epoch_for(&self, executor_id: ExecutorId, shard_id: ShardId) -> ShardEpoch { + pub fn next_epoch_for(&self, executor_id: ExecutorId, shard_id: ShardId) -> Option { match self.shard_assignments.get(&shard_id) { - Some(entry) if entry.executor_id == executor_id => entry.epoch, + Some(entry) if entry.executor_id == executor_id => Some(entry.epoch), _ => match self.shard_epochs.get(&shard_id) { - Some(last) => last.next(), - None => ShardEpoch::initial(), + Some(last) => last.checked_next().filter(|epoch| epoch.0 != u64::MAX), + None => Some(ShardEpoch::initial()), }, } } - pub fn assign_shard(&mut self, executor_id: ExecutorId, shard_id: ShardId) -> ShardEpoch { - let epoch = self.next_epoch_for(executor_id, shard_id); + /// Assigns `shard_id` to `executor_id` at a freshly minted epoch, or leaves it alone when the + /// epoch cannot advance. An unassigned shard is re-homed by the next plan; a panic here would + /// take the manager down and do it again on the next attempt. + pub fn assign_shard( + &mut self, + executor_id: ExecutorId, + shard_id: ShardId, + ) -> Option { + let Some(epoch) = self.next_epoch_for(executor_id, shard_id) else { + warn!( + shard_id = %shard_id, + executor_id = %executor_id, + "Refusing to assign a shard whose epoch cannot advance; leaving it unassigned" + ); + return None; + }; self.assign_shard_with_epoch(executor_id, shard_id, epoch); - epoch + Some(epoch) } /// Records `epoch` for `shard_id`. [`Self::assign_shard`] is the only caller, and mints the @@ -895,6 +1101,13 @@ mod tests { ExecutorId(Uuid::from_u128(idx)) } + /// An executor that holds nothing, for tests that only care about the epochs a fenced report + /// carries. The reporter matters only when it is the shard's current assignee - the collision + /// case, which `a_fence_reported_by_the_assignee_at_its_own_epoch_is_minted_past` covers. + fn reporting_executor() -> ExecutorId { + executor(9999) + } + fn addr(idx: u8) -> ExecutorAddr { ExecutorAddr { ip: IpAddr::V4(Ipv4Addr::new(10, 0, 0, idx)), @@ -1054,20 +1267,20 @@ mod tests { // first assignment starts at the initial epoch assert_eq!( shard_state.assign_shard(executor(1), shard(0)), - ShardEpoch::initial() + Some(ShardEpoch::initial()) ); assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(0))); // re-assigning to the same owner is idempotent assert_eq!( shard_state.assign_shard(executor(1), shard(0)), - ShardEpoch(0) + Some(ShardEpoch(0)) ); // moving to another owner advances the epoch; stored == returned assert_eq!( shard_state.assign_shard(executor(2), shard(0)), - ShardEpoch(1) + Some(ShardEpoch(1)) ); assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); assert_eq!( @@ -1089,7 +1302,7 @@ mod tests { ); assert_eq!( shard_state.assign_shard(executor(1), shard(0)), - ShardEpoch(2) + Some(ShardEpoch(2)) ); assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(2))); } @@ -1107,18 +1320,18 @@ mod tests { // ... and later handed to other executors with epochs the evicted owner never held assert_eq!( shard_state.assign_shard(executor(2), shard(0)), - ShardEpoch(1) + Some(ShardEpoch(1)) ); assert_eq!( shard_state.assign_shard(executor(3), shard(1)), - ShardEpoch(1) + Some(ShardEpoch(1)) ); // a second eviction keeps advancing shard_state.remove_executor(executor(2)); assert_eq!( shard_state.assign_shard(executor(3), shard(0)), - ShardEpoch(2) + Some(ShardEpoch(2)) ); // housekeep-driven eviction behaves the same way @@ -1130,13 +1343,414 @@ mod tests { shard_state.add_executor(executor(4), addr(4), None, expired, TTL); assert_eq!( shard_state.assign_shard(executor(4), shard(0)), - ShardEpoch(3) + Some(ShardEpoch(3)) ); assert_eq!( shard_state.assign_shard(executor(4), shard(1)), - ShardEpoch(2) + Some(ShardEpoch(2)) + ); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_renewal_raises_epochs_the_state_lost() { + // The store was wiped and rebuilt: executor 1 re-registered and was handed its shards + // back, but from a state with no memory of what they were worth. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0, 1])]); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(0))); + assert_eq!(shard_state.epoch_for_shard(shard(1)), Some(ShardEpoch(0))); + + // The executor still holds - and its oplog rows are still fenced against - the epochs it + // was granted before the wipe. + let held = BTreeMap::from([(shard(0), ShardEpoch(5)), (shard(1), ShardEpoch(3))]); + assert_eq!( + shard_state.raise_epoch_floor(executor(1), &held), + vec![shard(0), shard(1)] + ); + + // Both halves move together, so the state stays consistent ... + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(5))); + assert_eq!(shard_state.epoch_for_shard(shard(1)), Some(ShardEpoch(3))); + assert_eq!( + shard_state.shard_epochs.get(&shard(0)), + Some(&ShardEpoch(5)) + ); + assert_eq!( + shard_state.shard_epochs.get(&shard(1)), + Some(&ShardEpoch(3)) + ); + assert!(shard_state.check_invariants().is_ok()); + + // ... and the next owner change mints above the restored high-water rather than + // re-issuing an epoch some oplog row already holds. + shard_state.add_executor(executor(2), addr(2), None, t0(), TTL); + assert_eq!( + shard_state.assign_shard(executor(2), shard(0)), + Some(ShardEpoch(6)) + ); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn raise_epoch_floor_only_ever_climbs() { + let mut shard_state = shard_state_with(4, &[(1, 1, &[0])]); + shard_state.remove_executor(executor(1)); + shard_state.add_executor(executor(2), addr(2), None, t0(), TTL); + shard_state.assign_shard(executor(2), shard(0)); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + + // The ordinary case: a held epoch from an executor that missed a push is BEHIND the record, + // and must not walk the epoch back to one the previous owner still holds. + let stale = BTreeMap::from([(shard(0), ShardEpoch(0))]); + assert!( + shard_state + .raise_epoch_floor(executor(2), &stale) + .is_empty() ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + + // A held epoch that matches is not a regression either. + let matching = BTreeMap::from([(shard(0), ShardEpoch(1))]); + assert!( + shard_state + .raise_epoch_floor(executor(2), &matching) + .is_empty() + ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + + // A held epoch naming a shard outside the cluster's count is ignored rather than recorded: + // an epoch there would fail the invariants. + let out_of_range = BTreeMap::from([(shard(9), ShardEpoch(7))]); + assert!( + shard_state + .raise_epoch_floor(executor(2), &out_of_range) + .is_empty() + ); + assert!(!shard_state.shard_epochs.contains_key(&shard(9))); + + // An unassigned shard can still have its floor restored, so the next assignment mints + // above it. + let unassigned = BTreeMap::from([(shard(2), ShardEpoch(4))]); + assert_eq!( + shard_state.raise_epoch_floor(executor(2), &unassigned), + vec![shard(2)] + ); + assert_eq!( + shard_state.shard_epochs.get(&shard(2)), + Some(&ShardEpoch(4)) + ); + assert_eq!(shard_state.epoch_for_shard(shard(2)), None); + assert!(shard_state.check_invariants().is_ok()); + assert_eq!( + shard_state.assign_shard(executor(2), shard(2)), + Some(ShardEpoch(5)) + ); + } + + #[test] + // A record near the top of the range is reachable one bounded jump at a time. Whatever a floor + // raise stores from there, `next_epoch_for` must still be able to mint past, so neither funnel + // stores anything above `u64::MAX - 2`. + fn an_out_of_range_epoch_is_ignored_rather_than_overflowing() { + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[1])]); + let near_the_top = ShardEpoch(u64::MAX - 4); + for shard_id in [shard(0), shard(1), shard(2)] { + shard_state.shard_epochs.insert(shard_id, near_the_top); + if let Some(entry) = shard_state.shard_assignments.get_mut(&shard_id) { + entry.epoch = near_the_top; + } + } assert!(shard_state.check_invariants().is_ok()); + + // A held epoch is stored as it is, on the holder's own shard and on an unassigned one. + for out_of_range in [u64::MAX, u64::MAX - 1] { + let held = BTreeMap::from([ + (shard(0), ShardEpoch(out_of_range)), + (shard(2), ShardEpoch(out_of_range)), + ]); + assert!( + shard_state.raise_epoch_floor(executor(1), &held).is_empty(), + "a held epoch of {out_of_range} must be ignored" + ); + } + // A report mints one past what it carries, so it runs out one epoch earlier. + for out_of_range in [u64::MAX, u64::MAX - 1, u64::MAX - 2] { + let fenced = BTreeMap::from([(shard(1), ShardEpoch(out_of_range))]); + assert!( + shard_state + .raise_epoch_floor_past(reporting_executor(), &fenced) + .is_empty(), + "a fenced epoch of {out_of_range} must be ignored, not minted past" + ); + } + for shard_id in [shard(0), shard(1), shard(2)] { + assert_eq!(shard_state.shard_epochs.get(&shard_id), Some(&near_the_top)); + } + assert!(shard_state.check_invariants().is_ok()); + + // The highest epoch a floor raise does store still leaves the shard one to move at. + let highest = BTreeMap::from([(shard(0), ShardEpoch(u64::MAX - 2))]); + assert_eq!( + shard_state.raise_epoch_floor(executor(1), &highest), + vec![shard(0)] + ); + assert_eq!( + shard_state.assign_shard(executor(2), shard(0)), + Some(ShardEpoch(u64::MAX - 1)) + ); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + // One held or fenced epoch cannot use up a shard's epochs: anything more than `MAX_EPOCH_JUMP` + // above the record is ignored, however it arrives, while a large but plausible jump - a state + // restored from an old backup - is still repaired. + fn an_epoch_implausibly_far_above_the_record_is_ignored() { + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[1])]); + let too_far = ShardEpoch(ShardLeaseState::MAX_EPOCH_JUMP + 1); + + let held = BTreeMap::from([(shard(0), too_far), (shard(2), too_far)]); + assert!(shard_state.raise_epoch_floor(executor(1), &held).is_empty()); + let fenced = BTreeMap::from([(shard(1), too_far)]); + assert!( + shard_state + .raise_epoch_floor_past(reporting_executor(), &fenced) + .is_empty() + ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(0))); + assert_eq!(shard_state.epoch_for_shard(shard(1)), Some(ShardEpoch(0))); + assert!(!shard_state.shard_epochs.contains_key(&shard(2))); + + let far_but_plausible = ShardEpoch(ShardLeaseState::MAX_EPOCH_JUMP); + let fenced = BTreeMap::from([(shard(1), far_but_plausible)]); + assert_eq!( + shard_state.raise_epoch_floor_past(reporting_executor(), &fenced), + vec![shard(1)] + ); + assert_eq!( + shard_state.epoch_for_shard(shard(1)), + Some(ShardEpoch(ShardLeaseState::MAX_EPOCH_JUMP + 1)) + ); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_held_epoch_on_another_executors_shard_moves_nothing() { + // Executor 1 holds shard 0 at epoch 0 in a state that lost history; executor 2 says it + // held it at epoch 9 before the loss. Nothing corroborates that, and believing it would + // let any executor push another one off its epoch. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[1])]); + let owner_of_shard_0 = |shard_state: &ShardLeaseState| { + shard_state + .shard_assignments + .get(&shard(0)) + .map(|entry| entry.executor_id) + }; + + let ahead = BTreeMap::from([(shard(0), ShardEpoch(9))]); + assert!( + shard_state + .raise_epoch_floor(executor(2), &ahead) + .is_empty() + ); + assert_eq!(owner_of_shard_0(&shard_state), Some(executor(1))); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(0))); + assert_eq!( + shard_state.shard_epochs.get(&shard(0)), + Some(&ShardEpoch(0)) + ); + + // If executor 2 did write rows at epoch 9 they refuse executor 1's writes, and the report + // of what the storage holds is what mints the owner past them. + assert_eq!( + shard_state.raise_epoch_floor_past(executor(1), &ahead), + vec![shard(0)] + ); + assert_eq!(owner_of_shard_0(&shard_state), Some(executor(1))); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(10))); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_fence_reported_by_the_assignee_at_its_own_epoch_is_minted_past() { + // The wiped-store collision. Executor 1 is assigned shard 0 at epoch 0 - all a fresh state + // can mint - and its write is refused by a row another executor still holds at that same + // epoch. Equality is ordinarily the loser of a shard move and moves nothing, but the + // reporter here IS the assignee: the storage refused the executor this state believes owns + // the shard, so somebody else holds the generation and it has to be minted past. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0])]); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(0))); + + let fenced = BTreeMap::from([(shard(0), ShardEpoch(0))]); + assert_eq!( + shard_state.raise_epoch_floor_past(executor(1), &fenced), + vec![shard(0)] + ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_fence_reported_at_the_record_by_anyone_else_moves_nothing() { + // The ordinary shard move, which must not churn: executor 2 took shard 0 over at epoch 1 + // and recorded it, so executor 1's refused write reports 1 while the state already says 1. + // The reporter is not the assignee, so this is the loser of the move, not a collision. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[])]); + shard_state.assign_shard(executor(2), shard(0)); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + + let fenced = BTreeMap::from([(shard(0), ShardEpoch(1))]); + assert!( + shard_state + .raise_epoch_floor_past(executor(1), &fenced) + .is_empty() + ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(1))); + } + + #[test] + fn a_shard_whose_epoch_cannot_advance_is_left_unassigned_rather_than_panicking() { + // `u64::MAX - 1` is the last epoch minted. The next owner change then has none, and the + // shard stays where it is: a panic would take the manager down and do it again on every + // retry. No floor raise stores an epoch this high, so it is planted on the fields. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[])]); + let last_epoch = ShardEpoch(u64::MAX - 1); + shard_state.shard_epochs.insert(shard(0), last_epoch); + shard_state + .shard_assignments + .get_mut(&shard(0)) + .expect("shard 0 is assigned") + .epoch = last_epoch; + assert!(shard_state.check_invariants().is_ok()); + assert_eq!( + shard_state.epoch_for_shard(shard(0)), + Some(ShardEpoch(u64::MAX - 1)) + ); + + assert_eq!(shard_state.next_epoch_for(executor(2), shard(0)), None); + assert_eq!(shard_state.assign_shard(executor(2), shard(0)), None); + assert_eq!( + shard_state + .shard_assignments + .get(&shard(0)) + .map(|entry| entry.executor_id), + Some(executor(1)), + "the shard keeps its owner rather than moving to an epoch that does not exist" + ); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_fenced_epoch_re_mints_every_owner_above_it_the_reporters_own_included() { + // Writes were refused by rows at epoch 3 on shards 0, 1 and 2, in a state that lost + // history. Nothing says who wrote those rows, so whoever reported them, executor 1's shard + // is minted past them like executor 2's: stamping 3 onto an entry could leave its owner on + // the rows' `(shard, epoch)`. + let mut shard_state = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[1])]); + let stored = BTreeMap::from([ + (shard(0), ShardEpoch(3)), + (shard(1), ShardEpoch(3)), + (shard(2), ShardEpoch(3)), + ]); + assert_eq!( + shard_state.raise_epoch_floor_past(reporting_executor(), &stored), + vec![shard(0), shard(1), shard(2)] + ); + for (shard_id, owner) in [(0, 1), (1, 2)] { + assert_eq!( + shard_state + .shard_assignments + .get(&shard(shard_id)) + .map(|entry| entry.executor_id), + Some(executor(owner)), + "a fenced epoch moved a shard" + ); + assert_eq!( + shard_state.epoch_for_shard(shard(shard_id)), + Some(ShardEpoch(4)) + ); + assert_eq!( + shard_state.shard_epochs.get(&shard(shard_id)), + Some(&ShardEpoch(4)) + ); + } + // An unassigned shard has no entry to put on the rows' epoch: its high-water takes it, and + // the next mint lands one past. + assert_eq!( + shard_state.shard_epochs.get(&shard(2)), + Some(&ShardEpoch(3)) + ); + assert_eq!( + shard_state.next_epoch_for(executor(1), shard(2)), + Some(ShardEpoch(4)) + ); + assert!(shard_state.check_invariants().is_ok()); + + // Reported again, every epoch is below the record: nothing moves a second time. + assert!( + shard_state + .raise_epoch_floor_past(reporting_executor(), &stored) + .is_empty() + ); + // At the record is the ordinary loser of a shard move. + assert!( + shard_state + .raise_epoch_floor_past( + reporting_executor(), + &BTreeMap::from([(shard(0), ShardEpoch(4))]) + ) + .is_empty() + ); + assert_eq!(shard_state.epoch_for_shard(shard(0)), Some(ShardEpoch(4))); + assert!(shard_state.check_invariants().is_ok()); + } + + #[test] + fn a_held_and_a_fenced_epoch_commute_unless_they_are_equal() { + // Executor 1 holds shard 0 at epoch 0. Each case applies a held epoch on shard 0 and a fenced + // epoch on it, in both orders, to its own copy of that state. + let base = shard_state_with(4, &[(1, 1, &[0]), (2, 2, &[1])]); + let epoch_of_shard_0 = |holder: u128, held: u64, fenced: u64, fenced_first: bool| { + let mut shard_state = base.clone(); + let held = BTreeMap::from([(shard(0), ShardEpoch(held))]); + let fenced = BTreeMap::from([(shard(0), ShardEpoch(fenced))]); + if fenced_first { + shard_state.raise_epoch_floor_past(reporting_executor(), &fenced); + shard_state.raise_epoch_floor(executor(holder), &held); + } else { + shard_state.raise_epoch_floor(executor(holder), &held); + shard_state.raise_epoch_floor_past(reporting_executor(), &fenced); + } + assert!(shard_state.check_invariants().is_ok()); + assert_eq!( + shard_state + .shard_assignments + .get(&shard(0)) + .map(|entry| entry.executor_id), + Some(executor(1)) + ); + shard_state + .epoch_for_shard(shard(0)) + .expect("shard 0 is assigned") + }; + + for fenced_first in [false, true] { + // A held epoch above the fenced epoch wins. + assert_eq!(epoch_of_shard_0(1, 5, 3, fenced_first), ShardEpoch(5)); + // A fenced epoch above the held one ends one past it. + assert_eq!(epoch_of_shard_0(1, 3, 5, fenced_first), ShardEpoch(6)); + // Another executor's held epoch moves nothing, so the fenced epoch alone mints the owner + // one past. + assert_eq!(epoch_of_shard_0(2, 3, 3, fenced_first), ShardEpoch(4)); + } + + // The pair that does not commute: the owner's own held epoch equal to the fenced epoch. Held + // first is what happens when the held epoch reached the state in an earlier request - the + // fenced epoch is then at the record, the equality nothing can tell from a shard move. + assert_eq!(epoch_of_shard_0(1, 3, 3, false), ShardEpoch(3)); + // Fenced first is the order a renewal carrying both uses, which is why it uses it: the + // owner ends one past the rows' epoch rather than on it. + assert_eq!(epoch_of_shard_0(1, 3, 3, true), ShardEpoch(4)); } #[test] diff --git a/golem-shard-manager/src/sharding/shard_management.rs b/golem-shard-manager/src/sharding/shard_management.rs index 2608f584d0..69f8234ec5 100644 --- a/golem-shard-manager/src/sharding/shard_management.rs +++ b/golem-shard-manager/src/sharding/shard_management.rs @@ -208,6 +208,19 @@ impl ShardManagement { Ok(shard_management) } + /// Registers the executor instance `executor_id`, listening at `addr`, carrying nothing over + /// from an earlier instance: [`Self::register_executor_with_previous_epochs`] with an empty + /// set. + pub async fn register_executor( + &self, + executor_id: ExecutorId, + addr: ExecutorAddr, + pod_name: Option, + ) -> Result { + self.register_executor_with_previous_epochs(executor_id, addr, pod_name, BTreeMap::new()) + .await + } + /// Registers the executor instance `executor_id`, listening at `addr`, and grants it a lease. /// /// The lease is written before this returns, so an acknowledged registration is a durable one: @@ -217,13 +230,28 @@ impl ShardManagement { /// the same registration, so the same id at the same address refreshes that lease and returns /// it; it neither creates a second lease nor counts as a replacement. A *different* id at a /// known address is a restarted instance, and inherits its predecessor's shards. - pub async fn register_executor( + /// + /// `previous_epochs` is the set the executor held under an earlier id that was answered + /// [`ShardManagerError::ShardLeaseNotFound`], and is empty on a process's first registration. + /// It is evidence and never assigns a shard. Against a store that kept its history it is at or + /// below the record and moves nothing. A store that was wiped or replaced no longer lists the + /// earlier id, so it refused the renewal that would have repaired it, and this is the moment + /// the executor can tell it what it forgot - see [`ShardLeaseState::raise_epoch_floor`]. An + /// unassigned shard's high-water rises to the held epoch, so the loop's first mint lands one past + /// it; a shard another executor already holds is left alone. + pub async fn register_executor_with_previous_epochs( &self, executor_id: ExecutorId, addr: ExecutorAddr, pod_name: Option, + previous_epochs: BTreeMap, ) -> Result { - debug!(executor_id = %executor_id, addr = %addr, "Registering executor"); + debug!( + executor_id = %executor_id, + addr = %addr, + previous_shards = previous_epochs.len(), + "Registering executor" + ); let now = Utc::now(); let lease_ttl = self.lease_ttl; @@ -232,6 +260,20 @@ impl ShardManagement { let already_known = shard_state.has_executor(executor_id); let replaced = shard_state.add_executor(executor_id, addr, pod_name, now, lease_ttl); + + // After `add_executor`, so a replaced predecessor's shards are already this + // executor's and its held epochs on them count; ahead of the grant, so the grant + // carries the repaired epochs. + let raised = shard_state.raise_epoch_floor(executor_id, &previous_epochs); + if !raised.is_empty() { + warn!( + executor_id = %executor_id, + raised_shards = raised.iter().join(", "), + "Registration carried epochs ahead of the stored state; raising them. \ + The shard state has lost history - it was wiped, replaced or restored" + ); + } + let pending = shard_state.lease_grant_for(executor_id).ok_or_else(|| { ShardManagerError::Internal(format!( "executor {executor_id} holds no lease right after being registered" @@ -264,7 +306,8 @@ impl ShardManagement { self.updates.lock().await.retry_full_assignment(executor_id); } else if already_known { // A retried registration. The lease clock restarts and the same set comes back; the - // shards are not touched, so their epochs do not move. + // shards are not touched, and the epochs it carries were already applied by the attempt + // that stored the lease, so their epochs do not move. info!(executor_id = %executor_id, addr = %addr, "Executor lease refreshed"); } else { info!(executor_id = %executor_id, addr = %addr, "Executor added"); @@ -274,9 +317,19 @@ impl ShardManagement { Ok(ack) } + /// [`Self::renew_shard_lease_with_fenced_epochs`] reporting no fenced epochs. + pub async fn renew_shard_lease( + &self, + executor_id: ExecutorId, + held: BTreeMap, + ) -> Result { + self.renew_shard_lease_with_fenced_epochs(executor_id, held, BTreeMap::new()) + .await + } + /// Extends `executor_id`'s shard lease and returns the manager's set for it. /// - /// The claimed shards are what the executor believes it holds. A claim that does not match - + /// `held` is the set the executor believes it holds. A set that does not match - /// a shard not assigned to this executor, or assigned at another epoch - is renewed all the /// same and logged: the grant returned carries the manager's set, which the executor adopts, /// so the renewal is the guaranteed second delivery of a push that was lost. Only a lease the @@ -284,21 +337,34 @@ impl ShardManagement { /// that refusal stores nothing: the mutation runs on a clone that is dropped when the closure /// refuses. /// - /// A renewal never advances an epoch: the epoch is an ownership generation, and moving it on a - /// renewal would make a lost response permanently fatal for a shard the executor still owns. + /// A renewal never mints a new epoch against a record that is intact: the epoch is an + /// ownership generation, and moving it on a renewal would make a lost response permanently + /// fatal for a shard the executor still owns. The exception is a store that lost history - + /// see [`ShardLeaseState::raise_epoch_floor`]: a held epoch ahead of the record restores the + /// holder's own epochs, and moves nothing on a shard another executor owns. + /// + /// `fenced` is what the oplog writes this executor was refused found on the rows - see + /// [`ShardLeaseState::raise_epoch_floor_past`]. Ahead of the record it re-mints every owner of + /// the shard one past it, this executor included; at or below the record, which is the + /// ordinary loser of a shard move, it moves nothing. It is applied before `held`, so a + /// held epoch equal to a fenced epoch cannot leave this executor on the rows' `(shard, epoch)`. /// /// Leases that have already lapsed are reaped *before* this one is looked up, so an executor /// whose lease expired while its renewal was in flight is told - /// [`ShardManagerError::ShardLeaseNotFound`] rather than silently resurrected. This does not - /// notify the loop; the shards that reaping freed are picked up by the next tick. - pub async fn renew_shard_lease( + /// [`ShardManagerError::ShardLeaseNotFound`] rather than silently resurrected. The loop is + /// notified only when a fenced epoch re-minted another executor's shard, so that + /// owner is pushed its new epoch instead of waiting for its own renewal; the shards that + /// reaping freed are picked up by the next tick. + pub async fn renew_shard_lease_with_fenced_epochs( &self, executor_id: ExecutorId, - claimed: BTreeMap, + held: BTreeMap, + fenced: BTreeMap, ) -> Result { debug!( executor_id = %executor_id, - claimed_shards = claimed.len(), + held_shards = held.len(), + fenced_shards = fenced.len(), "Renewing shard lease" ); let now = Utc::now(); @@ -321,19 +387,19 @@ impl ShardManagement { }) .await?; - let (pending, stored_at) = self + let ((pending, re_minted_owners), stored_at) = self .persist_for_request(move |shard_state| { if !shard_state.has_executor(executor_id) { return Err(ShardManagerError::ShardLeaseNotFound { executor_id }); } - // The claim is what the executor believes it holds, not a condition of the renewal. - // A claim that does not match is an executor that missed a push, and the grant this + // The held set is what the executor believes it holds, not a condition of the renewal. + // A set that does not match is an executor that missed a push, and the grant this // returns is the manager's set, which the executor adopts - so the renewal is the // guaranteed second delivery path for a push that was lost. Refusing it would only // hold the executor on a picture the manager already knows is wrong. The mismatch is // logged because it is the one signal that pushes to this executor are not landing. - let mismatched: Vec = claimed + let mismatched: Vec = held .iter() .filter(|(shard_id, provided)| { shard_state @@ -349,10 +415,46 @@ impl ShardManagement { warn!( executor_id = %executor_id, mismatched_shards = mismatched.iter().join(", "), - "Shard lease claim does not match the manager's view; renewing and correcting" + "Shard lease held set does not match the manager's view; renewing and correcting" + ); + } + + // Both repairs run ahead of the renewal, so the grant read below carries the repaired + // epochs, and both only ever fire when the stored state is behind the cluster it is + // managing. + // + // The fenced epochs go first. A fence only says somebody wrote rows at that epoch while + // this executor asserted a lower one. If this request's held epoch equals it, applying the + // held epoch first would record it and leave this executor on the rows' `(shard, epoch)`; + // applied first, a fenced epoch at or above the held one ends one past it, and a higher + // held epoch still wins. They reach a state that was wiped or replaced as well, because the + // executor keeps reporting them until a renewal under its re-registered id is granted. + let re_minted = shard_state.raise_epoch_floor_past(executor_id, &fenced); + // A re-minted shard another executor owns: this renewal's grant does not reach that + // owner, so it has to be pushed the new epoch. + let re_minted_owners = owners_re_minted_by(shard_state, &re_minted, executor_id); + if !re_minted.is_empty() { + warn!( + executor_id = %executor_id, + re_minted_shards = re_minted.iter().join(", "), + re_minted_owners = re_minted_owners.iter().join(", "), + "Fenced oplog writes reported epochs ahead of the stored state; re-minting above \ + them. The shard state has lost history - it was wiped, replaced or restored" ); } + // A held epoch ahead of the record reaches only a state that still lists this executor: one + // that was wiped or replaced refused the renewal above, and is repaired by the + // re-registration that follows it. + let raised = shard_state.raise_epoch_floor(executor_id, &held); + if !raised.is_empty() { + warn!( + executor_id = %executor_id, + raised_shards = raised.iter().join(", "), + "Shard lease carried held epochs ahead of the stored state; raising them. \ + The shard state has lost history - it was restored from a backup" + ); + } if !shard_state.renew_lease(executor_id, now, lease_ttl) { return Err(ShardManagerError::Internal(format!( "executor {executor_id} holds no lease right after it was found" @@ -361,13 +463,30 @@ impl ShardManagement { // Read off the mutated clone, so the grant this returns is exactly the state that is // about to be stored - never a state that a failed write then rolls back. - shard_state.lease_grant_for(executor_id).ok_or_else(|| { - ShardManagerError::Internal(format!( - "executor {executor_id} holds no lease right after it was renewed" - )) - }) + shard_state + .lease_grant_for(executor_id) + .map(|pending| (pending, re_minted_owners)) + .ok_or_else(|| { + ShardManagerError::Internal(format!( + "executor {executor_id} holds no lease right after it was renewed" + )) + }) }) .await?; + + if !re_minted_owners.is_empty() { + // After the persist, so the pass pushes a stored epoch. Until the owner adopts it, its + // writes on the re-minted shards carry the epoch the store forgot and the holder's + // oplog rows refuse them; its own renewal could be a third of a lease away. + { + let mut updates = self.updates.lock().await; + for owner in &re_minted_owners { + updates.retry_full_assignment(*owner); + } + } + self.change.notify_one(); + } + // Read off the clone before its revision was bumped; stamped with the revision the state // was then stored at, so the grant names exactly the persisted state it describes. Ok(pending.stamp(stored_at)) @@ -375,9 +494,9 @@ impl ShardManagement { /// Releases `executor_id`'s shard lease on a graceful shutdown. /// - /// Lenient by contract: an executor the manager does not know, and a `claimed` set that no + /// Lenient by contract: an executor the manager does not know, and a `held` set that no /// longer matches what it records, are both `Ok`. A shutdown must never fail on bookkeeping, - /// so the claim is only logged. + /// so a mismatch is only logged. /// /// Removing the lease drops its shard assignments and leaves them on `pending_rebalance`. /// This does not notify the loop: the next tick re-homes them, which bounds a graceful @@ -385,11 +504,11 @@ impl ShardManagement { pub async fn deregister_executor( &self, executor_id: ExecutorId, - claimed: BTreeMap, + held: BTreeMap, ) -> Result<(), ShardManagerError> { debug!( executor_id = %executor_id, - claimed_shards = claimed.len(), + held_shards = held.len(), "Deregistering executor" ); @@ -402,7 +521,7 @@ impl ShardManagement { return Ok(()); } - let stale: Vec = claimed + let stale: Vec = held .iter() .filter(|(shard_id, epoch)| { shard_state @@ -453,9 +572,10 @@ impl ShardManagement { threshold: f64, ) -> Result<(), ShardManagerError> { // The timer is what makes the pull-based half of the lease protocol work. `RenewShardLease` - // and `Deregister` never wake the loop - they only leave shards behind - so without a tick - // an expired lease in a quiet cluster would never be reaped and a graceful shutdown's - // shards would never be re-homed. A third of the lease is the same cadence the executors + // and `Deregister` only leave shards behind - the one wake-up is a renewal whose held epochs + // re-minted another executor's shard after the store lost history, which the loop must + // push to that owner - so without a tick an expired lease in a quiet cluster would never be + // reaped and a graceful shutdown's shards would never be re-homed. A third of the lease is the same cadence the executors // renew at, and it is derived rather than configured so there is no second knob to keep // consistent with the lease duration. let tick_period = shard_lease::renewal_interval(self.lease_ttl); @@ -841,6 +961,22 @@ impl ShardManagement { } } +/// The executors other than `holder` that own a shard in `raised`: the owners +/// [`ShardLeaseState::raise_epoch_floor`] re-minted one past the held epochs. The holder's grant does +/// not reach them, so each is owed a push of its new epoch. +fn owners_re_minted_by( + shard_state: &ShardLeaseState, + raised: &[ShardId], + holder: ExecutorId, +) -> BTreeSet { + raised + .iter() + .filter_map(|shard_id| shard_state.shard_assignments.get(shard_id)) + .map(|entry| entry.executor_id) + .filter(|owner| *owner != holder) + .collect() +} + /// The full-replace payloads for `executor_ids`, read off `shard_state`. /// /// The one place a push is built, so the never-zero `number_of_shards` guard has a single home. diff --git a/golem-shard-manager/src/sharding/worker_executor.rs b/golem-shard-manager/src/sharding/worker_executor.rs index 3f96e33fb4..0f09e31c8e 100644 --- a/golem-shard-manager/src/sharding/worker_executor.rs +++ b/golem-shard-manager/src/sharding/worker_executor.rs @@ -269,6 +269,7 @@ impl WorkerExecutorServiceDefault { .collect(), revision: assignment.revision.0, number_of_shards: assignment.number_of_shards as u32, + incarnation_id: super::model::incarnation_id(), }; let assign_shards_response = timeout( @@ -317,6 +318,7 @@ impl WorkerExecutorServiceDefault { .map(|shard_id| shard_id.into()) .collect(), revision: revision.0, + incarnation_id: super::model::incarnation_id(), }; let revoke_shards_response = timeout( diff --git a/golem-shard-manager/tests/etcd_backed/mod.rs b/golem-shard-manager/tests/etcd_backed/mod.rs index 7b61f46896..b221d0ea84 100644 --- a/golem-shard-manager/tests/etcd_backed/mod.rs +++ b/golem-shard-manager/tests/etcd_backed/mod.rs @@ -17,11 +17,33 @@ mod distributed_startup; mod leader_election; -mod persistence; +pub(crate) mod persistence; mod proxy; +mod service; +use crate::etcd_backed::persistence::GetRoutingTablePersistence; use golem_test_framework::components::etcd::docker_etcd::DockerEtcd; use std::sync::Arc; -use test_r::inherit_test_dep; +use test_r::{inherit_test_dep, test_dep}; inherit_test_dep!(Arc); + +// The persistence-backend fixtures live here rather than in one of the modules below, because +// `define_matrix_dimension!` emits a module-local helper: a dimension can only be declared in the +// module whose tests use it, and every such module inherits these deps from their common parent. +// Their constructors are in `persistence`. + +#[test_dep(scope = Shared, tagged_as = "sqlite")] +async fn sqlite_persistence() -> Arc { + persistence::sqlite_persistence().await +} + +#[test_dep(scope = Shared, tagged_as = "postgres")] +async fn postgres_persistence() -> Arc { + persistence::postgres_persistence().await +} + +#[test_dep(scope = PerWorker, tagged_as = "etcd")] +async fn etcd_persistence(etcd: &Arc) -> Arc { + persistence::etcd_persistence(etcd).await +} diff --git a/golem-shard-manager/tests/etcd_backed/persistence.rs b/golem-shard-manager/tests/etcd_backed/persistence.rs index 778fabc1a2..7f784353bd 100644 --- a/golem-shard-manager/tests/etcd_backed/persistence.rs +++ b/golem-shard-manager/tests/etcd_backed/persistence.rs @@ -30,11 +30,14 @@ use std::net::{IpAddr, Ipv4Addr}; use std::sync::Arc; use std::time::Duration; use tempfile::TempDir; -use test_r::{define_matrix_dimension, inherit_test_dep, test, test_dep}; +use test_r::{define_matrix_dimension, inherit_test_dep, test}; use url::Url; use uuid::Uuid; inherit_test_dep!(Arc); +inherit_test_dep!(#[tagged_as("sqlite")] Arc); +inherit_test_dep!(#[tagged_as("postgres")] Arc); +inherit_test_dep!(#[tagged_as("etcd")] Arc); /// One `executor_leases` row, as a person reading the table would see it. #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] @@ -59,7 +62,7 @@ type RawLeaseRow = ( /// What the local-mode mirror tables hold, normalized for comparison. #[derive(Debug, PartialEq, Eq)] -struct MirrorSnapshot { +pub(crate) struct MirrorSnapshot { leases: Vec, /// `(shard_id, executor_id, epoch)` per `shard_assignments` row, sorted. assignments: Vec<(i32, Uuid, i64)>, @@ -124,15 +127,17 @@ impl MirrorSnapshot { } } -const LEASE_TTL: Duration = Duration::from_secs(60); -const NUMBER_OF_SHARDS: usize = 16; +/// Shared with `service.rs`: the fixtures below construct every persistence with this count, and a +/// persistence refuses to serve a state that disagrees with it, so the tests must use the same one. +pub(crate) const LEASE_TTL: Duration = Duration::from_secs(60); +pub(crate) const NUMBER_OF_SHARDS: usize = 16; /// A place where shard lease state can be stored. /// /// Hands out any number of independent clients over the *same* underlying store, which is what /// the compare-and-swap tests need: two clients must see each other's writes. #[async_trait] -trait PersistenceStore: std::fmt::Debug + Send + Sync { +pub(crate) trait PersistenceStore: std::fmt::Debug + Send + Sync { async fn connect(&self) -> Arc; /// Writes something unrelated to the shard state on the same backend, so a test can check @@ -148,7 +153,7 @@ trait PersistenceStore: std::fmt::Debug + Send + Sync { /// Creates isolated stores: two stores never see each other's data. #[async_trait] -trait GetRoutingTablePersistence: std::fmt::Debug + Send + Sync { +pub(crate) trait GetRoutingTablePersistence: std::fmt::Debug + Send + Sync { async fn new_store(&self) -> Arc; /// For the tests that only need a single client over a fresh store. @@ -494,21 +499,22 @@ async fn fenced_etcd_persistence( (store, leader_key, persistence) } -#[test_dep(scope = Shared, tagged_as = "sqlite")] -async fn sqlite_persistence() -> Arc { +/// The dimension's fixtures live in [`super`], so that this module and `service.rs` can both +/// inherit them; these are the constructors behind them. +pub(crate) async fn sqlite_persistence() -> Arc { let temp_dir = TempDir::new().expect("Cannot create temp dir"); Arc::new(SqliteRoutingTablePersistence { temp_dir }) } -#[test_dep(scope = Shared, tagged_as = "postgres")] -async fn postgres_persistence() -> Arc { +pub(crate) async fn postgres_persistence() -> Arc { let unique_network_id = Uuid::new_v4().to_string(); let postgres = DockerPostgresRdb::new(&unique_network_id, false).await; Arc::new(PostgresRoutingTablePersistence { postgres }) } -#[test_dep(scope = PerWorker, tagged_as = "etcd")] -async fn etcd_persistence(etcd: &Arc) -> Arc { +pub(crate) async fn etcd_persistence( + etcd: &Arc, +) -> Arc { Arc::new(EtcdRoutingTablePersistenceFactory { etcd: etcd.clone() }) } diff --git a/golem-shard-manager/tests/etcd_backed/service.rs b/golem-shard-manager/tests/etcd_backed/service.rs new file mode 100644 index 0000000000..63377b69b7 --- /dev/null +++ b/golem-shard-manager/tests/etcd_backed/service.rs @@ -0,0 +1,339 @@ +// Copyright 2024-2026 Golem Cloud +// +// Licensed under the Golem Source License v1.1 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://license.golem.cloud/LICENSE +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! [`ShardManagement`] driven against the real persistence backends. +//! +//! `shard_management.rs` covers the service's behaviour in depth, but every one of those tests runs +//! against an in-memory double. What is only provable here is that the same lifecycle survives a +//! real store: that a registration is durable, that a renewal moves the stored expiry rather than +//! only the returned grant, that a lapsed lease is reclaimed from state that was read back out of +//! the backend, and that a write refused by the compare-and-swap stops the loop on every backend. +//! +//! Every test is multiplied over the `persistence` dimension declared below, so each one runs as +//! `_sqlite`, `_postgres` and `_etcd`. The fixtures behind the dimension live in +//! [`super`]; the etcd case shares the per-worker container, which is why this module sits under +//! `etcd_backed` and runs in its sequential suite. + +use crate::etcd_backed::persistence::{ + GetRoutingTablePersistence, LEASE_TTL, NUMBER_OF_SHARDS, PersistenceStore, +}; +use crate::shard_management::{TestHealthCheck, TestWorkerExecutors, executor, pod, shard_ids}; +use golem_common::model::ShardId; +use golem_shard_manager::{ + ExecutorAddr, RoutingTablePersistence, ShardEpoch, ShardLeaseState, ShardManagement, + ShardManagerError, +}; +use golem_test_framework::components::etcd::docker_etcd::DockerEtcd; +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; +use std::time::Duration; +use test_r::{define_matrix_dimension, inherit_test_dep, test}; +use tokio::task::JoinSet; +use tokio::time::Instant; + +inherit_test_dep!(Arc); +inherit_test_dep!(#[tagged_as("sqlite")] Arc); +inherit_test_dep!(#[tagged_as("postgres")] Arc); +inherit_test_dep!(#[tagged_as("etcd")] Arc); + +// Declared per module on purpose: the macro emits a module-local helper that the generated test +// cases call unqualified, so it cannot be shared from a sibling. +define_matrix_dimension!(persistence: Arc -> "sqlite", "postgres", "etcd"); + +/// Short enough that the loop's tick - a third of it - reclaims a lapsed lease inside a test, long +/// enough that the registration and its rebalance cannot themselves be outrun by the expiry. +const RECLAIM_LEASE_TTL: Duration = Duration::from_secs(3); + +/// A `ShardManagement` over `persistence`, plus the join set holding its loop. +async fn start( + persistence: Arc, + worker_executors: Arc, + lease_ttl: Duration, +) -> (ShardManagement, JoinSet>) { + let mut join_set = JoinSet::new(); + let shard_management = ShardManagement::new( + persistence, + worker_executors, + Arc::new(TestHealthCheck::all_healthy()), + 0.0, + lease_ttl, + NUMBER_OF_SHARDS, + &mut join_set, + ) + .await + .expect("the shard management loop should have started over a real store"); + (shard_management, join_set) +} + +/// A second client over the same store, for reading what the service actually persisted rather +/// than what its own cache holds. +/// +/// Taken once per test and reused: `connect` is expensive on every backend - two pools on SQLite, a +/// connection and a round trip on Postgres, and on etcd two clients plus a leader key that is +/// written and never deleted - so connecting per poll would litter the shared etcd server. +async fn client(store: &Arc) -> Arc { + store.connect().await +} + +async fn stored(reader: &Arc) -> ShardLeaseState { + reader + .read() + .await + .expect("reading the stored shard lease state should succeed") + .0 +} + +/// Polls the store until `done` accepts the state it holds. +async fn wait_for_stored( + reader: &Arc, + what: &str, + done: impl Fn(&ShardLeaseState) -> bool, +) -> ShardLeaseState { + let start = Instant::now(); + loop { + let shard_state = stored(reader).await; + if done(&shard_state) { + return shard_state; + } + if start.elapsed() > Duration::from_secs(10) { + panic!("timed out waiting for {what}; stored state: {shard_state}"); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } +} + +fn all_shards() -> BTreeSet { + shard_ids(&(0..NUMBER_OF_SHARDS as i64).collect::>()) +} + +fn holds_every_shard(shard_state: &ShardLeaseState) -> bool { + shard_state.shards_for_executor(executor(1)) == Some(all_shards()) +} + +#[test] +#[tracing::instrument] +// A registration is only useful if it outlives the process that served it: the next leader reads +// its state from the store, not from the handler that granted the lease. +async fn a_registration_is_durable( + #[dimension(persistence)] persistence: &Arc, +) { + let store = persistence.new_store().await; + let reader = client(&store).await; + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, mut join_set) = + start(client(&store).await, worker_executors.clone(), LEASE_TTL).await; + + let registering = pod(1, 9000); + let ack = shard_management + .register_executor( + executor(1), + ExecutorAddr::from(registering), + Some("worker-executor-0".into()), + ) + .await + .expect("the registration should have been persisted"); + assert_eq!(ack.number_of_shards, NUMBER_OF_SHARDS); + + // The whole cluster is one executor, so the rebalance that follows gives it every shard. + let shard_state = wait_for_stored( + &reader, + "the registered executor to hold every shard", + holds_every_shard, + ) + .await; + + assert_eq!(shard_state.executor_count(), 1); + assert_eq!( + shard_state.executor_for_addr(registering.into()), + Some(executor(1)), + "the address the executor registered under was not stored" + ); + assert!(shard_state.get_unassigned_shards().is_empty()); + assert!( + shard_state.check_invariants().is_ok(), + "the stored state broke its own invariants: {shard_state}" + ); + + join_set.abort_all(); +} + +#[test] +#[tracing::instrument] +// The grant a renewal returns is read off the state it just persisted, so the stored expiry has to +// move with it. A renewal that only extended the reply would leave the next leader reclaiming a +// lease the executor believes it holds. +async fn a_renewal_extends_the_stored_lease( + #[dimension(persistence)] persistence: &Arc, +) { + let store = persistence.new_store().await; + let reader = client(&store).await; + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, mut join_set) = + start(client(&store).await, worker_executors.clone(), LEASE_TTL).await; + + shard_management + .register_executor(executor(1), ExecutorAddr::from(pod(1, 9000)), None) + .await + .expect("the registration should have been persisted"); + let before = wait_for_stored(&reader, "the initial assignment", holds_every_shard).await; + let held: BTreeMap = before + .shard_assignments + .iter() + .filter(|(_, entry)| entry.executor_id == executor(1)) + .map(|(shard_id, entry)| (*shard_id, entry.epoch)) + .collect(); + let expiry_before = before.executor_leases[&executor(1)].expires_at; + + let grant = shard_management + .renew_shard_lease(executor(1), held.clone()) + .await + .expect("held epochs matching the manager's set should have been renewed"); + + assert_eq!( + grant.shard_epochs, held, + "the renewal moved an epoch for a shard the executor still owns" + ); + + let after = stored(&reader).await; + assert!( + after.executor_leases[&executor(1)].expires_at > expiry_before, + "the renewal extended the returned grant but not the stored lease" + ); + assert_eq!(after.shards_for_executor(executor(1)), Some(all_shards())); + + join_set.abort_all(); +} + +#[test] +#[tracing::instrument] +// Reclamation is driven by the loop's timer against state it reads back from the store. Nothing +// else happens here - no deregistration, no failed push - so only the tick can notice the expiry. +async fn a_lapsed_lease_is_reclaimed_from_the_store( + #[dimension(persistence)] persistence: &Arc, +) { + let store = persistence.new_store().await; + let reader = client(&store).await; + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, mut join_set) = start( + client(&store).await, + worker_executors.clone(), + RECLAIM_LEASE_TTL, + ) + .await; + + shard_management + .register_executor(executor(1), ExecutorAddr::from(pod(1, 9000)), None) + .await + .expect("the registration should have been persisted"); + + // Either state is a legitimate observation: on a machine slow enough for the lease to lapse + // before the first poll, the reclamation this test is about has simply already happened. + // Requiring the assignment here would turn that into a timeout blamed on the wrong step. + wait_for_stored( + &reader, + "the initial assignment, or the lease already reclaimed", + |s| holds_every_shard(s) || s.executor_count() == 0, + ) + .await; + + // Nothing renews it from here. + let reclaimed = wait_for_stored(&reader, "the lapsed lease to be reclaimed", |s| { + s.executor_count() == 0 + }) + .await; + + assert_eq!( + reclaimed.get_unassigned_shards(), + all_shards(), + "the lapsed lease was reclaimed but its shards were not released" + ); + assert!( + reclaimed.shard_assignments.is_empty(), + "an assignment survived the executor that held it: {reclaimed}" + ); + assert!( + reclaimed.check_invariants().is_ok(), + "the stored state broke its own invariants: {reclaimed}" + ); + + join_set.abort_all(); +} + +#[test] +#[tracing::instrument] +// A write refused by the compare-and-swap is what the stored revision exists for, and the service +// does not retry it: the revision it cached is its fencing token, so the write is reported to the +// caller and the loop stops rather than reapplying anything on top of the winner's state. +// +// The second writer here is another client over the same store, not a second shard manager - on +// etcd it mints its own leader key, so both writers pass their own fence and what is exercised is +// the revision check, not leadership. Two real shard managers cannot both hold the election key. +async fn a_write_that_lost_the_race_is_refused( + #[dimension(persistence)] persistence: &Arc, +) { + let store = persistence.new_store().await; + let reader = client(&store).await; + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, mut join_set) = + start(client(&store).await, worker_executors.clone(), LEASE_TTL).await; + + shard_management + .register_executor(executor(1), ExecutorAddr::from(pod(1, 9000)), None) + .await + .expect("the registration should have been persisted"); + wait_for_stored(&reader, "the initial assignment", holds_every_shard).await; + + // The other writer commits, which advances the stored revision. The content is deliberately + // unchanged: what the service's next write collides with is the revision, nothing else. + let other = client(&store).await; + let (state, revision) = other + .read() + .await + .expect("the second client should be able to read the store"); + other + .write(&state, revision) + .await + .expect("the second client should win the race it is the only entrant in"); + + let refused = shard_management + .register_executor(executor(2), ExecutorAddr::from(pod(2, 9001)), None) + .await; + + assert!( + matches!(refused, Err(ShardManagerError::ConcurrentModification)), + "expected the losing write to be refused as a concurrent modification, got {refused:?}" + ); + + // Read before the loop is awaited below: this is the state the winner left behind, and nothing + // of the refused registration may appear in it. + let after = stored(&reader).await; + assert_eq!(after.executor_count(), 1); + assert_eq!(after.executor_for_addr(pod(2, 9001).into()), None); + + // The refusal is a fail-stop, not a retry: a leader that lost its revision must not go on + // commanding executors. Same shape as the fail-stop assertions in `shard_management.rs`. + let outcome = tokio::time::timeout(Duration::from_secs(5), join_set.join_next()) + .await + .expect("the shard management loop should have stopped after its write was refused") + .expect("the loop task should exist") + .expect("the loop task should not panic"); + let loop_err = outcome.expect_err("the loop must end with the refused write"); + assert!( + matches!( + loop_err.downcast_ref::(), + Some(ShardManagerError::ConcurrentModification) + ), + "the loop ended, but not with the concurrent modification that ended it: {loop_err:#}" + ); +} diff --git a/golem-shard-manager/tests/shard_management.rs b/golem-shard-manager/tests/shard_management.rs index 3ccae105e2..fbab79d0a5 100644 --- a/golem-shard-manager/tests/shard_management.rs +++ b/golem-shard-manager/tests/shard_management.rs @@ -193,7 +193,7 @@ impl RoutingTablePersistence for TestPersistence { } #[derive(Clone, Default)] -struct TestWorkerExecutors { +pub(crate) struct TestWorkerExecutors { local_assignments: Arc>>>, failed_assignments: Arc>>, failed_revocations: Arc>>, @@ -212,11 +212,11 @@ struct TestWorkerExecutors { /// The fan-out is exactly when an executor's own renewal timer can fire, and what that /// renewal is told is what decides whether a shard it has just been revoked comes back. #[allow(clippy::type_complexity)] - renew_during_revoke: Arc>>, + renew_during_revoke: Arc>>, grant_during_revoke: Arc>>, } -type Claim = BTreeMap; +type HeldEpochs = BTreeMap; /// A revoke as the double received it: the pod, the shards, and the revision it carried. type RecordedRevoke = (Pod, BTreeSet, ShardLeaseRevision); @@ -228,7 +228,7 @@ impl TestWorkerExecutors { .insert(pod, shard_ids.iter().copied().map(ShardId::new).collect()); } - async fn local_assignment(&self, pod: Pod) -> BTreeSet { + pub(crate) async fn local_assignment(&self, pod: Pod) -> BTreeSet { self.local_assignments .lock() .await @@ -250,9 +250,9 @@ impl TestWorkerExecutors { &self, shard_management: ShardManagement, executor_id: ExecutorId, - claimed: Claim, + held: HeldEpochs, ) { - *self.renew_during_revoke.lock().await = Some((shard_management, executor_id, claimed)); + *self.renew_during_revoke.lock().await = Some((shard_management, executor_id, held)); } async fn grant_served_during_revoke(&self) -> Option { @@ -341,9 +341,9 @@ impl WorkerExecutorService for TestWorkerExecutors { // Serve an executor's renewal in the middle of the fan-out, which is when a real one // would arrive, and keep what it was granted for the test to look at. let armed = self.renew_during_revoke.lock().await.take(); - if let Some((shard_management, executor_id, claimed)) = armed { + if let Some((shard_management, executor_id, held)) = armed { let grant = shard_management - .renew_shard_lease(executor_id, claimed) + .renew_shard_lease(executor_id, held) .await .expect("the renewal served during the fan-out should have been granted"); *self.grant_during_revoke.lock().await = Some(grant); @@ -361,7 +361,7 @@ impl WorkerExecutorService for TestWorkerExecutors { } #[derive(Clone, Debug)] -struct TestHealthCheck { +pub(crate) struct TestHealthCheck { healthy: Arc>>, /// Pod whose check never answers, standing in for an executor that went silent. never_answers: Option, @@ -371,7 +371,7 @@ struct TestHealthCheck { } impl TestHealthCheck { - fn all_healthy() -> Self { + pub(crate) fn all_healthy() -> Self { Self { healthy: Arc::new(Mutex::new(HashMap::new())), never_answers: None, @@ -402,14 +402,14 @@ impl HealthCheck for TestHealthCheck { } } -fn pod(last_octet: u8, port: u16) -> Pod { +pub(crate) fn pod(last_octet: u8, port: u16) -> Pod { Pod { ip: IpAddr::V4(Ipv4Addr::new(10, 0, 0, last_octet)), port, } } -fn executor(idx: u128) -> ExecutorId { +pub(crate) fn executor(idx: u128) -> ExecutorId { ExecutorId(Uuid::from_u128(idx)) } @@ -447,12 +447,12 @@ fn shards_at(shard_state: &ShardLeaseState, pod: Pod) -> BTreeSet { .expect("executor should hold a lease") } -fn shard_ids(ids: &[i64]) -> BTreeSet { +pub(crate) fn shard_ids(ids: &[i64]) -> BTreeSet { ids.iter().copied().map(ShardId::new).collect() } -/// The claim an executor holding exactly what `shard_state` records for it would send. -fn claim_of( +/// The held epochs an executor holding exactly what `shard_state` records for it would send. +fn epochs_of( shard_state: &ShardLeaseState, executor_id: ExecutorId, ) -> BTreeMap { @@ -806,14 +806,14 @@ async fn a_renewal_served_during_the_revoke_fan_out_does_not_hand_the_shard_back ) .await; - // The claim is what the losing executor still believes it holds - every shard, including the + // The held set is what the losing executor still believes it holds - every shard, including the // two about to move. let before = persistence.latest().await; worker_executors .renew_during_next_revoke( shard_management.clone(), executor(1), - claim_of(&before, executor(1)), + epochs_of(&before, executor(1)), ) .await; @@ -1119,7 +1119,7 @@ async fn same_address_reregistration_transfers_shards_and_reconciles() { .state_at(ack.grant.revision) .await .expect("the ack must name a revision the store really held"); - assert_eq!(claim_of(&stored, new_executor_id), ack.grant.shard_epochs); + assert_eq!(epochs_of(&stored, new_executor_id), ack.grant.shard_epochs); wait_for_local_assignment(&worker_executors, restarted_pod, shard_ids(&[0, 1])).await; assert_eq!( @@ -1256,6 +1256,173 @@ async fn a_repeated_registration_of_the_same_executor_refreshes_its_lease() { join_set.abort_all(); } +#[test] +// A store that was wiped or replaced no longer lists the executor, so its renewal is refused as a +// lease not found and never reaches the repair a renewal makes. The executor re-registers under a +// fresh id carrying the set it held, and the oplog rows it wrote are fenced at those epochs: the +// manager has to mint above them, or every write to those agents is refused for good. +async fn a_re_registration_after_a_wiped_store_mints_above_the_epochs_it_held() { + let restarted_pod = pod(1, 9000); + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(ShardLeaseState::new(4), worker_executors.clone()).await; + + let held = BTreeMap::from([ + (ShardId::new(0), ShardEpoch(2)), + (ShardId::new(1), ShardEpoch(2)), + (ShardId::new(2), ShardEpoch(5)), + (ShardId::new(3), ShardEpoch(2)), + ]); + let ack = shard_management + .register_executor_with_previous_epochs(executor(1), restarted_pod.into(), None, held) + .await + .expect("the registration should have been persisted"); + // Evidence, not a request: nothing is assigned until the loop's pass. + assert!(ack.grant.shard_epochs.is_empty()); + + // The pass mints after the registration was stored, so it is the barrier for the epochs. + wait_for_local_assignment(&worker_executors, restarted_pod, shard_ids(&[0, 1, 2, 3])).await; + wait_for_quiescence(&persistence).await; + + let minted = BTreeMap::from([ + (ShardId::new(0), ShardEpoch(3)), + (ShardId::new(1), ShardEpoch(3)), + (ShardId::new(2), ShardEpoch(6)), + (ShardId::new(3), ShardEpoch(3)), + ]); + let after = persistence.latest().await; + assert_eq!( + epochs_of(&after, executor(1)), + minted, + "the pass minted from a floor the store forgot" + ); + assert!(after.check_invariants().is_ok()); + + let pushed = worker_executors + .pushes_to(restarted_pod) + .await + .pop() + .expect("the registered executor should have been pushed its set"); + assert_eq!(pushed.shard_epochs, minted); + + join_set.abort_all(); +} + +#[test] +// The same carried set against a store that kept its history is at or below the record, so it moves +// nothing: the shards a re-registered executor is given are minted exactly as they are for a +// registration that carried nothing, one past the record. +async fn a_re_registration_against_an_intact_store_mints_as_if_it_carried_nothing() { + let restarted_pod = pod(1, 9000); + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + + let before = persistence.latest().await; + let held = epochs_of(&before, executor(1)); + // The lease is gone, as it is once reaped, so the executor comes back under a fresh id - with + // nothing applied locally, as after it cleared its assignment. + shard_management + .deregister_executor(executor(1), held.clone()) + .await + .expect("the deregistration should have been persisted"); + worker_executors + .set_local_assignment(restarted_pod, &[]) + .await; + + let ack = shard_management + .register_executor_with_previous_epochs( + executor(3), + restarted_pod.into(), + Some("worker-executor-0".to_string()), + held, + ) + .await + .expect("the registration should have been persisted"); + assert!(ack.grant.shard_epochs.is_empty()); + + wait_for_local_assignment(&worker_executors, restarted_pod, shard_ids(&[0, 1])).await; + wait_for_quiescence(&persistence).await; + + let after = persistence.latest().await; + assert_eq!( + epochs_of(&after, executor(3)), + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(1)), + (ShardId::new(1), ShardEpoch(1)), + ]), + "a carried set at the record moved an epoch" + ); + assert_eq!( + epochs_of(&after, executor(2)), + epochs_of(&before, executor(2)) + ); + assert!(after.check_invariants().is_ok()); + + join_set.abort_all(); +} + +#[test] +// A re-registration's carried set can name a shard the manager has given to another executor. +// Nothing corroborates a held epoch, so it moves nothing there: if the holder did write oplog rows +// above the owner's epoch, the owner's refused writes report them, and that is what re-mints it. +async fn a_re_registration_carrying_a_held_epoch_on_another_executors_shard_leaves_its_owner_alone() +{ + let restarted_pod = pod(1, 9000); + let owner_pod = pod(2, 9001); + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + assert_eq!(worker_executors.pushes_to(restarted_pod).await.len(), 1); + assert_eq!(worker_executors.pushes_to(owner_pod).await.len(), 1); + + let before = persistence.latest().await; + let mut held = epochs_of(&before, executor(1)); + held.insert(ShardId::new(2), ShardEpoch(5)); + let ack = shard_management + .register_executor_with_previous_epochs( + executor(3), + restarted_pod.into(), + Some("worker-executor-0".to_string()), + held, + ) + .await + .expect("the registration should have been persisted"); + // A new instance at a known address inherits its predecessor's shards one past the record; the + // held epoch on the owner's shard does not hand that shard over. + assert_eq!( + ack.grant.shard_epochs, + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(1)), + (ShardId::new(1), ShardEpoch(1)), + ]) + ); + + let stored = persistence + .state_at(ack.grant.revision) + .await + .expect("the ack must name a revision the store really held"); + assert_eq!( + stored.shards_for_executor(executor(2)), + Some(shard_ids(&[2, 3])) + ); + assert_eq!(stored.epoch_for_shard(ShardId::new(2)), Some(ShardEpoch(0))); + assert_eq!(stored.epoch_for_shard(ShardId::new(3)), Some(ShardEpoch(0))); + assert!(stored.check_invariants().is_ok()); + + // Startup pushed once to each executor and the replacement pushes the restarted one. The + // owner's epochs did not move, so it is owed nothing. + wait_for_pushes(&worker_executors, 3).await; + wait_for_quiescence(&persistence).await; + assert_eq!( + worker_executors.pushes_to(owner_pod).await.len(), + 1, + "a held epoch on the owner's shard pushed it" + ); + + join_set.abort_all(); +} + #[test] // The loop ends with the error rather than carry on against a store it can no longer trust, so // the process restarts and re-reads. On a conflict the cached revision is deliberately not @@ -1969,7 +2136,7 @@ async fn a_shard_count_mismatch_is_refused_before_the_worker_can_act() { #[test] // A renewal asserts the set the executor holds, it does not re-grant it. Advancing the epoch would -// make the protocol non-idempotent: one lost response and the executor's next renewal claims an +// make the protocol non-idempotent: one lost response and the executor's next renewal sends an // epoch one behind, and is refused for shards it never stopped owning. async fn renewing_twice_with_the_same_epochs_moves_nothing() { let worker_executors = Arc::new(TestWorkerExecutors::default()); @@ -1977,31 +2144,31 @@ async fn renewing_twice_with_the_same_epochs_moves_nothing() { new_shard_management(balanced_pair(), worker_executors.clone()).await; let before = persistence.latest().await; - let claimed = claim_of(&before, executor(1)); + let held = epochs_of(&before, executor(1)); assert_eq!( - claimed.keys().copied().collect::>(), + held.keys().copied().collect::>(), shard_ids(&[0, 1]) ); let first = shard_management - .renew_shard_lease(executor(1), claimed.clone()) + .renew_shard_lease(executor(1), held.clone()) .await .expect("the first renewal should have been granted"); let second = shard_management - .renew_shard_lease(executor(1), claimed.clone()) + .renew_shard_lease(executor(1), held.clone()) .await .expect("the second renewal of the same set should have been granted too"); - assert_eq!(first.shard_epochs, claimed); + assert_eq!(first.shard_epochs, held); assert_eq!( - second.shard_epochs, claimed, + second.shard_epochs, held, "a renewal advanced an epoch, so the executor is now one behind for a shard it still owns" ); assert!(second.expires_at >= first.expires_at); assert!(second.expires_at > expiry_of(&before, executor(1))); let after = persistence.latest().await; - assert_eq!(claim_of(&after, executor(1)), claimed); + assert_eq!(epochs_of(&after, executor(1)), held); assert_eq!( after.shards_for_executor(executor(1)), Some(shard_ids(&[0, 1])) @@ -2012,26 +2179,26 @@ async fn renewing_twice_with_the_same_epochs_moves_nothing() { "renewing one executor's lease disturbed another's shards" ); assert_eq!( - claim_of(&after, executor(2)), - claim_of(&before, executor(2)) + epochs_of(&after, executor(2)), + epochs_of(&before, executor(2)) ); join_set.abort_all(); } #[test] -// The claim is what the executor believes it holds, not a condition of the renewal. A wrong epoch, -// another executor's shard and a released shard are all an executor that missed a push, and -// refusing its renewal would only hold it on a picture the manager knows is wrong. It is renewed, -// and the grant carries the manager's set: the renewal is the guaranteed second delivery path for a -// push that was lost. Only an executor the manager has never heard of is refused. -async fn a_mismatched_claim_is_renewed_and_corrected() { +// The held set is what the executor believes it holds, not a condition of the renewal. Another +// executor's shard - at any epoch - and a released shard are an executor that missed a push; +// refusing the renewal would only hold the executor on a picture the manager knows is wrong. It is +// renewed, and the grant carries the manager's set: the renewal is the guaranteed second delivery +// path for a push that was lost. Only an executor the manager has never heard of is refused. +async fn a_mismatched_held_set_is_renewed_and_corrected() { let worker_executors = Arc::new(TestWorkerExecutors::default()); let (shard_management, persistence, mut join_set) = new_shard_management(balanced_pair(), worker_executors.clone()).await; let before = persistence.latest().await; - let truth = claim_of(&before, executor(1)); + let truth = epochs_of(&before, executor(1)); // an executor the manager has never heard of: still the one refusal let err = shard_management @@ -2046,32 +2213,46 @@ async fn a_mismatched_claim_is_renewed_and_corrected() { "got {err:?}" ); - // a wrong epoch, alongside a claim entry that is perfectly valid - let mut wrong_epoch = truth.clone(); - wrong_epoch.insert(ShardId::new(1), ShardEpoch(7)); + // an epoch ahead of the record on a shard that belongs to somebody else, alongside held + // entries that are perfectly valid. Corrected and never adopted, and the owner's epoch does not + // move on the holder's word. + let mut ahead_of_owner = truth.clone(); + ahead_of_owner.insert(ShardId::new(2), ShardEpoch(7)); let expiry_before = expiry_of(&persistence.latest().await, executor(1)); let grant = shard_management - .renew_shard_lease(executor(1), wrong_epoch) + .renew_shard_lease(executor(1), ahead_of_owner) .await - .expect("a claim at the wrong epoch is renewed and corrected"); + .expect("a held epoch ahead of another executor's shard is renewed and corrected"); assert_eq!( grant.shard_epochs, truth, - "the grant is the manager's set, not the claim" + "the grant is the manager's set, not the held one" ); assert!(grant.expires_at > expiry_before, "the lease was extended"); + let after_renewal = persistence.latest().await; + assert_eq!( + after_renewal.epoch_for_shard(ShardId::new(2)), + Some(ShardEpoch(0)), + "the held set moved another executor's epoch" + ); + assert_eq!( + after_renewal.shards_for_executor(executor(2)), + Some(shard_ids(&[2, 3])), + "the held set moved another executor's shard" + ); + // a shard that belongs to another executor let moved = BTreeMap::from([(ShardId::new(2), ShardEpoch(0))]); let grant = shard_management .renew_shard_lease(executor(1), moved) .await - .expect("claiming another executor's shard is renewed and corrected"); + .expect("holding another executor's shard is renewed and corrected"); assert_eq!(grant.shard_epochs, truth); // ...and a shard nobody owns any more. Deregistering executor 2 releases its shards without // waking the loop, so they stay unassigned for the rest of this test. shard_management - .deregister_executor(executor(2), claim_of(&before, executor(2))) + .deregister_executor(executor(2), epochs_of(&before, executor(2))) .await .expect("a graceful deregistration should have been persisted"); let revoked = BTreeMap::from([ @@ -2081,15 +2262,361 @@ async fn a_mismatched_claim_is_renewed_and_corrected() { let grant = shard_management .renew_shard_lease(executor(1), revoked) .await - .expect("claiming a released shard is renewed and corrected"); + .expect("holding a released shard is renewed and corrected"); assert_eq!(grant.shard_epochs, truth); - // None of the corrections moved anything: executor 1's set and epochs are exactly what they - // were, only its lease clock moved, and executor 2's released shards stayed released. + // No correction moved executor 1's set or anybody's epochs: they are exactly what they were, + // only its lease clock moved, and executor 2's released shards stayed released. let after = persistence.latest().await; - assert_eq!(claim_of(&after, executor(1)), truth); + assert_eq!(epochs_of(&after, executor(1)), truth); assert!(expiry_of(&after, executor(1)) > expiry_before); assert_eq!(after.get_unassigned_shards(), shard_ids(&[2, 3])); + assert_eq!(after.shard_epochs[&ShardId::new(2)], ShardEpoch(0)); + + join_set.abort_all(); +} + +#[test] +// The one case where held epochs move the manager's state rather than being corrected by it. An +// executor is never told an epoch the store did not hold first, so a held epoch ahead of the record is +// only possible when the store lost history - wiped, restored from a backup, or replaced. The +// executors' oplog rows are still fenced against the epochs they were granted before the loss, so +// a manager that went on granting from a lower floor would have every one of their writes refused +// for good. The renewal is the one moment the cluster can tell the manager what it forgot. +async fn held_epochs_ahead_of_the_record_raise_the_managers_floor() { + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + + let before = persistence.latest().await; + assert_eq!(before.epoch_for_shard(ShardId::new(1)), Some(ShardEpoch(0))); + + let mut ahead = epochs_of(&before, executor(1)); + ahead.insert(ShardId::new(1), ShardEpoch(7)); + let grant = shard_management + .renew_shard_lease(executor(1), ahead.clone()) + .await + .expect("a held epoch ahead of the record is renewed, not refused"); + + // The grant is still read off the manager's state - that state was repaired first, so the + // owner is told the epoch it already holds instead of one its oplog rows would refuse. + assert_eq!(grant.shard_epochs, ahead); + + let after = persistence.latest().await; + assert_eq!(after.epoch_for_shard(ShardId::new(1)), Some(ShardEpoch(7))); + assert_eq!(epochs_of(&after, executor(1)), ahead); + assert_eq!( + after.shards_for_executor(executor(1)), + Some(shard_ids(&[0, 1])), + "repairing an epoch moved a shard" + ); + assert_eq!( + after.shards_for_executor(executor(2)), + Some(shard_ids(&[2, 3])), + "repairing an epoch disturbed another executor" + ); + + join_set.abort_all(); +} + +#[test] +// A held epoch ahead of the record on a shard the manager has since given to another executor. It is one +// executor's word about another's shard, so it moves nothing and nobody is pushed. If the holder +// did write oplog rows at that epoch, the owner's refused write reports them, and that report - +// see `a_fenced_epoch_reported_by_a_non_owner_re_mints_and_pushes_the_owner` and its neighbours - +// is what mints the owner past them. +async fn a_higher_held_epoch_on_another_executors_shard_moves_nothing() { + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + assert_eq!(worker_executors.pushes_to(pod(1, 9000)).await.len(), 1); + assert_eq!(worker_executors.pushes_to(pod(2, 9001)).await.len(), 1); + + let before = persistence.latest().await; + let mut ahead_of_owner = epochs_of(&before, executor(1)); + ahead_of_owner.insert(ShardId::new(2), ShardEpoch(7)); + let grant = shard_management + .renew_shard_lease(executor(1), ahead_of_owner) + .await + .expect("a held epoch ahead of another executor's shard is renewed, not refused"); + assert_eq!( + grant.shard_epochs, + epochs_of(&before, executor(1)), + "the holder was given another executor's shard" + ); + + let after = persistence.latest().await; + assert_eq!( + after.shards_for_executor(executor(2)), + Some(shard_ids(&[2, 3])) + ); + assert_eq!(after.epoch_for_shard(ShardId::new(2)), Some(ShardEpoch(0))); + assert_eq!(after.epoch_for_shard(ShardId::new(3)), Some(ShardEpoch(0))); + assert_eq!( + epochs_of(&after, executor(1)), + epochs_of(&before, executor(1)) + ); + assert!(after.check_invariants().is_ok()); + + wait_for_quiescence(&persistence).await; + for pod in [pod(1, 9000), pod(2, 9001)] { + assert_eq!( + worker_executors.pushes_to(pod).await.len(), + 1, + "a held epoch that moved nothing pushed somebody" + ); + } + + join_set.abort_all(); +} + +#[test] +// A renewal can carry the epoch an executor's refused oplog write found on the rows. Above the +// record it proves the store lost history, but not that the reporter wrote those rows, so even the +// reporter's own shard is minted one past that epoch rather than onto it. When the same renewal also +// holds that epoch, the fenced epoch is applied first: held first would record the held epoch and +// leave the reporter on the rows' `(shard, epoch)`. +async fn a_renewal_reporting_a_fenced_epoch_re_mints_the_reporters_own_shard_above_it() { + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + + let before = persistence.latest().await; + let grant = shard_management + .renew_shard_lease_with_fenced_epochs( + executor(1), + epochs_of(&before, executor(1)), + BTreeMap::from([(ShardId::new(1), ShardEpoch(3))]), + ) + .await + .expect("a renewal reporting a fenced epoch is renewed, not refused"); + assert_eq!( + grant.shard_epochs, + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(0)), + (ShardId::new(1), ShardEpoch(4)), + ]), + "the grant does not carry the re-minted epoch" + ); + + let after = persistence.latest().await; + assert_eq!(after.epoch_for_shard(ShardId::new(1)), Some(ShardEpoch(4))); + assert_eq!( + after.shards_for_executor(executor(1)), + Some(shard_ids(&[0, 1])), + "re-minting an epoch moved a shard" + ); + assert_eq!( + epochs_of(&after, executor(2)), + epochs_of(&before, executor(2)), + "a fenced epoch on one executor's shard disturbed another executor" + ); + assert!(after.check_invariants().is_ok()); + + // The reporter learns its re-minted epoch from the grant, so nobody is owed a push. + wait_for_quiescence(&persistence).await; + assert_eq!(worker_executors.pushes_to(pod(1, 9000)).await.len(), 1); + assert_eq!(worker_executors.pushes_to(pod(2, 9001)).await.len(), 1); + join_set.abort_all(); + + // The same epoch held and reported in one request. + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + let grant = shard_management + .renew_shard_lease_with_fenced_epochs( + executor(1), + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(0)), + (ShardId::new(1), ShardEpoch(3)), + ]), + BTreeMap::from([(ShardId::new(1), ShardEpoch(3))]), + ) + .await + .expect("a renewal holding and reporting one epoch is renewed, not refused"); + assert_eq!( + grant.shard_epochs, + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(0)), + (ShardId::new(1), ShardEpoch(4)), + ]), + "the held epoch was applied before the fenced one, leaving the executor on the rows' epoch" + ); + assert_eq!( + persistence.latest().await.epoch_for_shard(ShardId::new(1)), + Some(ShardEpoch(4)) + ); + + join_set.abort_all(); +} + +#[test] +// A fenced epoch on a shard the manager has given to another executor: the reporter lost the shard +// and was refused by rows above the owner's recorded epoch, which only a store that lost history can +// produce. The owner keeps the shard, is minted one past the rows, and is pushed that epoch at once +// rather than left to find out on its own renewal; the reporter is not given the shard. +async fn a_fenced_epoch_reported_by_a_non_owner_re_mints_and_pushes_the_owner() { + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + + let before = persistence.latest().await; + let grant = shard_management + .renew_shard_lease_with_fenced_epochs( + executor(1), + epochs_of(&before, executor(1)), + BTreeMap::from([(ShardId::new(2), ShardEpoch(7))]), + ) + .await + .expect("a renewal reporting a fenced epoch on another executor's shard is renewed"); + assert_eq!( + grant.shard_epochs, + epochs_of(&before, executor(1)), + "the reporter was given another executor's shard" + ); + + let after = persistence.latest().await; + assert_eq!( + after.shards_for_executor(executor(2)), + Some(shard_ids(&[2, 3])) + ); + assert_eq!(after.epoch_for_shard(ShardId::new(2)), Some(ShardEpoch(8))); + assert_eq!(after.epoch_for_shard(ShardId::new(3)), Some(ShardEpoch(0))); + assert!(after.check_invariants().is_ok()); + + // Startup pushed once to each executor, so the owner's push is the third. The tick is a third + // of a 60s lease away, so only the renewal waking the loop can send it within the wait. + wait_for_pushes(&worker_executors, 3).await; + wait_for_quiescence(&persistence).await; + + let owner_pushes = worker_executors.pushes_to(pod(2, 9001)).await; + assert_eq!(owner_pushes.len(), 2); + let pushed = owner_pushes.last().expect("the owner was pushed"); + assert_eq!( + pushed.shard_epochs, + BTreeMap::from([ + (ShardId::new(2), ShardEpoch(8)), + (ShardId::new(3), ShardEpoch(0)), + ]) + ); + assert!( + pushed.revision >= grant.revision, + "the push was read off a state older than the re-mint" + ); + assert_eq!( + worker_executors.pushes_to(pod(1, 9000)).await.len(), + 1, + "the reporter was pushed as well" + ); + + join_set.abort_all(); +} + +#[test] +// A fenced epoch at or below the record is the ordinary loser of a shard move - the new owner +// recorded its epoch, and the old one was refused - and a report the manager already applied is at +// the record the second time. Neither moves an epoch or owes anyone a push. A renewal the manager +// refuses applies nothing it carried. +async fn a_fenced_epoch_at_or_below_the_record_moves_nothing() { + let worker_executors = Arc::new(TestWorkerExecutors::default()); + let (shard_management, persistence, mut join_set) = + new_shard_management(balanced_pair(), worker_executors.clone()).await; + + // A restarted instance at executor 1's address inherits its shards at epoch 1, so executor + // 1's writes at epoch 0 are what the rows refuse from now on. + shard_management + .register_executor( + executor(3), + pod(1, 9000).into(), + Some("worker-executor-0".to_string()), + ) + .await + .expect("the registration should have been persisted"); + wait_for_pushes(&worker_executors, 3).await; + wait_for_quiescence(&persistence).await; + let moved = persistence.latest().await; + assert_eq!( + epochs_of(&moved, executor(3)), + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(1)), + (ShardId::new(1), ShardEpoch(1)), + ]) + ); + let pushes = worker_executors.pushes.lock().await.len(); + + let grant = shard_management + .renew_shard_lease_with_fenced_epochs( + executor(2), + epochs_of(&moved, executor(2)), + BTreeMap::from([ + (ShardId::new(0), ShardEpoch(1)), + (ShardId::new(1), ShardEpoch(0)), + ]), + ) + .await + .expect("a renewal reporting a fenced epoch at the record is renewed"); + assert_eq!(grant.shard_epochs, epochs_of(&moved, executor(2))); + let after = persistence.latest().await; + assert_eq!(after.shard_epochs, moved.shard_epochs); + assert_eq!(after.shard_assignments, moved.shard_assignments); + wait_for_quiescence(&persistence).await; + assert_eq!( + worker_executors.pushes.lock().await.len(), + pushes, + "a fenced epoch at the record owed somebody a push" + ); + + // A genuine report re-mints shard 1's owner once. The same report on the next renewal is at the + // record, so the state stays exactly as the first one left it. + let genuine = BTreeMap::from([(ShardId::new(1), ShardEpoch(5))]); + shard_management + .renew_shard_lease_with_fenced_epochs( + executor(2), + epochs_of(&moved, executor(2)), + genuine.clone(), + ) + .await + .expect("a renewal reporting a fenced epoch ahead of the record is renewed"); + wait_for_pushes(&worker_executors, pushes + 1).await; + wait_for_quiescence(&persistence).await; + let raised = persistence.latest().await; + assert_eq!(raised.epoch_for_shard(ShardId::new(1)), Some(ShardEpoch(6))); + let pushes = worker_executors.pushes.lock().await.len(); + + shard_management + .renew_shard_lease_with_fenced_epochs(executor(2), epochs_of(&raised, executor(2)), genuine) + .await + .expect("a repeated report is renewed"); + wait_for_quiescence(&persistence).await; + let repeated = persistence.latest().await; + assert_eq!(repeated.shard_epochs, raised.shard_epochs); + assert_eq!(repeated.shard_assignments, raised.shard_assignments); + assert_eq!( + worker_executors.pushes.lock().await.len(), + pushes, + "a repeated report re-minted its owner a second time" + ); + + let err = shard_management + .renew_shard_lease_with_fenced_epochs( + executor(9), + BTreeMap::new(), + BTreeMap::from([(ShardId::new(0), ShardEpoch(9))]), + ) + .await + .expect_err("an unknown executor holds no lease to renew"); + assert!( + matches!( + err, + ShardManagerError::ShardLeaseNotFound { executor_id } if executor_id == executor(9) + ), + "got {err:?}" + ); + assert_eq!( + persistence.latest().await.epoch_for_shard(ShardId::new(0)), + Some(ShardEpoch(1)), + "a refused renewal applied the fenced epoch it carried" + ); join_set.abort_all(); } @@ -2105,9 +2632,9 @@ async fn a_grant_carries_the_revision_it_was_stored_at() { let before = persistence.latest().await; let grant = shard_management - .renew_shard_lease(executor(1), claim_of(&before, executor(1))) + .renew_shard_lease(executor(1), epochs_of(&before, executor(1))) .await - .expect("a valid claim should have been renewed"); + .expect("a valid held set should have been renewed"); let latest = persistence.latest().await; assert!( @@ -2118,16 +2645,16 @@ async fn a_grant_carries_the_revision_it_was_stored_at() { grant.revision, latest.revision, "the grant names the revision the state holding its set was stored at" ); - assert_eq!(claim_of(&latest, executor(1)), grant.shard_epochs); + assert_eq!(epochs_of(&latest, executor(1)), grant.shard_epochs); join_set.abort_all(); } #[test] -// The claim is checked in the direction the executor can be wrong about: a shard the manager has -// assigned to it that it does not claim is an executor that has not received its last push yet, and +// The held set is checked in the direction the executor can be wrong about: a shard the manager has +// assigned to it that it does not send is an executor that has not received its last push yet, and // the push - not a refused renewal - is what corrects that. The full set comes back either way. -async fn an_owned_shard_the_executor_did_not_claim_is_not_an_error() { +async fn an_owned_shard_the_executor_did_not_send_is_not_an_error() { let worker_executors = Arc::new(TestWorkerExecutors::default()); let (shard_management, persistence, mut join_set) = new_shard_management(balanced_pair(), worker_executors.clone()).await; @@ -2138,12 +2665,12 @@ async fn an_owned_shard_the_executor_did_not_claim_is_not_an_error() { let grant = shard_management .renew_shard_lease(executor(1), partial) .await - .expect("a claim that is behind the manager must still renew"); + .expect("a held set that is behind the manager must still renew"); assert_eq!( grant.shard_epochs, - claim_of(&before, executor(1)), - "the renewal answered the claim rather than the executor's full current set" + epochs_of(&before, executor(1)), + "the renewal answered the held set rather than the executor's full current set" ); assert!(grant.expires_at > expiry_of(&before, executor(1))); @@ -2165,7 +2692,7 @@ async fn a_lease_that_lapsed_before_its_renewal_is_not_found() { .await; let before = persistence.latest().await; - let claimed = claim_of(&before, executor(1)); + let held = epochs_of(&before, executor(1)); join_set.abort_all(); while join_set.join_next().await.is_some() {} @@ -2174,7 +2701,7 @@ async fn a_lease_that_lapsed_before_its_renewal_is_not_found() { tokio::time::sleep(Duration::from_millis(1200)).await; let err = shard_management - .renew_shard_lease(executor(1), claimed) + .renew_shard_lease(executor(1), held) .await .expect_err("a lease that had already lapsed must not be renewable"); assert!( @@ -2208,9 +2735,9 @@ async fn a_renewal_under_a_shortened_lease_keeps_the_outstanding_deadline() { "the startup re-grant must not shorten an outstanding lease either" ); - let claimed = claim_of(&persistence.latest().await, executor(1)); + let held = epochs_of(&persistence.latest().await, executor(1)); let grant = shard_management - .renew_shard_lease(executor(1), claimed) + .renew_shard_lease(executor(1), held) .await .expect("a renewal of the held set is granted"); assert!( @@ -2308,12 +2835,12 @@ async fn past_expiries_do_not_evict_a_healthy_cluster_on_restart() { assert!(expiry_of(&after, executor(2)) > Utc::now()); // the re-grant moves the lease clock only: no shard changed owner, so no epoch moved assert_eq!( - claim_of(&after, executor(1)), - claim_of(&seeded, executor(1)) + epochs_of(&after, executor(1)), + epochs_of(&seeded, executor(1)) ); assert_eq!( - claim_of(&after, executor(2)), - claim_of(&seeded, executor(2)) + epochs_of(&after, executor(2)), + epochs_of(&seeded, executor(2)) ); join_set.abort_all(); @@ -2381,9 +2908,9 @@ async fn the_loop_compacts_after_each_pass_and_carries_on_when_compaction_fails( } #[test] -// The lease paths never wake the loop, so the timer is the only thing that can notice an expiry in -// a cluster where nothing else is happening. Without it a lapsed lease is held forever and its -// shards are never re-homed. +// The lease paths wake the loop only to push an owner a renewal re-minted, so the timer is the only +// thing that can notice an expiry in a cluster where nothing else is happening. Without it a lapsed +// lease is held forever and its shards are never re-homed. async fn an_expired_lease_is_reclaimed_within_one_tick() { let worker_executors = Arc::new(TestWorkerExecutors::default()); let (_shard_management, persistence, mut join_set) = start_shard_management( @@ -2431,7 +2958,7 @@ async fn deregistering_an_executor_re_homes_its_shards_within_one_tick() { let before = persistence.latest().await; shard_management - .deregister_executor(executor(1), claim_of(&before, executor(1))) + .deregister_executor(executor(1), epochs_of(&before, executor(1))) .await .expect("a graceful deregistration should have been persisted"); @@ -2480,7 +3007,7 @@ async fn deregistering_an_executor_re_homes_its_shards_within_one_tick() { assert_eq!(after.epoch_for_shard(ShardId::new(0)), Some(ShardEpoch(1))); assert_eq!(after.epoch_for_shard(ShardId::new(2)), Some(ShardEpoch(0))); // Nothing is sent to the executor that left: it asked to be released because it is shutting - // down, and the manager holds no lease to revoke against any more. If it comes back claiming + // down, and the manager holds no lease to revoke against any more. If it comes back holding // those shards, its renewal is refused with `ShardLeaseNotFound` and it re-registers. assert!( worker_executors.pushes_to(leaving_pod).await.len() <= 1, @@ -2492,7 +3019,7 @@ async fn deregistering_an_executor_re_homes_its_shards_within_one_tick() { #[test] // Deregistering an executor the manager does not know is not a failure: a shutdown must never fail -// on bookkeeping, and neither must a stale claim. +// on bookkeeping, and neither must a stale held set. async fn deregistering_an_unknown_executor_succeeds() { let worker_executors = Arc::new(TestWorkerExecutors::default()); let (shard_management, persistence, mut join_set) = @@ -2580,7 +3107,7 @@ async fn a_persist_failure_while_renewing_stops_the_loop() { new_shard_management(balanced_pair(), worker_executors.clone()).await; let before = persistence.latest().await; - let claimed = claim_of(&before, executor(1)); + let held = epochs_of(&before, executor(1)); persistence .fail_writes(vec![Some(ShardManagerError::LeadershipLost { @@ -2590,7 +3117,7 @@ async fn a_persist_failure_while_renewing_stops_the_loop() { .await; let err = shard_management - .renew_shard_lease(executor(1), claimed) + .renew_shard_lease(executor(1), held) .await .expect_err("a renewal whose persist was refused must not be granted"); assert!( diff --git a/golem-test-framework/Cargo.toml b/golem-test-framework/Cargo.toml index 807d2c6586..e8d1bb861f 100644 --- a/golem-test-framework/Cargo.toml +++ b/golem-test-framework/Cargo.toml @@ -12,6 +12,11 @@ license-file = "../LICENSE" [lib] harness = false +[[test]] +name = "signal_unreaped_child" +path = "tests/signal_unreaped_child.rs" +harness = false + [dependencies] golem-api-grpc = { workspace = true } golem-client = { workspace = true } @@ -61,5 +66,8 @@ url = { workspace = true } uuid = { workspace = true } wasm-metadata = { workspace = true } +[target.'cfg(unix)'.dependencies] +libc = { workspace = true } + [features] default = [] diff --git a/golem-test-framework/src/components/mod.rs b/golem-test-framework/src/components/mod.rs index 20757aec0b..616bf69490 100644 --- a/golem-test-framework/src/components/mod.rs +++ b/golem-test-framework/src/components/mod.rs @@ -249,6 +249,28 @@ fn check_child_process_alive(child: &mut Child, name: &str) { } } +/// One gRPC health check with a bounded wait: whether the service at `host:grpc_port` answers +/// `Serving` right now. Unlike a process-liveness check this also fails for a process that has +/// stopped serving but not yet exited - one aborting while the OS writes its crash report. +pub async fn is_serving_grpc(host: &str, grpc_port: u16, timeout: Duration) -> bool { + let probe = async { + let mut client = + golem_api_grpc::proto::grpc::health::v1::health_client::HealthClient::connect(format!( + "http://{host}:{grpc_port}" + )) + .await + .ok()?; + let response = client + .check(HealthCheckRequest { + service: "".to_string(), + }) + .await + .ok()?; + Some(response.into_inner().status == ServingStatus::Serving as i32) + }; + matches!(tokio::time::timeout(timeout, probe).await, Ok(Some(true))) +} + pub async fn wait_for_startup_grpc( host: &str, grpc_port: u16, diff --git a/golem-test-framework/src/components/worker_executor/mod.rs b/golem-test-framework/src/components/worker_executor/mod.rs index 569f3202da..3fd78c673d 100644 --- a/golem-test-framework/src/components/worker_executor/mod.rs +++ b/golem-test-framework/src/components/worker_executor/mod.rs @@ -53,6 +53,24 @@ pub trait WorkerExecutor: Send + Sync { ); } + /// Freezes this worker executor's process in place (SIGSTOP) without killing it: it keeps its + /// sockets, its memory and every lease it believes it holds, but answers nothing until + /// [`WorkerExecutor::resume`]. This is how an executor is made to look dead to the rest of + /// the cluster while it still thinks it owns its shards, which a kill cannot do. + /// + /// Default implementation panics: only `SpawnedWorkerExecutor` owns a process to freeze. + async fn pause(&self) { + panic!("WorkerExecutor::pause is only supported by SpawnedWorkerExecutor"); + } + + /// Thaws a process frozen by [`WorkerExecutor::pause`] (SIGCONT). It carries on from exactly + /// where it stopped. + /// + /// Default implementation panics: only `SpawnedWorkerExecutor` owns a process to thaw. + async fn resume(&self) { + panic!("WorkerExecutor::resume is only supported by SpawnedWorkerExecutor"); + } + async fn is_running(&self) -> bool; } diff --git a/golem-test-framework/src/components/worker_executor/spawned.rs b/golem-test-framework/src/components/worker_executor/spawned.rs index b472c99c59..a170e25b29 100644 --- a/golem-test-framework/src/components/worker_executor/spawned.rs +++ b/golem-test-framework/src/components/worker_executor/spawned.rs @@ -185,6 +185,54 @@ impl SpawnedWorkerExecutor { } let _logger = self.logger.lock().unwrap().take(); } + + #[cfg(unix)] + fn signal_child(&self, signal: libc::c_int, action: &str) { + // The guard is held across the liveness check and the signal: `is_running` and + // `blocking_kill`, the only other reapers of this child, both take the same lock. + let mut child_field = self.child.lock().unwrap(); + let child = child_field.as_mut().unwrap_or_else(|| { + panic!( + "Cannot {action} golem-worker-executor {}: it is not running", + self.grpc_port + ) + }); + signal_unreaped_child( + child, + signal, + &format!("{action} golem-worker-executor {}", self.grpc_port), + ); + } +} + +/// Sends `signal` to `child`, refusing to if the child has already been reaped. `what` names the +/// action and the process for the panic messages. +/// +/// A reaped child's pid is free for the OS to hand to an unrelated process, and `is_running`'s +/// `try_wait` reaps an exited child while leaving it in place, so a raw `kill` on `child.id()` +/// alone could signal a stranger. `Child::kill` has this guard built in; `kill(2)` does not. +/// +/// `pub` so its process-supervision behavior can be pinned by an integration test under +/// `golem-test-framework/tests/` rather than a `--lib` unit test that would spawn a process. +#[cfg(unix)] +pub fn signal_unreaped_child(child: &mut Child, signal: libc::c_int, what: &str) { + match child.try_wait() { + Ok(None) => {} + Ok(Some(status)) => panic!("Cannot {what}: it has already exited ({status})"), + Err(err) => panic!("Cannot {what}: its state is unknown: {err}"), + } + let pid = libc::pid_t::try_from(child.id()).expect("child pid does not fit into pid_t"); + // SAFETY: `kill` has no memory-safety preconditions. `try_wait` has just reported the child + // alive, and reaping it needs the `&mut Child` held here, so it has not been reaped and its pid + // cannot belong to another process. If it exited since, it is a zombie still holding that pid, + // and the signal changes nothing. + let result = unsafe { libc::kill(pid, signal) }; + assert_eq!( + result, + 0, + "Failed to {what}: {}", + std::io::Error::last_os_error() + ); } #[async_trait] @@ -257,6 +305,36 @@ impl WorkerExecutor for SpawnedWorkerExecutor { false } } + + #[cfg(unix)] + async fn pause(&self) { + info!("Pausing golem-worker-executor {}", self.grpc_port); + self.signal_child(libc::SIGSTOP, "pause"); + } + + #[cfg(unix)] + async fn resume(&self) { + info!("Resuming golem-worker-executor {}", self.grpc_port); + self.signal_child(libc::SIGCONT, "resume"); + } + + // Without these the trait default would panic claiming this is not a SpawnedWorkerExecutor, + // when the real reason is the platform. + #[cfg(not(unix))] + async fn pause(&self) { + panic!( + "Cannot pause golem-worker-executor {}: pausing is SIGSTOP, which this platform does not have", + self.grpc_port + ); + } + + #[cfg(not(unix))] + async fn resume(&self) { + panic!( + "Cannot resume golem-worker-executor {}: resuming is SIGCONT, which this platform does not have", + self.grpc_port + ); + } } impl Drop for SpawnedWorkerExecutor { @@ -264,3 +342,8 @@ impl Drop for SpawnedWorkerExecutor { self.blocking_kill(); } } + +// `signal_unreaped_child`'s process-supervision behavior is pinned by +// `golem-test-framework/tests/signal_unreaped_child.rs` instead of a `--lib` unit test: unit +// tests must never spawn external processes (AGENTS.md), and `cargo make unit-tests` runs +// `--workspace --lib`. diff --git a/golem-test-framework/src/components/worker_executor_cluster/mod.rs b/golem-test-framework/src/components/worker_executor_cluster/mod.rs index 1ca1014dce..4d0e64da8d 100644 --- a/golem-test-framework/src/components/worker_executor_cluster/mod.rs +++ b/golem-test-framework/src/components/worker_executor_cluster/mod.rs @@ -46,6 +46,21 @@ pub trait WorkerExecutorCluster: Send + Sync { async fn stop(&self, index: usize); async fn start(&self, index: usize); + /// Freezes the executor at `index` in place; see [`WorkerExecutor::pause`]. It still counts + /// as started: only a stop takes it out of the cluster. + /// + /// Default implementation panics: only `SpawnedWorkerExecutorCluster` owns the processes. + async fn pause(&self, _index: usize) { + panic!("WorkerExecutorCluster::pause is only supported by SpawnedWorkerExecutorCluster"); + } + + /// Thaws the executor at `index`; see [`WorkerExecutor::resume`]. + /// + /// Default implementation panics: only `SpawnedWorkerExecutorCluster` owns the processes. + async fn resume(&self, _index: usize) { + panic!("WorkerExecutorCluster::resume is only supported by SpawnedWorkerExecutorCluster"); + } + fn to_vec(&self) -> Vec>; async fn stopped_indices(&self) -> Vec; diff --git a/golem-test-framework/src/components/worker_executor_cluster/spawned.rs b/golem-test-framework/src/components/worker_executor_cluster/spawned.rs index 684b92a6c3..00c2b85aad 100644 --- a/golem-test-framework/src/components/worker_executor_cluster/spawned.rs +++ b/golem-test-framework/src/components/worker_executor_cluster/spawned.rs @@ -194,6 +194,14 @@ impl WorkerExecutorCluster for SpawnedWorkerExecutorCluster { } } + async fn pause(&self, index: usize) { + self.worker_executors[index].pause().await; + } + + async fn resume(&self, index: usize) { + self.worker_executors[index].resume().await; + } + fn to_vec(&self) -> Vec> { self.worker_executors.to_vec() } diff --git a/golem-test-framework/src/config/env.rs b/golem-test-framework/src/config/env.rs index 82faa22886..173d114656 100644 --- a/golem-test-framework/src/config/env.rs +++ b/golem-test-framework/src/config/env.rs @@ -913,9 +913,15 @@ pub trait WorkerExecutorClusterControl { async fn restart_all_with_env_vars(&self, vars: Vec<(String, String)>); async fn stop(&self, idx: u16); async fn start(&self, idx: u16); + async fn pause(&self, idx: u16); + async fn resume(&self, idx: u16); async fn started_indices(&self) -> Vec; async fn stopped_indices(&self) -> Vec; async fn is_running(&self, idx: u16) -> bool; + /// Whether the executor at `idx` answers its gRPC health check right now. Stricter than + /// [`Self::is_running`]: a process that is aborting still counts as running until the OS has + /// finished with it, but it no longer serves. + async fn is_serving(&self, idx: u16) -> bool; async fn cluster_size(&self) -> u16; async fn stop_shard_manager(&self); @@ -962,6 +968,14 @@ impl WorkerExecutorClusterControl for EnvBasedTestDependencies { self.worker_executor_cluster.start(usize::from(idx)).await; } + async fn pause(&self, idx: u16) { + self.worker_executor_cluster.pause(usize::from(idx)).await; + } + + async fn resume(&self, idx: u16) { + self.worker_executor_cluster.resume(usize::from(idx)).await; + } + async fn started_indices(&self) -> Vec { self.worker_executor_cluster .started_indices() @@ -988,6 +1002,19 @@ impl WorkerExecutorClusterControl for EnvBasedTestDependencies { worker_executor.is_running().await } + async fn is_serving(&self, idx: u16) -> bool { + let worker_executors = self.worker_executor_cluster.to_vec(); + let Some(worker_executor) = worker_executors.get(usize::from(idx)).cloned() else { + return false; + }; + crate::components::is_serving_grpc( + &worker_executor.grpc_host(), + worker_executor.grpc_port(), + Duration::from_secs(5), + ) + .await + } + async fn cluster_size(&self) -> u16 { Self::usize_to_u16(self.worker_executor_cluster.size()) } diff --git a/golem-test-framework/src/dsl/mod.rs b/golem-test-framework/src/dsl/mod.rs index d9946d55d9..981a2db9e1 100644 --- a/golem-test-framework/src/dsl/mod.rs +++ b/golem-test-framework/src/dsl/mod.rs @@ -1232,6 +1232,13 @@ pub fn worker_error_message(error: &WorkerExecutorError) -> String { match error { WorkerExecutorError::InvalidRequest { details } => details.clone(), WorkerExecutorError::PermissionDenied { details } => details.clone(), + WorkerExecutorError::OplogFenced { + agent_id, + expected_epoch, + actual_epoch, + } => format!( + "Oplog write for {agent_id:?} fenced: asserted epoch {expected_epoch}, stored {actual_epoch:?}" + ), WorkerExecutorError::AgentAlreadyExists { agent_id } => { format!("Worker already exists: {:?}", agent_id) } diff --git a/golem-test-framework/tests/signal_unreaped_child.rs b/golem-test-framework/tests/signal_unreaped_child.rs new file mode 100644 index 0000000000..0abf9c4e06 --- /dev/null +++ b/golem-test-framework/tests/signal_unreaped_child.rs @@ -0,0 +1,78 @@ +// Copyright 2024-2026 Golem Cloud +// +// Licensed under the Golem Source License v1.1 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://license.golem.cloud/LICENSE +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Pins `signal_unreaped_child`'s process-supervision behavior. +//! +//! It is worth pinning because the guard it implements has no second line of defence: a reaped +//! child's pid is free for the OS to reuse, so a `kill(2)` on it can signal an unrelated process +//! on the machine running the tests. Nothing else in the suite would notice. +//! +//! It lives here rather than as a `--lib` unit test in `spawned.rs` because it needs a real child +//! process to signal, and unit tests must never spawn external processes (AGENTS.md); +//! `cargo make unit-tests` runs `--workspace --lib` and would otherwise pick it up. +//! +//! Everything here is `#[cfg(unix)]`, because `signal_unreaped_child` is. Windows has no +//! `SIGSTOP`/`SIGCONT`, and neither `std` nor `libc`'s Windows shim can suspend a running process +//! and resume it in place - so the stalled-executor scenario cannot be simulated there at all. +//! The gate is for the compiler rather than the test runner: `libc::SIGSTOP` and `libc::kill` do +//! not exist on Windows, so without it the daily Windows job, which only builds, would fail to +//! compile. It runs no tests, and the sharding suite that uses the pause is Linux-only in CI, so +//! the gate costs no coverage. `SpawnedWorkerExecutor::pause`/`resume` keep `#[cfg(not(unix))]` +//! arms that panic naming the platform, so running the suite there fails with the real reason. + +test_r::enable!(); + +#[cfg(unix)] +mod unix { + use golem_test_framework::components::worker_executor::spawned::signal_unreaped_child; + use std::process::{Child, Command}; + use test_r::test; + + /// Kills and reaps the child when the test ends, also when an assertion panics, so a failing + /// run does not leave a stopped process behind. + struct KilledOnDrop(Child); + + impl Drop for KilledOnDrop { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } + } + + #[test] + #[should_panic(expected = "has already exited")] + fn a_child_that_has_been_reaped_is_never_signalled() { + let mut child = Command::new("true") + .spawn() + .expect("failed to spawn `true`"); + // Records the exit status, exactly as `is_running`'s `try_wait` does for an exited child. + child.wait().expect("failed to wait for `true`"); + + signal_unreaped_child(&mut child, libc::SIGSTOP, "pause `true`"); + } + + #[test] + fn a_live_child_can_be_stopped_and_continued() { + let mut child = KilledOnDrop( + Command::new("sleep") + .arg("30") + .spawn() + .expect("failed to spawn `sleep`"), + ); + + signal_unreaped_child(&mut child.0, libc::SIGSTOP, "pause `sleep`"); + // `try_wait` does not report a stopped child, so continuing it is not refused as exited. + signal_unreaped_child(&mut child.0, libc::SIGCONT, "resume `sleep`"); + } +} diff --git a/golem-worker-executor-test-utils/src/dsl_impl.rs b/golem-worker-executor-test-utils/src/dsl_impl.rs index c0f696cf9b..993c511ecc 100644 --- a/golem-worker-executor-test-utils/src/dsl_impl.rs +++ b/golem-worker-executor-test-utils/src/dsl_impl.rs @@ -127,8 +127,16 @@ impl TestWorkerExecutor { match response.response { Some(invocation_response::Response::Accepted(_)) => {} Some(invocation_response::Response::Rejected(rejected)) => { + // Kept alongside the message: a rejection is refused before acceptance, and + // the reason says as what. + let reason = + golem_api_grpc::proto::golem::worker::InvocationRejectionReason::try_from( + rejected.reason, + ) + .map(|reason| reason.as_str_name()) + .unwrap_or("UNKNOWN"); terminal = Some(Err(anyhow!( - "Agent invocation rejected: {}", + "Agent invocation rejected ({reason}): {}", rejected.error ))); } diff --git a/golem-worker-executor-test-utils/src/lib.rs b/golem-worker-executor-test-utils/src/lib.rs index a0ef4852f1..39d1abfd25 100644 --- a/golem-worker-executor-test-utils/src/lib.rs +++ b/golem-worker-executor-test-utils/src/lib.rs @@ -161,6 +161,8 @@ use golem_worker_executor::services::worker_proxy::{RemoteWorkerProxy, WorkerPro use golem_worker_executor::services::{ HasActiveAgents, HasAll, HasWorkerService, NoAdditionalDeps, rdbms, }; +use golem_worker_executor::storage::indexed::sqlite::SqliteIndexedStorage; +use golem_worker_executor::storage::indexed::{IndexedStorage, IndexedStorageNamespace}; use golem_worker_executor::storage::keyvalue::KeyValueStorage; use golem_worker_executor::worker::{RetryDecision, Worker, WorkerDeletionHook}; use golem_worker_executor::workerctx::{ @@ -169,7 +171,9 @@ use golem_worker_executor::workerctx::{ InvocationManagement, LogEventEmitBehaviour, P3HttpBodyProducerHook, StatusManagement, UpdateManagement, WorkerCtx, WorkerFilesystemContext, }; -use golem_worker_executor::{Bootstrap, RunDetails, bootstrap_and_run_worker_executor}; +use golem_worker_executor::{ + Bootstrap, RunDetails, bootstrap_and_run_worker_executor, derive_disjoint_sqlite_config, +}; use prometheus::Registry; use regex::Regex; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; @@ -958,7 +962,8 @@ impl TestWorkerExecutor { .ok_or_else(|| anyhow!("worker is not loaded: {owned_agent_id}"))?; golem_worker_executor::services::HasOplog::oplog(worker.as_ref()) .commit(CommitLevel::Always) - .await; + .await + .map_err(|error| anyhow!("oplog commit failed: {error}"))?; Ok(()) } @@ -994,8 +999,8 @@ impl TestWorkerExecutor { .await .ok_or_else(|| anyhow!("worker is not loaded: {owned_agent_id}"))?; let oplog = golem_worker_executor::services::HasOplog::oplog(worker.as_ref()); - let oplog_index = oplog.add(entry).await; - oplog.commit(CommitLevel::Always).await; + let oplog_index = oplog.add(entry).await?; + oplog.commit(CommitLevel::Always).await?; Ok(oplog_index) } @@ -1010,7 +1015,7 @@ impl TestWorkerExecutor { .try_get_worker(&owned_agent_id) .await .ok_or_else(|| anyhow!("worker is not loaded: {owned_agent_id}"))?; - worker.queue_card_revocation(card_id).await; + worker.queue_card_revocation(card_id).await?; Ok(()) } @@ -1032,7 +1037,7 @@ impl TestWorkerExecutor { card_id, )), )) - .await) + .await?) } pub async fn queue_card_install( @@ -1053,7 +1058,7 @@ impl TestWorkerExecutor { card, )), )) - .await; + .await?; Ok(()) } @@ -1357,6 +1362,20 @@ impl TestWorkerExecutor { .test_gate_next_monotonic_clock_start()) } + /// Pauses the next invocation once its `AgentInvocationStarted` is buffered and before it is + /// committed, so the entry sits in the buffer while the test acts. + pub async fn gate_next_invocation_started( + &self, + owned_agent_id: &OwnedAgentId, + ) -> anyhow::Result { + let worker = self + .additional_test_deps + .try_get_worker(owned_agent_id) + .await + .ok_or_else(|| anyhow!("worker {owned_agent_id} is not currently in ActiveAgents"))?; + Ok(worker.owner_execution().test_gate_next_invocation_started()) + } + /// Pauses the next exclusive wall-clock `now` call before it starts durability. pub async fn gate_next_wall_clock_now( &self, @@ -1937,6 +1956,67 @@ pub fn scheduler_sqlite_storage_config( } } +/// Raises the owning epoch stored for `owned_agent_id`'s oplog to `epoch`, as a newer owner +/// opening it on another executor would. The executor's own shard assignment is left alone, so +/// its next oplog write is refused: this is the zombie side of a shard move. +/// +/// Reaches the storage of executors started with the SQLite storage config (`start`, +/// `start_with_overrides`, `start_customized`), whose indexed storage lives in its own file next to +/// the key-value one. +pub async fn take_agent_oplog_over_at_epoch( + deps: &WorkerExecutorTestDependencies, + context: &TestContext, + owned_agent_id: &OwnedAgentId, + epoch: u64, +) -> anyhow::Result<()> { + let storage = SqliteIndexedStorage::configured(&derive_disjoint_sqlite_config( + &sqlite_storage_config(deps, context), + "indexed", + )) + .await + .map_err(|err| anyhow!(err))?; + // The namespace and key the executor's own open records its epoch under. + storage + .set_key_epoch( + "oplog", + "test_take_over", + IndexedStorageNamespace::OpLog { + agent_id: owned_agent_id.agent_id(), + agent_mode: AgentMode::Durable, + }, + &owned_agent_id.agent_id.to_redis_key(), + ShardEpoch(epoch), + ) + .await?; + Ok(()) +} + +/// How many entries the agent's durable oplog holds in storage, read without going through any +/// executor - the same storage [`take_agent_oplog_over_at_epoch`] writes to. +pub async fn agent_oplog_length( + deps: &WorkerExecutorTestDependencies, + context: &TestContext, + owned_agent_id: &OwnedAgentId, +) -> anyhow::Result { + let storage = SqliteIndexedStorage::configured(&derive_disjoint_sqlite_config( + &sqlite_storage_config(deps, context), + "indexed", + )) + .await + .map_err(|err| anyhow!(err))?; + Ok(storage + .length( + "oplog", + "test_length", + IndexedStorageNamespace::OpLog { + agent_id: owned_agent_id.agent_id(), + agent_mode: AgentMode::Durable, + }, + &owned_agent_id.agent_id.to_redis_key(), + ) + .await?) +} + fn apply_sqlite_storage_config( config: &mut GolemConfig, deps: &WorkerExecutorTestDependencies, @@ -2500,7 +2580,7 @@ impl UpdateManagement for TestWorkerCtx { &self, target_revision: ComponentRevision, details: Option, - ) { + ) -> Result<(), WorkerExecutorError> { self.durable_ctx .on_worker_update_failed(target_revision, details) .await @@ -2511,7 +2591,7 @@ impl UpdateManagement for TestWorkerCtx { target_revision: ComponentRevision, new_component_size: u64, new_active_plugins: HashSet, - ) { + ) -> Result<(), WorkerExecutorError> { self.durable_ctx .on_worker_update_succeeded(target_revision, new_component_size, new_active_plugins) .await @@ -4207,7 +4287,10 @@ impl TestOplog { .has_fire_and_forget_rpc_commit_gate(&self.owned_agent_id.agent_id, checkpoint) .await { - self.oplog.commit(CommitLevel::Always).await; + self.oplog + .commit(CommitLevel::Always) + .await + .expect("oplog commit failed at the fire-and-forget RPC gate"); self.additional_test_deps .pause_after_fire_and_forget_rpc_commit(&self.owned_agent_id.agent_id, checkpoint) .await; @@ -4426,8 +4509,17 @@ impl Oplog for TestOplog { self.oplog.task_owner() } - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + entry: OplogEntry, + ) -> Result { self.pause_before_agent_initialization_enqueue(&entry).await; + // Tests inject write failures by entry name. + if let Err(details) = self.check_oplog_add(&entry).await { + return Err(golem_worker_executor::services::oplog::OplogError::Storage( + details, + )); + } if Self::is_consume_body_scope_start(&entry) && self.pause_before_consume_body_scope_start().await { @@ -4448,7 +4540,9 @@ impl Oplog for TestOplog { OplogEntry::CompletionDelivered { start_index, .. } => Some(*start_index), _ => None, }; - let index = self.oplog.add(entry.clone()).await; + // A refused write never reaches storage, so the boundaries below stay unarmed and the + // error propagates to the fence handling instead. + let index = self.oplog.add(entry.clone()).await?; if let Some(start_index) = ended_start { self.observe_rpc_memory_end(start_index); } @@ -4469,7 +4563,7 @@ impl Oplog for TestOplog { if gated { self.pause_at_consume_body_chunk_end_gate().await; } - index + Ok(index) } fn enqueue_add(&self, entry: OplogEntry) -> OplogAddReceipt { @@ -4490,44 +4584,35 @@ impl Oplog for TestOplog { } let this = self.clone(); Box::pin(async move { - let index = pending.await; + // A refused receipt means the entry never landed, so the end boundary stays unarmed. + let index = pending.await?; if let Some(start_index) = ended_start { this.observe_rpc_memory_end(start_index); } if gated { this.pause_at_consume_body_chunk_end_gate().await; } - index + Ok(index) }) } async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, golem_worker_executor::services::oplog::OplogError> + { self.oplog.add_durable_stream_batch(make_batch).await } - async fn fallible_add(&self, entry: OplogEntry) -> Result<(), String> { - self.check_oplog_add(&entry).await?; - self.oplog.fallible_add(entry).await - } - - async fn fallible_add_pair( - &self, - first: OplogEntry, - second: OplogEntry, - ) -> Result<(OplogIndex, OplogIndex), String> { - self.check_oplog_add(&first).await?; - self.check_oplog_add(&second).await?; - self.oplog.fallible_add_pair(first, second).await - } - async fn drop_prefix(&self, last_dropped_id: OplogIndex) -> u64 { self.oplog.drop_prefix(last_dropped_id).await } - async fn commit(&self, level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + level: CommitLevel, + ) -> Result, golem_worker_executor::services::oplog::OplogError> + { self.additional_test_deps .record_oplog_call(&self.owned_agent_id, "commit"); let committed = self.oplog.commit(level).await; @@ -4538,11 +4623,14 @@ impl Oplog for TestOplog { .unwrap() .remove(&self.owned_agent_id.agent_id); if append { + // The injected entry is the point of the hook, so a refusal fails the test rather + // than letting it pass against an oplog that never received it. self.oplog .add(OplogEntry::Suspend { timestamp: golem_common::model::Timestamp::now_utc(), }) - .await; + .await + .expect("append_after_next_oplog_commit was refused by the oplog"); } committed } @@ -4690,7 +4778,7 @@ impl Oplog for TestOplog { &self, serialized_request: Vec, build_start: Box Result + Send>, - ) -> Result { + ) -> Result { let ordered = self .oplog .add_start_with_reserved_raw_payload(serialized_request, build_start) @@ -4721,7 +4809,7 @@ impl Oplog for TestOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let ordered = self .oplog .add_start_with_indexed_reserved_raw_payload(build_request) @@ -4753,7 +4841,14 @@ impl Oplog for TestOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), golem_worker_executor::services::oplog::OplogError> { + // The second entry is only built once the first has its index, so the injected failure + // check covers the pair through the first entry alone. + if let Err(details) = self.check_oplog_add(&start).await { + return Err(golem_worker_executor::services::oplog::OplogError::Storage( + details, + )); + } self.oplog.add_pair(start, make_second).await } @@ -6028,6 +6123,11 @@ struct FakeOwnershipState { /// arrived yet fails, with the `Unknown` that `sharding_not_ready_error` /// produces. That must never be read as "the agent moved". assignment_missing: AtomicBool, + /// Report every check the way an executor whose shard has moved away reports + /// it, without a revoke ever arriving. A revoke gives the agent up here and + /// answers its callers directly, so it is the wrong instrument for a test + /// aimed at the periodic ownership re-check. + agent_moved: AtomicBool, /// Make the next check announce itself, wait, and only then report that the /// agent is not ours. hold_next_check: AtomicBool, @@ -6060,6 +6160,16 @@ impl ShardService for FakeOwnership { }); } + if self.state.agent_moved.load(Ordering::SeqCst) { + self.state + .agent_moved_reports + .fetch_add(1, Ordering::SeqCst); + return Err(WorkerExecutorError::invalid_shard_id( + ShardId::new(0), + HashSet::new(), + )); + } + // Taken, not read, so concurrent checks from other calls fall straight // through to the truth while this one is held at the gate. if self.state.hold_next_check.swap(false, Ordering::SeqCst) { @@ -6154,6 +6264,23 @@ impl ShardService for FakeOwnership { fn try_get_current_assignment(&self) -> Option { self.inner.try_get_current_assignment() } + + fn fence_learned_epochs(&self) -> std::collections::BTreeMap { + self.inner.fence_learned_epochs() + } + + fn retire_fence_learned_epochs( + &self, + reported: &std::collections::BTreeMap, + ) { + self.inner.retire_fence_learned_epochs(reported) + } +} + +impl golem_worker_executor::services::oplog::OplogFenceObserver for FakeOwnership { + fn fenced(&self, fence: &golem_worker_executor::services::oplog::OplogFence) { + self.inner.fenced(fence) + } } /// A test's handle on a [`FakeOwnership`]: what it should report, when to hold @@ -6171,8 +6298,15 @@ impl OwnershipControls { self.state.assignment_missing.store(true, Ordering::SeqCst); } + /// Report every ownership check the way an executor whose shard has moved + /// away reports it. Lasts until [`Self::stop_pretending`]. + pub fn pretend_the_agent_moved(&self) { + self.state.agent_moved.store(true, Ordering::SeqCst); + } + pub fn stop_pretending(&self) { self.state.assignment_missing.store(false, Ordering::SeqCst); + self.state.agent_moved.store(false, Ordering::SeqCst); } /// Hold the next ownership check open, and return once one has arrived. @@ -6215,6 +6349,7 @@ pub fn fake_ownership() -> (TestExecutorOverrides, OwnershipControls) { let state = Arc::new(FakeOwnershipState { assignment_missing: AtomicBool::new(false), + agent_moved: AtomicBool::new(false), hold_next_check: AtomicBool::new(false), assignment_missing_reports: AtomicUsize::new(0), agent_moved_reports: AtomicUsize::new(0), diff --git a/golem-worker-executor/benches/oplog_read.rs b/golem-worker-executor/benches/oplog_read.rs index e9a6df2e55..1f9bc4dc20 100644 --- a/golem-worker-executor/benches/oplog_read.rs +++ b/golem-worker-executor/benches/oplog_read.rs @@ -72,6 +72,7 @@ impl Fixture { self.initial_metadata.clone(), last_known_status(), execution_status(), + None, ) .await } @@ -176,13 +177,14 @@ async fn open_fixture(initial_entries: u64) -> Fixture { initial_metadata.clone(), last_known_status(), execution_status(), + None, ) .await; for value in 1..initial_entries { - oplog.add(entry(value)).await; + oplog.add(entry(value)).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); Fixture { oplog, @@ -200,7 +202,7 @@ async fn primary_fixture() -> Fixture { async fn buffered_fixture() -> Fixture { let fixture = open_fixture(ENTRY_COUNT - 8).await; for value in ENTRY_COUNT - 8..ENTRY_COUNT { - fixture.oplog.add(entry(value)).await; + fixture.oplog.add(entry(value)).await.unwrap(); } fixture } @@ -230,9 +232,9 @@ async fn cross_tier_fixture() -> Fixture { Some(true) ); for value in ARCHIVE_BOUNDARY..ENTRY_COUNT { - fixture.oplog.add(entry(value)).await; + fixture.oplog.add(entry(value)).await.unwrap(); } - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); fixture } diff --git a/golem-worker-executor/config/worker-executor.sample.env b/golem-worker-executor/config/worker-executor.sample.env index 94dac07687..ce1f2848c1 100644 --- a/golem-worker-executor/config/worker-executor.sample.env +++ b/golem-worker-executor/config/worker-executor.sample.env @@ -65,7 +65,10 @@ GOLEM__HTTP_CLIENT__CONFIG__MAX_CONNECTIONS_PER_HOST=20 GOLEM__HTTP_CLIENT__CONFIG__MAX_HOST_ENTRIES=1024 GOLEM__HTTP_CLIENT__CONFIG__MAX_IDLE_PER_HOST=8 GOLEM__HTTP_CLIENT__CONFIG__MAX_TOTAL_CONNECTIONS=200 -GOLEM__INDEXED_STORAGE__TYPE="KVStoreRedis" +GOLEM__INDEXED_STORAGE__TYPE="Sqlite" +GOLEM__INDEXED_STORAGE__CONFIG__DATABASE="../data/worker-executor-indexed-storage.db" +GOLEM__INDEXED_STORAGE__CONFIG__FOREIGN_KEYS=false +GOLEM__INDEXED_STORAGE__CONFIG__MAX_CONNECTIONS=10 GOLEM__INDEXED_STORAGE_RETRY__MAX_ATTEMPTS=3 GOLEM__INDEXED_STORAGE_RETRY__MAX_DELAY="1s" GOLEM__INDEXED_STORAGE_RETRY__MAX_JITTER_FACTOR=0.15 diff --git a/golem-worker-executor/config/worker-executor.toml b/golem-worker-executor/config/worker-executor.toml index a0e8062a03..40f83dbee1 100644 --- a/golem-worker-executor/config/worker-executor.toml +++ b/golem-worker-executor/config/worker-executor.toml @@ -111,9 +111,12 @@ max_idle_per_host = 8 max_total_connections = 200 [indexed_storage] -type = "KVStoreRedis" +type = "Sqlite" [indexed_storage.config] +database = "../data/worker-executor-indexed-storage.db" +foreign_keys = false +max_connections = 10 [indexed_storage_retry] max_attempts = 3 diff --git a/golem-worker-executor/db/migration/indexed/postgres/003_indexed_key_epoch.sql b/golem-worker-executor/db/migration/indexed/postgres/003_indexed_key_epoch.sql new file mode 100644 index 0000000000..6fb3196dd0 --- /dev/null +++ b/golem-worker-executor/db/migration/indexed/postgres/003_indexed_key_epoch.sql @@ -0,0 +1,17 @@ +-- The writer generation recorded for a key in `index_storage`. +-- +-- One row per key: a monotonic epoch, and the writer that recorded it. An append that asserts an +-- epoch is checked against this row inside the same transaction as its insert, so a writer holding +-- a stale epoch is refused without leaving anything behind. A missing row refuses too: the row is +-- written before the first entry and removed before the entries are. +-- +-- The epoch alone cannot separate two writers presenting the same number, so the row names the +-- writer as well: the one that recorded it may go on writing at that epoch, and any other is +-- refused. +CREATE TABLE indexed_key_epoch ( + namespace TEXT NOT NULL, + key TEXT NOT NULL, + epoch BIGINT NOT NULL, + writer TEXT NOT NULL, + PRIMARY KEY (namespace, key) +); diff --git a/golem-worker-executor/db/migration/indexed/sqlite/003_indexed_key_epoch.sql b/golem-worker-executor/db/migration/indexed/sqlite/003_indexed_key_epoch.sql new file mode 100644 index 0000000000..16acdd11fe --- /dev/null +++ b/golem-worker-executor/db/migration/indexed/sqlite/003_indexed_key_epoch.sql @@ -0,0 +1,9 @@ +-- The writer generation recorded for a key in `index_storage`. See the postgres migration of the +-- same name. +CREATE TABLE indexed_key_epoch ( + namespace TEXT NOT NULL, + key TEXT NOT NULL, + epoch INTEGER NOT NULL, + writer TEXT NOT NULL, + PRIMARY KEY (namespace, key) +); diff --git a/golem-worker-executor/src/durable_host/call_coordinator.rs b/golem-worker-executor/src/durable_host/call_coordinator.rs index ba1565785b..6045d68a40 100644 --- a/golem-worker-executor/src/durable_host/call_coordinator.rs +++ b/golem-worker-executor/src/durable_host/call_coordinator.rs @@ -142,7 +142,7 @@ impl<'a, Ctx: WorkerCtx> DurableCallCoordinator<'a, Ctx> { .public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; // The status checkpoint is only safe after the durable boundary has committed. self.ctx.maybe_mid_invocation_checkpoint().await; } @@ -694,7 +694,7 @@ where reason, ), }; - worker.add_and_commit_oplog(entry).await; + worker.add_and_commit_oplog(entry).await?; Ok(()) } @@ -740,7 +740,7 @@ where reason, ), }; - worker.add_and_commit_oplog(entry).await; + worker.add_and_commit_oplog(entry).await?; Ok(()) } @@ -802,7 +802,7 @@ where card_id, wallet_generation, )) - .await; + .await?; } Ok(()) } @@ -1045,7 +1045,7 @@ where } worker .queue_card_revocations_locked(&revoked_card_ids) - .await; + .await?; Ok(()) } @@ -1176,7 +1176,7 @@ where target_holder, store.with(|mut access| get_ctx(access.data_mut()).state.wallet_generation), )) - .await; + .await?; Ok(()) } @@ -1258,7 +1258,7 @@ where retry.installed_card.card_id(), target_holder, )) - .await; + .await?; Ok(()) } @@ -1327,7 +1327,7 @@ where affected_wallets, local_wallet_generation: wallet_generation, }) - .await; + .await?; Ok(()) } @@ -1564,7 +1564,7 @@ where target_revision, Some(details.clone()), )) - .await; + .await?; tracing::warn!( "Worker failed to update to {}: {}, update attempt aborted", target_revision, @@ -1600,6 +1600,6 @@ where component_size, active_plugins, ) - .await; + .await?; Ok(()) } diff --git a/golem-worker-executor/src/durable_host/clocks/monotonic_clock.rs b/golem-worker-executor/src/durable_host/clocks/monotonic_clock.rs index 4fe3dfc653..343d149688 100644 --- a/golem-worker-executor/src/durable_host/clocks/monotonic_clock.rs +++ b/golem-worker-executor/src/durable_host/clocks/monotonic_clock.rs @@ -138,7 +138,7 @@ impl HostWithStore for HasSelf InFunctionRetryHost for ScopedRetryHo retry_from: OplogIndex, inside_atomic_region: bool, retry_policy_state: Option, - ) { + ) -> Result<(), crate::services::oplog::OplogError> { self.inner .append_retry_error_entry(retry_from, inside_atomic_region, retry_policy_state) - .await; + .await } } @@ -2137,8 +2137,16 @@ impl DurableCallSession { } ScopeReplayRecovery::Default => {} } - let begin_index = - Self::append_access_scope_start(prepared, scope_name, function_type).await; + let begin_index = Self::append_access_scope_start(prepared, scope_name, function_type) + .await + .map_err(|error| { + ( + error, + AccessStartCleanup { + atomic_lease: prepared.atomic_lease.clone(), + }, + ) + })?; Ok(AccessOpenedScope { begin_index, replay_handle: None, @@ -2334,7 +2342,16 @@ impl DurableCallSession { }; let Some((begin_index, replay_handle)) = claimed_scope else { let begin_index = - Self::append_access_scope_start(prepared, scope_name, function_type).await; + Self::append_access_scope_start(prepared, scope_name, function_type) + .await + .map_err(|error| { + ( + error, + AccessStartCleanup { + atomic_lease: prepared.atomic_lease.clone(), + }, + ) + })?; prepared .public_state .worker() @@ -2434,6 +2451,8 @@ impl DurableCallSession { start: begin_index.next(), end: pending.replay_target().next(), }; + // Refused, the scope must not re-run live: its first attempt would be + // replayed by the shard's new owner with no `Jump` skipping it. prepared .public_state .worker() @@ -2441,7 +2460,15 @@ impl DurableCallSession { prepared.entity_parent_start_index, deleted_region, )) - .await; + .await + .map_err(|error| { + ( + WorkerExecutorError::from(error), + AccessStartCleanup { + atomic_lease: prepared.atomic_lease.clone(), + }, + ) + })?; prepared .public_state .worker() @@ -2500,11 +2527,14 @@ impl DurableCallSession { } } + /// Appends the scope `Start` that the call's side effect waits on. A `Start` the storage + /// refused is returned as the fence, so the effect never runs for a scope the shard's new + /// owner cannot see. async fn append_access_scope_start( prepared: &mut PreparedAccessStart, scope_name: HostFunctionName, function_type: DurableFunctionType, - ) -> OplogIndex { + ) -> Result { prepared .public_state .worker() @@ -2518,6 +2548,7 @@ impl DurableCallSession { durable_function_type: function_type, }) .await + .map_err(WorkerExecutorError::from) } fn finish_access_start( @@ -3261,7 +3292,7 @@ impl DurableCallSession { self.start_idx ))); } - oplog.add(end).await; + oplog.add(end).await?; self.execution_scope.release_atomic_lease(); DurableCallCoordinator::new(ctx) .finish(self.retry.function_type(), self.boundary, false) @@ -3470,13 +3501,13 @@ impl DurableCallSession { let end_append = oplog.enqueue_add(end); let post_end_append = post_end_entry.map(|entry| oplog.enqueue_add(entry)); let terminal = tokio::spawn(async move { - end_append.await; + end_append.await?; // A deferred-delivery call's mandatory post-`End` entry (e.g. its durable // `FinishSpan`) is appended by the same owned task: it is recorded even when the // completing future is torn right after the `End`, so replay can rely on it // unconditionally following the `End` (any discard marker chains after this task). if let Some(append) = post_end_append { - append.await; + append.await?; } Ok(()) }); @@ -4822,7 +4853,7 @@ where response: None, forced_commit: true, }) - .await; + .await?; } else if let Some(handle) = replay_handle { match replay_state.await_resolution_outcome(handle).await? { ResolutionOutcome::Resolved(Resolution::Completed { .. }) => {} @@ -4855,7 +4886,7 @@ where response: None, forced_commit: true, }) - .await; + .await?; } } } @@ -4877,7 +4908,7 @@ where public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; if let Some(min_exposed_marker) = store.with(|mut access| { let ctx = get_ctx(access.data_mut()); if ctx.state.at_clean_checkpoint_boundary() { @@ -5004,7 +5035,7 @@ where if is_live { worker .add_to_oplog(OplogEntry::finish_span(parent_start_index, span_id.clone())) - .await; + .await?; } store.with(|mut access| { diff --git a/golem-worker-executor/src/durable_host/concurrent/delivery.rs b/golem-worker-executor/src/durable_host/concurrent/delivery.rs index 691c3aedbe..74e1047487 100644 --- a/golem-worker-executor/src/durable_host/concurrent/delivery.rs +++ b/golem-worker-executor/src/durable_host/concurrent/delivery.rs @@ -64,7 +64,7 @@ impl OrderedAppend { async fn wait(self) -> Result<(), WorkerExecutorError> { match self { Self::Receipt(receipt) => { - receipt.await; + receipt.await?; Ok(()) } Self::Task(task) => task.await.map_err(|err| { @@ -120,7 +120,16 @@ impl CompletionMarkerRecorder { let _ = done.send(Err(error)); return; } - let marker_idx = marker_append.await; + // Reported, not panicked: the process is built with `panic = "abort"`, so panicking + // in this task would take the whole executor down over one agent. The awaiter + // classifies a fenced marker as `ShardLost` and gives that agent up on its own. + let marker_idx = match marker_append.await { + Ok(index) => index, + Err(error) => { + let _ = done.send(Err(error.into())); + return; + } + }; match kind { CompletionMarkerKind::Delivered => { replay_state.record_delivered_completion(start_idx, marker_idx) diff --git a/golem-worker-executor/src/durable_host/concurrent/drop_events.rs b/golem-worker-executor/src/durable_host/concurrent/drop_events.rs index ab200031f5..9582020c6d 100644 --- a/golem-worker-executor/src/durable_host/concurrent/drop_events.rs +++ b/golem-worker-executor/src/durable_host/concurrent/drop_events.rs @@ -117,7 +117,7 @@ impl DroppedCall { start_index: self.start_idx, partial, }; - oplog.add(cancelled).await; + oplog.add(cancelled).await?; Ok(()) } } diff --git a/golem-worker-executor/src/durable_host/concurrent/tests.rs b/golem-worker-executor/src/durable_host/concurrent/tests.rs index a32d31fe1f..ee68652bac 100644 --- a/golem-worker-executor/src/durable_host/concurrent/tests.rs +++ b/golem-worker-executor/src/durable_host/concurrent/tests.rs @@ -365,7 +365,8 @@ async fn live_delivery_token( )))), durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); seed_oplog .add(OplogEntry::End { timestamp: Timestamp::now_utc(), @@ -373,7 +374,8 @@ async fn live_delivery_token( response: None, forced_commit: false, }) - .await; + .await + .unwrap(); let seed_oplog_dyn: Arc = seed_oplog; let replay_state = ReplayState::new_for_owner( golem_common::model::OwnedAgentId { @@ -461,6 +463,60 @@ async fn completion_delivery_delivered_records_marker_via_drain() { } } +/// A completion marker whose oplog write is refused because the shard moved must be reported to +/// whoever awaits the receipt. Panicking here would abort the executor - and every other healthy +/// agent resident on it - over one agent that another executor now owns. +#[test] +async fn a_fenced_completion_marker_is_reported_rather_than_panicked() { + let agent_id = golem_common::model::AgentId { + component_id: golem_common::model::component::ComponentId::new(), + agent_id: "fenced-completion-marker-test".to_string(), + }; + let oplog = Arc::new(InMemoryOplog::fenced(crate::services::oplog::OplogFence { + agent_id: agent_id.clone(), + expected_epoch: golem_common::model::ShardEpoch(3), + actual_epoch: Some(golem_common::model::ShardEpoch(4)), + writer_conflict: false, + })); + let seed_oplog = Arc::new(InMemoryOplog::new()); + seed_oplog + .add(OplogEntry::NoOp { + timestamp: Timestamp::now_utc(), + entity_parent_start_index: None, + }) + .await + .unwrap(); + let seed_oplog_dyn: Arc = seed_oplog; + let replay_state = ReplayState::new_for_owner( + golem_common::model::OwnedAgentId { + environment_id: golem_common::model::environment::EnvironmentId::new(), + agent_id, + }, + seed_oplog_dyn, + golem_common::model::regions::DeletedRegions::default(), + None, + crate::durable_host::tool::operation::OwnerToolOperations::new(), + ) + .await + .expect("failed to build replay state"); + let oplog_dyn: Arc = oplog; + let recorder = CompletionMarkerRecorder::new(oplog_dyn, replay_state); + + let mut receipt = recorder.record(idx(10), CompletionMarkerKind::Delivered, None); + + match await_marker_receipt(&mut receipt).await { + Err(WorkerExecutorError::OplogFenced { + expected_epoch, + actual_epoch, + .. + }) => { + assert_eq!(expected_epoch, 3); + assert_eq!(actual_epoch, Some(4)); + } + other => panic!("expected the fenced marker append to be reported, got {other:?}"), + } +} + #[test] async fn completion_delivery_markers_preserve_handoff_order() { let oplog = Arc::new(InMemoryOplog::new()); @@ -470,7 +526,8 @@ async fn completion_delivery_markers_preserve_handoff_order() { timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); let seed_oplog_dyn: Arc = seed_oplog; let replay_state = ReplayState::new_for_owner( golem_common::model::OwnedAgentId { @@ -684,7 +741,8 @@ async fn tail_gated_token_over_crash_tail( timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); oplog .add(OplogEntry::Start { timestamp: Timestamp::now_utc(), @@ -697,7 +755,8 @@ async fn tail_gated_token_over_crash_tail( )))), durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); oplog .add(OplogEntry::End { timestamp: Timestamp::now_utc(), @@ -709,9 +768,10 @@ async fn tail_gated_token_over_crash_tail( ))), forced_commit: false, }) - .await; + .await + .unwrap(); for entry in extra_tail { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog_dyn: Arc = oplog.clone(); let replay_state = ReplayState::new_for_owner( @@ -953,6 +1013,9 @@ struct InMemoryOplog { next_reserved: Arc, next_commit: Arc>, append_progress: Arc, + /// When set, every append is refused the way a fencing backend refuses one whose asserted + /// shard epoch is stale: the shard moved to another executor while this oplog was open. + fence: Option, } impl InMemoryOplog { @@ -963,6 +1026,15 @@ impl InMemoryOplog { next_reserved: Arc::new(std::sync::atomic::AtomicU64::new(1)), next_commit: Arc::new(tokio::sync::Mutex::new(1)), append_progress: Arc::new(tokio::sync::Notify::new()), + fence: None, + } + } + + /// An oplog whose shard has already moved: every append is refused with `fence`. + fn fenced(fence: crate::services::oplog::OplogFence) -> Self { + Self { + fence: Some(fence), + ..Self::new() } } @@ -976,17 +1048,24 @@ impl InMemoryOplog { next_reserved: Arc::new(std::sync::atomic::AtomicU64::new(1)), next_commit: Arc::new(tokio::sync::Mutex::new(1)), append_progress: Arc::new(tokio::sync::Notify::new()), + fence: None, } } } #[async_trait] impl Oplog for InMemoryOplog { - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + entry: OplogEntry, + ) -> Result { self.enqueue_add(entry).await } fn enqueue_add(&self, entry: OplogEntry) -> crate::services::oplog::OplogAddReceipt { + if let Some(fence) = self.fence.clone() { + return Box::pin(async move { Err(crate::services::oplog::OplogError::Fenced(fence)) }); + } let index = self .next_reserved .fetch_add(1, std::sync::atomic::Ordering::Relaxed); @@ -1027,9 +1106,11 @@ impl Oplog for InMemoryOplog { } }); Box::pin(async move { - receipt - .await - .expect("the in-memory oplog append task must reply") + Ok({ + receipt + .await + .expect("the in-memory oplog append task must reply") + }) }) } @@ -1037,7 +1118,7 @@ impl Oplog for InMemoryOplog { &self, _start: OplogEntry, _make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { // The concurrent (p3) durability path under test never writes sequential-adapter pairs, // and a routed-through-`add` implementation could not honor the atomic pair contract. unreachable!("add_pair is not used by the concurrent durability tests") @@ -1050,11 +1131,11 @@ impl Oplog for InMemoryOplog { dyn FnOnce(golem_common::model::oplog::RawOplogPayload) -> Result + Send, >, - ) -> Result { + ) -> Result { let entry = build_start( golem_common::model::oplog::RawOplogPayload::SerializedInline(serialized_request), )?; - let index = self.add(entry.clone()).await; + let index = self.add(entry.clone()).await?; Ok(crate::services::oplog::OrderedOplogStart { index, entry, @@ -1065,7 +1146,7 @@ impl Oplog for InMemoryOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let mut entries = self.entries.lock().await; let index = OplogIndex::from_u64(entries.len() as u64 + 1); let (serialized_request, build_start) = build_request(index)?; @@ -1087,8 +1168,11 @@ impl Oplog for InMemoryOplog { async fn commit( &self, _level: CommitLevel, - ) -> std::collections::BTreeMap { - std::collections::BTreeMap::new() + ) -> Result< + std::collections::BTreeMap, + crate::services::oplog::OplogError, + > { + Ok(std::collections::BTreeMap::new()) } async fn current_oplog_index(&self) -> OplogIndex { @@ -1164,7 +1248,8 @@ async fn dropped_cancellable_call_records_cancelled_at_next_drain_point() { )))), durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); let (tx, mut rx) = mpsc::unbounded_channel(); { @@ -1226,7 +1311,8 @@ async fn access_terminal_end_is_appended_before_cleanup_and_permit_release() { )))), durable_function_type: DurableFunctionType::ReadRemote, }) - .await; + .await + .unwrap(); let permit_counter = Arc::new(AtomicUsize::new(0)); let (cleanup_tx, mut cleanup_rx) = mpsc::unbounded_channel(); @@ -1593,10 +1679,11 @@ impl InFunctionRetryHost for RetryHostProbe { retry_from: OplogIndex, inside_atomic_region: bool, _retry_policy_state: Option, - ) { + ) -> Result<(), crate::services::oplog::OplogError> { self.appended_retry_from.push(retry_from); self.appended_inside_atomic_region .push(inside_atomic_region); + Ok(()) } } diff --git a/golem-worker-executor/src/durable_host/durability.rs b/golem-worker-executor/src/durable_host/durability.rs index c83ea165bb..151397b2ac 100644 --- a/golem-worker-executor/src/durable_host/durability.rs +++ b/golem-worker-executor/src/durable_host/durability.rs @@ -24,7 +24,7 @@ use crate::metrics::wasm::{ use crate::model::ExecutionStatus; use crate::preview2::golem::durability::durability; use crate::services::environment_state::EnvironmentStateService; -use crate::services::oplog::OplogOps; +use crate::services::oplog::{OplogError, OplogOps}; use crate::services::{HasOplog, HasWorker}; use crate::workerctx::WorkerCtx; use anyhow::Error; @@ -673,12 +673,14 @@ pub trait InFunctionRetryHost { } /// Writes an `OplogEntry::Error` entry for an in-function retry attempt, and commits. + /// + /// A refusal is returned: the retry must not run again for an attempt that was never recorded. async fn append_retry_error_entry( &mut self, retry_from: OplogIndex, inside_atomic_region: bool, retry_policy_state: Option, - ); + ) -> Result<(), OplogError>; } pub(crate) fn collect_named_retry_policies( @@ -960,8 +962,15 @@ impl InFunctionRetryState { } let inside_atomic_region = ctx.retry_context_atomic_region_had_side_effects(); - ctx.append_retry_error_entry(retry_point, inside_atomic_region, retry_policy_state) - .await; + // Refused, the shard has a new owner: the failure is returned instead of retried, and the + // trap it becomes is classified as the lost shard. + if ctx + .append_retry_error_entry(retry_point, inside_atomic_region, retry_policy_state) + .await + .is_err() + { + return AsyncRetryDecision::FallBackToTrap; + } self.retry_count += 1; debug!( @@ -1491,7 +1500,7 @@ impl durability::HostLiveCustomDurableInvocat response: Some(response), forced_commit, }) - .await; + .await?; let checkpoint = accessor.with(|mut access| { let ctx = access.get(); ctx.state.active_custom_invocations.remove(&start_index); @@ -1651,19 +1660,20 @@ impl durability::HostWithStore .upload_payload_owned(request) .await .map_err(|err| format!("Failed to store durable function request: {err}"))?; - Ok::<_, String>( - worker - .add_and_commit_oplog(OplogEntry::Start { - timestamp: Timestamp::now_utc(), - parent_start_index, - function_name, - invocation_id: Some(start_invocation_id), - observational_owner: None, - request: Some(persisted_request), - durable_function_type: start_function_type, - }) - .await, - ) + // A refused `Start` fails the begin: the guest must not perform a side effect the + // shard's new owner, finding no `Start`, would perform again. + worker + .add_and_commit_oplog(OplogEntry::Start { + timestamp: Timestamp::now_utc(), + parent_start_index, + function_name, + invocation_id: Some(start_invocation_id), + observational_owner: None, + request: Some(persisted_request), + durable_function_type: start_function_type, + }) + .await + .map_err(|err| format!("Failed to record durable function start: {err}")) }); let cancellation_worker = accessor.with(|mut access| access.get().public_state.worker()); @@ -1684,7 +1694,7 @@ impl durability::HostWithStore start_index, partial: None, }) - .await; + .await?; Ok(Some(start_index)) } (_, Err(err)) if matches!(verdict, CustomBeginVerdict::Cancelled) => { @@ -1856,9 +1866,9 @@ impl InFunctionRetryHost for DurableWorkerCtx { retry_from: OplogIndex, inside_atomic_region: bool, retry_policy_state: Option, - ) { + ) -> Result<(), OplogError> { if self.state.durability_is_suppressed() { - return; + return Ok(()); } use golem_common::model::oplog::AgentError; @@ -1870,7 +1880,11 @@ impl InFunctionRetryHost for DurableWorkerCtx { inside_atomic_region, retry_policy_state, ); - self.public_state.worker().add_and_commit_oplog(entry).await; + self.public_state + .worker() + .add_and_commit_oplog(entry) + .await?; + Ok(()) } } @@ -2376,7 +2390,7 @@ impl InFunctionRetryHost for TaskRetryContext { retry_from: OplogIndex, inside_atomic_region: bool, retry_policy_state: Option, - ) { + ) -> Result<(), OplogError> { use golem_common::model::oplog::AgentError; let entry = OplogEntry::error( self.entity_parent_start_index, @@ -2386,9 +2400,10 @@ impl InFunctionRetryHost for TaskRetryContext { inside_atomic_region, retry_policy_state.clone(), ); - self.worker.add_and_commit_oplog(entry).await; + self.worker.add_and_commit_oplog(entry).await?; self.current_retry_policy_state = retry_policy_state; + Ok(()) } } @@ -2685,9 +2700,10 @@ mod tests { _retry_from: OplogIndex, _inside_atomic_region: bool, retry_policy_state: Option, - ) { + ) -> Result<(), OplogError> { self.retry_entries_appended += 1; self.current_retry_policy_state = retry_policy_state; + Ok(()) } } diff --git a/golem-worker-executor/src/durable_host/durable_session/mod.rs b/golem-worker-executor/src/durable_host/durable_session/mod.rs index f098f52e73..b71fa7d21e 100644 --- a/golem-worker-executor/src/durable_host/durable_session/mod.rs +++ b/golem-worker-executor/src/durable_host/durable_session/mod.rs @@ -898,16 +898,19 @@ impl StreamSession { } /// Commits and indexes a session record through the producer's owned write path. - #[cfg(test)] + /// + /// Every failure goes back to the session, not only a fence: a producer retired or poisoned + /// under the session answers `RecoveryRequired`, and a failed publication `LiveBus`, neither + /// of which says the record is invalid. async fn append_record( &self, context: Option<&StreamWriteContext>, record: StreamSessionRecord, - ) { + ) -> Result<(), String> { self.producer .append_session_record_attributed(context, self.entity_parent_start_index, record) .await - .expect("internally generated durable session record is valid"); + .map_err(|error| error.to_string()) } async fn try_append_record( @@ -3368,14 +3371,8 @@ impl StreamSession { return Err("durable RPC result conflicts with its caller journal".to_string()); } } else { - self.producer - .append_session_record_attributed( - None, - self.entity_parent_start_index, - StreamSessionRecord::InvocationResult(record), - ) - .await - .map_err(|error| error.to_string())?; + self.append_record(None, StreamSessionRecord::InvocationResult(record)) + .await?; self.commit_consumer_journal().await?; } self.decode_initial(canonical, &mappings, SessionStreamRole::Output) @@ -6097,15 +6094,7 @@ impl DurableInputEndpoint { }), }; if !journaled { - streams - .producer - .append_session_record_attributed( - None, - streams.entity_parent_start_index, - record, - ) - .await - .map_err(|error| error.to_string())?; + streams.append_record(None, record).await?; streams.commit_consumer_journal().await?; let committed_through = queued_events .back() diff --git a/golem-worker-executor/src/durable_host/durable_session/tests.rs b/golem-worker-executor/src/durable_host/durable_session/tests.rs index 8b8e970c0f..d4f1b0f43f 100644 --- a/golem-worker-executor/src/durable_host/durable_session/tests.rs +++ b/golem-worker-executor/src/durable_host/durable_session/tests.rs @@ -1,7 +1,7 @@ use super::*; use crate::durable_host::durable_stream::AttachedStreamSegmentSource; use crate::durable_host::durable_stream::tests::{ - TestIdentity, TestOplog, attachment_key, identity, registration, + TestIdentity, TestOplog, attachment_key, identity, registration, test_fence, }; use crate::durable_host::schema_value_stream::ExecutorProjectionStreams; use crate::durable_host::stream_bus::LiveStreamEventPayload; @@ -319,7 +319,8 @@ async fn assert_fork_consumer_payloads(overlay_before_fork: bool) { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::ForkCut(cut))), }) - .await; + .await + .unwrap(); let stale = StreamSession::new( producer, oplog.clone(), @@ -508,7 +509,8 @@ async fn session_payload_reader_rejects_malformed_records_and_wrong_locators() { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::Prepared(prepared))), }) - .await; + .await + .unwrap(); assert!( producer .read_session_record(malformed) @@ -522,7 +524,8 @@ async fn session_payload_reader_rejects_malformed_records_and_wrong_locators() { timestamp: golem_common::model::Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); assert!( producer .read_session_record(wrong) @@ -894,7 +897,7 @@ struct TestConsumerJournal(Arc); #[async_trait::async_trait] impl DurableStreamConsumerJournal for TestConsumerJournal { async fn commit(&self) -> Result<(), String> { - self.0.commit(CommitLevel::Always).await; + self.0.commit(CommitLevel::Always).await.unwrap(); Ok(()) } @@ -979,6 +982,7 @@ async fn append_prepared_pending( Vec::new(), )) .await + .expect("oplog write") } #[test] @@ -1273,15 +1277,16 @@ async fn guest_byte_drain_after_partial_fork_replays_prefix_and_resumes_suffix() ) .await { - fork_oplog.add(entry).await; + fork_oplog.add(entry).await.unwrap(); } fork_oplog .add( DurableStreamOplogRecord::Session(None, Box::new(StreamSessionRecord::ForkCut(cut))) .into_inline_entry(), ) - .await; - fork_oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + fork_oplog.commit(CommitLevel::Always).await.unwrap(); let fork = DurableStreamStore::load( fork_oplog.clone(), target_owner.environment_id, @@ -1511,7 +1516,7 @@ async fn concurrent_nested_mapping_reuses_identity_before_commit_callback_finish let reached = reached.clone(); let release = release.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(receipt) = receipt { let _ = receipt.send(()); } @@ -2930,6 +2935,147 @@ async fn foreign_preparation_releases_local_owner_when_rpc_requests_retirement() retirement_from_foreign_rpc(true).await; } +/// A foreign mapping prepared while this agent's oplog is already fenced must fail as a fence and +/// leave the session unlocked. +/// +/// Activating a foreign mapping is one of the paths that runs while a shard is being taken away: +/// it writes a session record, so a latched fence has to stop it rather than let it record a +/// mapping the new owner will never see. The session lock matters as much as the error - a +/// preparation that failed while holding it would strand every later call on this session, and the +/// agent is about to be given up, not restarted. +#[test] +#[test_r::timeout("15s")] +async fn a_fenced_oplog_refuses_a_foreign_mapping_and_releases_the_session() { + let local = identity(); + let mut remote = identity(); + remote.agent_id.agent_id.push_str("-remote"); + remote.invocation.callee = remote.agent_id.clone(); + let remote_producer = DurableStreamStore::load( + Arc::new(TestOplog::default()), + remote.environment_id, + remote.agent_id.clone(), + remote.fingerprint, + None, + ) + .await + .unwrap(); + let handle = remote_producer + .register( + None, + registration( + &remote, + StreamRegistrationCoordinate::Root { + invocation_id: remote.invocation.clone(), + root_kind: StreamRootKind::MethodResult, + recursive_value_path: Vec::new(), + }, + StreamSourceKind::InvocationOutput, + ), + ) + .await + .unwrap() + .value; + + let oplog = Arc::new(TestOplog::default()); + let producer = DurableStreamStore::load( + oplog.clone(), + local.environment_id, + local.agent_id.clone(), + local.fingerprint, + None, + ) + .await + .unwrap(); + let attachment_id = AttachmentId::primary( + local.environment_id, + &local.agent_id, + &local.invocation.idempotency_key, + ) + .unwrap(); + let attempt_id = AttemptId::fresh(); + let pending_invocation_oplog_index = append_prepared_pending( + &producer, + &oplog, + &local, + attachment_id, + attempt_id, + &handle, + SessionStreamRole::Input, + ) + .await; + producer + .append_session_record( + None, + StreamSessionRecord::Attached(StreamSessionAttachedRecord { + format_version: DURABLE_STREAM_FORMAT_VERSION, + session_key: local.invocation.idempotency_key.clone(), + attachment_id, + attempt_id, + epoch: 1, + pending_invocation_oplog_index, + }), + ) + .await + .unwrap(); + let streams = StreamSession::new( + producer.clone(), + oplog.clone(), + StreamRegistrationInvocation::Local(local.invocation.idempotency_key.clone()), + [StreamSessionMappingRecord { + transport_stream_id: 7, + handle: handle.clone(), + role: SessionStreamRole::Input, + }], + ) + .with_consumer_journal(Arc::new(TestConsumerJournal(oplog.clone()))) + .with_attachment(1, attempt_id) + .with_rpc(Arc::new(AttachedProducerRpc { + producer: remote_producer, + cancellation_owner: None, + stall_next_cancel: Default::default(), + scripted_reads: Mutex::default(), + pending_read: Mutex::default(), + read_requests: Mutex::default(), + })) + .with_auth_ctx(AuthCtx::System); + + let committed_before = oplog.committed_length(); + // Both halves of what a real fenced oplog does: the latch every reader consults, and the + // refusal an add itself gets once it has latched (`PrimaryOplogState::refuse_if_fenced`). + oplog.latch_fence(test_fence()); + oplog.refuse_adds(test_fence()); + + let error = tokio::time::timeout( + Duration::from_secs(5), + streams.prepare_foreign_mapping( + StreamSessionMappingRecord { + transport_stream_id: 7, + handle, + role: SessionStreamRole::Input, + }, + 1, + ), + ) + .await + .expect("a foreign mapping on a fenced oplog hung instead of failing") + .unwrap_err(); + + assert!( + error.contains("Fenced"), + "a foreign mapping on a fenced oplog must report the fence itself, so the caller reroutes \ + instead of treating it as a local failure, got {error}" + ); + assert_eq!( + oplog.committed_length(), + committed_before, + "nothing may be committed for a mapping the storage refused" + ); + assert!( + streams.session_lock.try_lock().is_ok(), + "the session lock has to be released, or every later call on this session strands" + ); +} + async fn retirement_from_foreign_rpc(prepare: bool) { let local = identity(); let mut remote = identity(); @@ -3478,7 +3624,8 @@ async fn session_cancellation_retains_history_and_is_idempotent_under_backpressu }, }), ) - .await; + .await + .unwrap(); if index == 1 { producer .end(None, output.stream_id, 0, StreamEndResult::Ok) @@ -3587,6 +3734,7 @@ async fn late_output_cancellation_selects_fields_and_preserves_replay_drains() { }), ) .await + .unwrap(); } } let root = SchemaType::record( @@ -4044,7 +4192,7 @@ async fn root_union_stream_coordinates_use_the_selected_branch_and_survive_reloa #[async_trait::async_trait] impl DurableStreamConsumerJournal for RecordingConsumerJournal { async fn commit(&self) -> Result<(), String> { - self.oplog.commit(CommitLevel::Always).await; + self.oplog.commit(CommitLevel::Always).await.unwrap(); self.commits.fetch_add(1, Ordering::Relaxed); Ok(()) } @@ -5630,7 +5778,10 @@ async fn guest_authored_input_cancellation_targets_the_current_attachment_epoch( assert_eq!(intents[0].reason, StreamCancelReason::GuestDrop); for _ in 0..2050 { - oplog.add(OplogEntry::interrupted()).await; + oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } drop(guest.current_control_metadata().await.unwrap()); oplog.take_read_ranges(); @@ -5766,7 +5917,7 @@ async fn reverted_caller_reestablishes_foreign_reader_without_repeating_invocati consumer.invocation.callee_fingerprint = consumer.fingerprint; let owner = OwnedAgentId::new(consumer.environment_id, &consumer.agent_id); let oplog = Arc::new(TestOplog::default()); - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); let local = DurableStreamStore::load( oplog.clone(), consumer.environment_id, @@ -5805,7 +5956,7 @@ async fn reverted_caller_reestablishes_foreign_reader_without_repeating_invocati mapping: mapping.clone(), }), ] { - streams.append_record(None, record).await; + streams.append_record(None, record).await.unwrap(); } let reader_id = persist_mapping( &local, @@ -5827,7 +5978,8 @@ async fn reverted_caller_reestablishes_foreign_reader_without_repeating_invocati recursive_mappings: vec![], }), ) - .await; + .await + .unwrap(); let cut_index = oplog.current_oplog_index().await; streams .append_record( @@ -5841,7 +5993,8 @@ async fn reverted_caller_reestablishes_foreign_reader_without_repeating_invocati terminal: StreamConsumerTerminal::End(StreamEndResult::Ok), }), ) - .await; + .await + .unwrap(); if remote_state != "missing" { producer.prepare_attachment(old.clone(), 100).await.unwrap(); if remote_state != "prepared" { @@ -5880,7 +6033,8 @@ async fn reverted_caller_reestablishes_foreign_reader_without_repeating_invocati .into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); drop(streams); drop(local); let local = DurableStreamStore::load( @@ -6013,7 +6167,7 @@ async fn detached_continuation_activates_prepared_and_new_foreign_inputs() { read_requests: Mutex::default(), }); let oplog = Arc::new(TestOplog::default()); - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); let local = DurableStreamStore::load( oplog.clone(), consumer.environment_id, @@ -6082,7 +6236,7 @@ async fn detached_continuation_activates_prepared_and_new_foreign_inputs() { target.callee.agent_id = "forked-callee".into(); target.callee_fingerprint = AgentFingerprint::new(); } else { - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); } let owner = OwnedAgentId::new(target.callee_environment_id, &target.callee); let cut = DurableStreamStore::prepare_fork_cut( @@ -6106,9 +6260,10 @@ async fn detached_continuation_activates_prepared_and_new_foreign_inputs() { if let Some(region) = region { oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); } else { - oplog.add(marker).await; + oplog.add(marker).await.unwrap(); } drop(streams); drop(local); @@ -6312,7 +6467,7 @@ async fn closed_foreign_journal_replays_after_source_finalization_and_epoch_chan ), ] { assert!(record.has_supported_format()); - streams.append_record(None, record).await; + streams.append_record(None, record).await.unwrap(); } producer .prepare_attachment(attachment.clone(), 100) @@ -6349,7 +6504,8 @@ async fn closed_foreign_journal_replays_after_source_finalization_and_epoch_chan terminal: StreamConsumerTerminal::End(StreamEndResult::Ok), }), ) - .await; + .await + .unwrap(); streams.commit_consumer_journal().await.unwrap(); producer.finalize_attachment(attachment.clone(), golem_common::model::durable_stream::StreamAttachmentFinalizationReason::ConsumerFinalized, 101).await.unwrap(); let mut next_attachment = attachment; @@ -6477,7 +6633,8 @@ async fn mapped_source_unavailable_replays_as_permanent_without_cancelling_sourc }, }), ) - .await; + .await + .unwrap(); let reader_id = LocalStreamReaderId { introducing_oplog_index: streams.oplog.current_oplog_index().await, binding_slot: 0, @@ -6500,7 +6657,8 @@ async fn mapped_source_unavailable_replays_as_permanent_without_cancelling_sourc }, ), ) - .await; + .await + .unwrap(); let endpoint = streams .endpoint(handle, 0, SessionStreamRole::Input) @@ -6614,7 +6772,8 @@ async fn projected_durable_output_rematerializes_system_cancellation_as_permanen mapping: streams.binding(transport_stream_id).unwrap(), }), ) - .await; + .await + .unwrap(); } let reader_id = streams .current_control_metadata() @@ -6635,7 +6794,8 @@ async fn projected_durable_output_rematerializes_system_cancellation_as_permanen }, ), ) - .await; + .await + .unwrap(); let source_schema = SchemaGraph::anonymous(SchemaType::record(vec![NamedFieldType { name: "value".into(), @@ -7150,7 +7310,8 @@ async fn detach_resume_and_takeover_advance_authority_and_fence_old_epochs() { Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); producer .append_session_record( None, @@ -7751,7 +7912,8 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() stream_mappings: vec![StreamBindingRecord::foreign(&mapping)], }), ) - .await; + .await + .unwrap(); let pending_invocation_oplog_index = consumer_oplog .add(OplogEntry::pending_agent_invocation( consumer.invocation.idempotency_key.clone(), @@ -7760,7 +7922,8 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); streams .append_record( None, @@ -7775,7 +7938,8 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() }, ), ) - .await; + .await + .unwrap(); let streams = streams.with_attachment(1, attempt_id); remote_producer .prepare_attachment(attachment.clone(), 100) @@ -7864,7 +8028,10 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() assert_eq!(producer_oplog.current_oplog_index().await, producer_length); for _ in 0..2050 { - consumer_oplog.add(OplogEntry::interrupted()).await; + consumer_oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } assert_eq!( streams @@ -8106,7 +8273,8 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() mapping: partial_mapping.clone(), }), ) - .await; + .await + .unwrap(); let consumer_length = restarted.oplog.current_oplog_index().await; let producer_length = producer_oplog.current_oplog_index().await; assert!( @@ -8134,7 +8302,8 @@ async fn forwarded_topology_is_committed_before_visibility_and_replays_exactly() mapping: partial_mapping, }), ) - .await; + .await + .unwrap(); assert!(restarted.complete().await.is_err()); } @@ -8290,7 +8459,8 @@ async fn local_topology_cannot_activate_before_exact_session_attachment() { Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); producer .append_session_record( None, @@ -8711,7 +8881,8 @@ async fn output_catch_up_persists_a_missing_nested_transport_mapping_before_emit Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); let streams = StreamSession::open( producer.clone(), oplog.clone(), @@ -8735,7 +8906,8 @@ async fn output_catch_up_persists_a_missing_nested_transport_mapping_before_emit }, ), ) - .await; + .await + .unwrap(); let streams = streams.with_attachment(1, attempt_id); assert!( streams @@ -9223,7 +9395,8 @@ async fn resumed_foreign_parent_and_nested_output_cursors_use_the_accepted_epoch stream_mappings: Vec::new(), }), ) - .await; + .await + .unwrap(); let pending_invocation_oplog_index = consumer_oplog .add(OplogEntry::pending_agent_invocation( consumer.invocation.idempotency_key.clone(), @@ -9232,7 +9405,8 @@ async fn resumed_foreign_parent_and_nested_output_cursors_use_the_accepted_epoch Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); streams .append_record( None, @@ -9245,7 +9419,8 @@ async fn resumed_foreign_parent_and_nested_output_cursors_use_the_accepted_epoch pending_invocation_oplog_index, }), ) - .await; + .await + .unwrap(); let epoch1 = streams.with_attachment(1, start_attempt_id); let root_mapping = StreamSessionMappingRecord { transport_stream_id: 17, @@ -9495,7 +9670,10 @@ async fn session_control_metadata_pages_history_and_reads_only_raw_suffix_after_ [], ); for _ in 0..2050 { - oplog.add(OplogEntry::interrupted()).await; + oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } assert!( streams @@ -9531,11 +9709,12 @@ async fn session_control_metadata_pages_history_and_reads_only_raw_suffix_after_ }, ))), }) - .await; + .await + .unwrap(); // No commit: another local append must already be visible. assert_eq!(streams.caller_attempt_id().await.unwrap(), attempt_id); assert_eq!(oplog.take_read_ranges(), vec![(index, 1)]); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( streams.clone().caller_attempt_id().await.unwrap(), attempt_id @@ -9563,7 +9742,10 @@ async fn session_mapping_recovery_pages_once_and_shares_coverage_with_clones() { [], ); for _ in 0..2050 { - oplog.add(OplogEntry::interrupted()).await; + oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } streams.recover_session_mappings().await.unwrap(); assert_eq!( @@ -9572,7 +9754,10 @@ async fn session_mapping_recovery_pages_once_and_shares_coverage_with_clones() { ); streams.clone().recover_session_mappings().await.unwrap(); assert!(oplog.take_read_ranges().is_empty()); - let next = oplog.add(OplogEntry::interrupted()).await; + let next = oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); streams.recover_session_mappings().await.unwrap(); assert_eq!(oplog.take_read_ranges(), vec![(next, 1)]); } @@ -9726,9 +9911,10 @@ async fn finalization_after_retirement_requires_matching_committed_finished() { }, ))), }) - .await; + .await + .unwrap(); if committed { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); } producer.poison(); let journal = Arc::new(FinishedJournal { @@ -9751,7 +9937,7 @@ async fn finalization_after_retirement_requires_matching_committed_finished() { if hide_first || !committed { 2 } else { 1 } ); if !committed { - assert_eq!(oplog.commit(CommitLevel::Always).await.len(), 1); + assert_eq!(oplog.commit(CommitLevel::Always).await.unwrap().len(), 1); } } } @@ -9789,7 +9975,8 @@ async fn finished_in_raw_suffix_is_visible_and_cached() { }, ))), }) - .await; + .await + .unwrap(); assert_eq!( streams.persisted_finished().await.unwrap(), @@ -10290,3 +10477,67 @@ async fn forwarded_nested_stream_is_persisted_by_full_handle_without_re_registra [StreamRecordReference::Foreign(_)] )); } + +/// A session write that races the producer's retirement gets the retirement back as an error. It +/// used to panic as an "invalid record", and with `panic = "abort"` that took down every agent on +/// the executor, not just this session. +#[test] +async fn a_session_write_through_a_retired_producer_is_an_error_not_a_panic() { + let source = identity(); + let oplog = Arc::new(TestOplog::default()); + let producer = DurableStreamStore::load( + oplog.clone(), + source.environment_id, + source.agent_id.clone(), + source.fingerprint, + None, + ) + .await + .unwrap(); + let handle = producer + .register( + None, + registration( + &source, + StreamRegistrationCoordinate::Root { + invocation_id: source.invocation.clone(), + root_kind: StreamRootKind::MethodResult, + recursive_value_path: Vec::new(), + }, + StreamSourceKind::InvocationOutput, + ), + ) + .await + .unwrap() + .value; + let streams = StreamSession::new( + producer.clone(), + oplog.clone(), + StreamRegistrationInvocation::Remote(source.invocation.clone()), + [], + ); + + // Retirement poisons the producer without latching any fence on the oplog. + producer.poison(); + + let mapping = StreamSessionMappingRecord { + transport_stream_id: 1, + handle, + role: SessionStreamRole::Output, + }; + let record = StreamSessionRecord::Mapping( + golem_common::model::durable_stream::StreamSessionMappingUpdateRecord { + format_version: 1, + session_key: StreamRegistrationInvocation::Remote(source.invocation.clone()), + mapping: StreamBindingRecord::foreign(&mapping), + }, + ); + let error = streams + .append_record(None, record) + .await + .expect_err("a write through a retired producer must fail"); + assert!( + error.contains("RecoveryRequired"), + "the session must be told the producer needs recovery, got: {error}" + ); +} diff --git a/golem-worker-executor/src/durable_host/durable_stream/attachment.rs b/golem-worker-executor/src/durable_host/durable_stream/attachment.rs index 8acc8bd8fa..45872fdb5c 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/attachment.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/attachment.rs @@ -230,8 +230,8 @@ impl DurableStreamStore { entity_parent_start_index, OplogPayload::Inline(Box::new(record)), )) - .await; - self.commit(context).await; + .await?; + self.commit(context).await?; self.notify_session_records_changed(Some(context)); Ok(false) } @@ -336,8 +336,8 @@ impl DurableStreamStore { entity_parent_start_index, OplogPayload::Inline(Box::new(record)), )) - .await; - self.commit(context).await; + .await?; + self.commit(context).await?; match (was_reconcilable, is_reconcilable) { (false, true) => { self.reconcilable_attachment_count @@ -759,8 +759,8 @@ impl DurableStreamStore { records })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let mut terminal_events = Vec::new(); for (oplog_index, entry) in entries { match entry { @@ -877,8 +877,8 @@ impl DurableStreamStore { entity_parent_start_index, OplogPayload::Inline(Box::new(record.clone())), )) - .await; - self.commit(context).await; + .await?; + self.commit(context).await?; index.apply_session_references( entity_parent_start_index, &record, diff --git a/golem-worker-executor/src/durable_host/durable_stream/external_input.rs b/golem-worker-executor/src/durable_host/durable_stream/external_input.rs index 7336ac0a49..d7b89629a3 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/external_input.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/external_input.rs @@ -398,8 +398,8 @@ impl DurableStreamStore { records })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let AppliedWriteBatch { events, .. } = self .apply_committed_write_batch(&mut index, entries) diff --git a/golem-worker-executor/src/durable_host/durable_stream/fork.rs b/golem-worker-executor/src/durable_host/durable_stream/fork.rs index 4637953b28..ffd3967b34 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/fork.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/fork.rs @@ -483,7 +483,8 @@ mod tests { end: batch, }, )) - .await; + .await + .unwrap(); producer .write_items( None, @@ -521,7 +522,8 @@ mod tests { .into_inline_entry(); let (_, marker_index) = oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); let rebuilt = DurableStreamStore::load( oplog.clone(), owner.environment_id, @@ -676,7 +678,8 @@ mod tests { .into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); let rebuilt = DurableStreamStore::load( oplog.clone(), owner.environment_id, @@ -796,7 +799,7 @@ mod tests { .read_exact(OplogIndex::INITIAL, horizon.as_u64()) .await { - copied.add(entry).await; + copied.add(entry).await.unwrap(); } copied .add(OplogEntry::StreamSession { @@ -804,7 +807,8 @@ mod tests { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::ForkCut(cut))), }) - .await; + .await + .unwrap(); let rebuilt = DurableStreamStore::read_complete_index( copied.as_ref(), target.environment_id, @@ -955,7 +959,7 @@ mod tests { .read_exact(OplogIndex::INITIAL, cut_index.as_u64()) .await { - copied.add(entry).await; + copied.add(entry).await.unwrap(); } copied .add(OplogEntry::StreamSession { @@ -963,7 +967,8 @@ mod tests { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::ForkCut(cut))), }) - .await; + .await + .unwrap(); let forked = DurableStreamStore::load( copied.clone(), target.environment_id, @@ -1073,7 +1078,8 @@ mod tests { ) .into_inline_entry(), ) - .await; + .await + .unwrap(); let horizon = copied.current_oplog_index().await; let reverted = DurableStreamStore::prepare_fork_cut( copied.as_ref(), @@ -1097,7 +1103,8 @@ mod tests { .into_inline_entry(); let (_, marker_index) = copied .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); let rebuilt = DurableStreamStore::load( copied.clone(), target.environment_id, @@ -1178,7 +1185,8 @@ mod tests { .into_inline_entry(); copied .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); let historical = DurableStreamStore::prepare_fork_cut( copied.as_ref(), (&target, fingerprint), @@ -1215,7 +1223,8 @@ mod tests { ) .into_inline_entry(), ) - .await; + .await + .unwrap(); let horizon = copied.current_oplog_index().await; assert!(matches!( DurableStreamStore::prepare_fork_cut( @@ -1244,7 +1253,7 @@ mod tests { let identity = identity(); let owner = OwnedAgentId::new(identity.environment_id, &identity.agent_id); let oplog = TestOplog::default(); - let first_cut = oplog.add(OplogEntry::no_op(None)).await; + let first_cut = oplog.add(OplogEntry::no_op(None)).await.unwrap(); let entry = |record| DurableStreamOplogRecord::Session(None, Box::new(record)).into_inline_entry(); let attached = |epoch| { @@ -1266,8 +1275,9 @@ mod tests { .add(entry(StreamSessionRecord::Prepared(prepared( &identity.invocation, )))) - .await; - oplog.add(entry(attached(5))).await; + .await + .unwrap(); + oplog.add(entry(attached(5))).await.unwrap(); let cut = DurableStreamStore::prepare_fork_cut( &oplog, (&owner, identity.fingerprint), @@ -1284,13 +1294,15 @@ mod tests { let marker = entry(StreamSessionRecord::ForkCut(cut)); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); let second_cut = oplog .add(entry(StreamSessionRecord::Prepared(prepared( &identity.invocation, )))) - .await; - oplog.add(entry(attached(1))).await; + .await + .unwrap(); + oplog.add(entry(attached(1))).await.unwrap(); let cut = DurableStreamStore::prepare_fork_cut( &oplog, (&owner, identity.fingerprint), @@ -1313,7 +1325,7 @@ mod tests { let identity = identity(); let owner = OwnedAgentId::new(identity.environment_id, &identity.agent_id); let oplog = TestOplog::default(); - let cut_index = oplog.add(OplogEntry::no_op(None)).await; + let cut_index = oplog.add(OplogEntry::no_op(None)).await.unwrap(); let mut cut = DurableStreamStore::prepare_fork_cut( &oplog, (&owner, identity.fingerprint), @@ -1348,7 +1360,7 @@ mod tests { let identity = identity(); let owner = OwnedAgentId::new(identity.environment_id, &identity.agent_id); let oplog = Arc::new(TestOplog::default()); - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); let descriptor = prepared(&identity.invocation); let requests: Vec<_> = (0..2) .map(|path| { @@ -1443,7 +1455,8 @@ mod tests { .into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); } } @@ -1464,11 +1477,12 @@ mod tests { let fingerprint = AgentFingerprint(Uuid::from_u128(422)); let oplog = TestOplog::default(); for _ in 0..3 { - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); } let horizon = oplog .add(OplogEntry::revert(OplogRegion::from_range(2..=3))) - .await; + .await + .unwrap(); let old_cut = OplogIndex::from_u64(2); let cut = DurableStreamStore::prepare_fork_cut( &oplog, @@ -1503,7 +1517,8 @@ mod tests { ) .into_inline_entry(), ) - .await; + .await + .unwrap(); let cut = DurableStreamStore::prepare_fork_cut( &oplog, (&target, fingerprint), @@ -2335,7 +2350,7 @@ mod tests { entity_parent_start_index: None, }, }; - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let rebuilt = DurableStreamStore::read_complete_index( &oplog, diff --git a/golem-worker-executor/src/durable_host/durable_stream/items.rs b/golem-worker-executor/src/durable_host/durable_stream/items.rs index 2da61433bd..14d3a2508e 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/items.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/items.rs @@ -1120,8 +1120,8 @@ impl DurableStreamStore { records })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let AppliedWriteBatch { events: item_events, diff --git a/golem-worker-executor/src/durable_host/durable_stream/metadata.rs b/golem-worker-executor/src/durable_host/durable_stream/metadata.rs index 7ace021569..c6b12b1ff2 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/metadata.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/metadata.rs @@ -2135,6 +2135,7 @@ mod tests { timestamp: Timestamp::now_utc(), }, ))), + None, ) .await; Self { @@ -2151,7 +2152,10 @@ mod tests { let commit: DurableStreamCommit = Arc::new(move |published| { let oplog = oplog.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); if let Some(published) = published { let _ = published.send(()); } @@ -2204,7 +2208,10 @@ mod tests { async fn persist(&self) { let owner = OwnedAgentId::new(self.identity.environment_id, &self.identity.agent_id); - self.oplog.commit(CommitLevel::Always).await; + self.oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); self.service .lookup_durable_stream_producer_metadata( &owner, @@ -2757,7 +2764,11 @@ mod tests { offsets.push(outcome.value[0]); } for _ in 0..2100 { - fixture.oplog.add(OplogEntry::interrupted()).await; + fixture + .oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } fixture.persist().await; drop(producer); @@ -3092,7 +3103,11 @@ mod tests { .unwrap() .value; for _ in 0..1021 { - fixture.oplog.add(OplogEntry::interrupted()).await; + fixture + .oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } assert_eq!(fixture.oplog.current_oplog_index().await.as_u64(), 1023); let nested = registration( @@ -3117,7 +3132,11 @@ mod tests { .await .unwrap(); let nested_handles = producer.nested_handles(handle.stream_id, 0).await.unwrap(); - fixture.oplog.commit(CommitLevel::Always).await; + fixture + .oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); MultiLayerOplog::try_archive_blocking(&fixture.oplog) .await .expect("archive layer"); @@ -3401,7 +3420,7 @@ mod tests { vec![40, 41], ), ] { - fixture.oplog.add(entry).await; + fixture.oplog.add(entry).await.unwrap(); } fixture.persist().await; let owner = OwnedAgentId::new(fixture.identity.environment_id, &fixture.identity.agent_id); @@ -3574,7 +3593,8 @@ mod tests { ), ))), }) - .await; + .await + .unwrap(); let producer = fixture.producer().await; let original = producer .register(None, fixture.registration(0)) @@ -3607,7 +3627,8 @@ mod tests { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::ForkCut(cut))), }) - .await; + .await + .unwrap(); fixture.persist().await; let cold = fixture.producer().await; let events = cold.read_segment(&continuation, None, None).await.unwrap(); @@ -3640,7 +3661,8 @@ mod tests { let oplog = Arc::new(TestOplog::default()); oplog .add(fixture.oplog.read(OplogIndex::INITIAL).await) - .await; + .await + .unwrap(); let producer = DurableStreamStore::load( oplog.clone(), source.environment_id, @@ -3756,7 +3778,7 @@ mod tests { ) .await { - copied.add(entry).await; + copied.add(entry).await.unwrap(); } let mut cut = fork(Some(root_local_id), copied.current_oplog_index().await); cut.creation_fingerprint = target.fingerprint; @@ -3767,7 +3789,8 @@ mod tests { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(StreamSessionRecord::ForkCut(cut))), }) - .await; + .await + .unwrap(); let oplog = copied; for (_, entry) in oplog .read_exact( @@ -3776,7 +3799,7 @@ mod tests { ) .await { - fixture.oplog.add(entry).await; + fixture.oplog.add(entry).await.unwrap(); } fixture.persist().await; let raw_producer = DurableStreamStore::load( @@ -3930,7 +3953,8 @@ mod tests { entity_parent_start_index: None, record: OplogPayload::Inline(Box::new(record)), }) - .await; + .await + .unwrap(); if prepared { assert!( fixture @@ -4128,7 +4152,8 @@ mod tests { }), )), }) - .await; + .await + .unwrap(); fixture.persist().await; assert_eq!( fixture @@ -4340,8 +4365,13 @@ mod tests { fixture .oplog .add(OplogEntry::stream_session(None, record)) - .await; - fixture.oplog.commit(CommitLevel::Always).await; + .await + .expect("oplog write"); + fixture + .oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let horizon = fixture.oplog.current_oplog_index().await; let (started, release) = fixture.blobs.pause_next_read(); let mut query = Box::pin(producer.persisted_control_metadata(&session)); diff --git a/golem-worker-executor/src/durable_host/durable_stream/mod.rs b/golem-worker-executor/src/durable_host/durable_stream/mod.rs index 62d02d7620..e2f9334dd1 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/mod.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/mod.rs @@ -54,7 +54,8 @@ use crate::durable_host::stream_bus::{ }; use crate::services::activity::{ActivityGate, spawn_with_activity}; use crate::services::oplog::{ - CommitLevel, DurableStreamOplogRecord, Oplog, OplogOps, OplogService, OplogServiceOps, + CommitLevel, DurableStreamOplogRecord, Oplog, OplogError, OplogFence, OplogOps, OplogService, + OplogServiceOps, }; use crate::services::rpc::{DurableStreamReadError, Rpc}; use crate::services::worker::WorkerService; @@ -91,6 +92,7 @@ use golem_common::model::oplog::payload::OplogPayload; use golem_schema::schema::{ SchemaFingerprintV1, SchemaGraph, SchemaType, SchemaValue, TypedSchemaValue, }; +use golem_service_base::error::worker_executor::WorkerExecutorError; use golem_service_base::model::auth::AuthCtx; use std::collections::{BTreeMap, HashMap, HashSet, VecDeque}; use std::future::Future; @@ -421,11 +423,24 @@ pub enum StreamStoreError { ConsumerJournalAdvanced, DeletionBlocked(Vec), CorruptHistory(String), + Fenced(OplogFence), Oplog(String), RecoveryRequired, LiveBus(DurableLiveStreamBusError), } +/// A refused write keeps its type as `Fenced` instead of joining `Oplog` as text, so the +/// boundaries can report it as `OplogFenced` - a caller reroutes on that - rather than as a +/// failure of the request. +impl From for StreamStoreError { + fn from(error: OplogError) -> Self { + match error { + OplogError::Fenced(fence) => Self::Fenced(fence), + error @ OplogError::Storage(_) => Self::Oplog(error.to_string()), + } + } +} + impl std::fmt::Display for StreamStoreError { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!(formatter, "{self:?}") @@ -441,6 +456,18 @@ impl From for String { } impl StreamStoreError { + /// Converts at a boundary that reports `WorkerExecutorError`: a fence keeps its type, and every + /// other error is rendered through `otherwise`, which says how that boundary classifies it. + pub(crate) fn into_worker_executor_error( + self, + otherwise: impl FnOnce(String) -> WorkerExecutorError, + ) -> WorkerExecutorError { + match self { + Self::Fenced(fence) => WorkerExecutorError::from(OplogError::Fenced(fence)), + error => otherwise(error.to_string()), + } + } + /// Formats the dependent attachment identities that currently block deletion. pub fn deletion_blocked_evidence(&self) -> Option { let Self::DeletionBlocked(dependents) = self else { @@ -708,7 +735,10 @@ impl DurableStreamStore { let commit: DurableStreamCommit = Arc::new(move |committed| { let oplog = commit_oplog.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); if let Some(committed) = committed { let _ = committed.send(()); } diff --git a/golem-worker-executor/src/durable_host/durable_stream/mutation.rs b/golem-worker-executor/src/durable_host/durable_stream/mutation.rs index 8c5bf04fbe..a835befcbf 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/mutation.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/mutation.rs @@ -224,7 +224,12 @@ impl DurableStreamStore { /// Rejects work after the resident producer has been poisoned or retired. pub(crate) fn ensure_healthy(&self) -> Result<(), StreamStoreError> { if self.poisoned.load(Ordering::Acquire) { - Err(StreamStoreError::RecoveryRequired) + // A fenced write poisons the store, and every later write would be refused by the + // same latch: report the fence so a caller reroutes instead of recovering locally. + match self.oplog.fence() { + Some(fence) => Err(StreamStoreError::Fenced(fence)), + None => Err(StreamStoreError::RecoveryRequired), + } } else { Ok(()) } @@ -567,7 +572,10 @@ impl DurableStreamStore { .expect("durable stream producer-owned write terminated") } - pub(super) async fn commit(&self, context: &StreamWriteContext) { + pub(super) async fn commit( + &self, + context: &StreamWriteContext, + ) -> Result<(), StreamStoreError> { context.assert_owner(self); context.begin_durable_effect(); let scope = &context.scope; @@ -585,17 +593,32 @@ impl DurableStreamStore { .lock() .expect("commit tail list lock poisoned") .push(task); - receipt - .await - .expect("durable stream commit failed before durability receipt"); + let received = receipt.await; + // A fenced commit drops the receipt rather than signalling it, so the latch is read + // before a missing receipt is treated as a failed callback. + self.committed_unless_fenced()?; + received.expect("durable stream commit failed before durability receipt"); + Ok(()) } pub(super) async fn commit_notifying( &self, context: &StreamWriteContext, committed: oneshot::Sender<()>, - ) { - self.commit(context).await; + ) -> Result<(), StreamStoreError> { + self.commit(context).await?; let _ = committed.send(()); + Ok(()) + } + + /// The worker's commit swallows a refusal: it only spawns the give-up. The refused append + /// has latched the fence before the commit resolves, so the latch is what tells a persisted + /// write from one that must not be indexed, retained or published. A below-threshold add + /// answers `Ok` on a latched oplog, so no earlier result can stand in for this check. + fn committed_unless_fenced(&self) -> Result<(), StreamStoreError> { + match self.oplog.fence() { + Some(fence) => Err(StreamStoreError::Fenced(fence)), + None => Ok(()), + } } } diff --git a/golem-worker-executor/src/durable_host/durable_stream/registration.rs b/golem-worker-executor/src/durable_host/durable_stream/registration.rs index ca3555eb46..1edad11d5a 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/registration.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/registration.rs @@ -159,8 +159,8 @@ impl DurableStreamStore { )] })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let (oplog_index, entry) = entries .pop() .expect("registration batch returned no oplog entry"); @@ -613,8 +613,8 @@ impl DurableStreamStore { result })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let registered_count = registered_handles.len(); let mut handles = registered_handles; diff --git a/golem-worker-executor/src/durable_host/durable_stream/session.rs b/golem-worker-executor/src/durable_host/durable_stream/session.rs index 94b869bd6a..58afe6f341 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/session.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/session.rs @@ -358,14 +358,14 @@ impl DurableStreamStore { .collect() })) .await - .map_err(StreamStoreError::Oplog)?; + .map_err(StreamStoreError::from)?; for (position, key) in result_keys { staged .invocation_results .entry(key) .or_insert(entries[position].0); } - self.commit(context).await; + self.commit(context).await?; *index = staged; drop(index); self.notify_session_records_changed(Some(context)); @@ -417,8 +417,8 @@ impl DurableStreamStore { )] })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; self.notify_session_records_changed(Some(context)); Ok(()) } @@ -699,7 +699,7 @@ impl DurableStreamStore { result })) .await - .map_err(StreamStoreError::Oplog)?; + .map_err(StreamStoreError::from)?; let mut prepared = None; let mut topologies = Vec::new(); @@ -789,7 +789,7 @@ impl DurableStreamStore { )?; } - self.commit_notifying(context, committed).await; + self.commit_notifying(context, committed).await?; *index = updated_index; self.buses .write() @@ -991,8 +991,8 @@ impl DurableStreamStore { records })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let mut terminal_events = Vec::new(); for (oplog_index, entry) in entries { diff --git a/golem-worker-executor/src/durable_host/durable_stream/terminals.rs b/golem-worker-executor/src/durable_host/durable_stream/terminals.rs index 3e8e65c4c1..8ab4e6ac51 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/terminals.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/terminals.rs @@ -78,8 +78,8 @@ impl DurableStreamStore { )] })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let (oplog_index, entry) = entries .pop() .expect("resource exhaustion terminal batch returned no oplog entry"); @@ -235,8 +235,8 @@ impl DurableStreamStore { )] })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let (oplog_index, entry) = entries .pop() .expect("stream end batch returned no oplog entry"); @@ -366,8 +366,8 @@ impl DurableStreamStore { )] })) .await - .map_err(StreamStoreError::Oplog)?; - self.commit(context).await; + .map_err(StreamStoreError::from)?; + self.commit(context).await?; let (oplog_index, entry) = entries .pop() .expect("stream cancellation batch returned no oplog entry"); diff --git a/golem-worker-executor/src/durable_host/durable_stream/tests.rs b/golem-worker-executor/src/durable_host/durable_stream/tests.rs index 4d1c7ff787..e9879374ef 100644 --- a/golem-worker-executor/src/durable_host/durable_stream/tests.rs +++ b/golem-worker-executor/src/durable_host/durable_stream/tests.rs @@ -251,6 +251,11 @@ struct TestOplogState { entries: BTreeMap, committed: OplogIndex, commit_count: u64, + /// When set, `add` answers the way a primary oplog does when the threshold commit behind + /// the add is refused by the storage. + refused_adds: Option, + /// The fence a primary oplog latches when the storage refuses one of its commits. + fence: Option, } #[derive(Default)] @@ -271,7 +276,7 @@ impl TestOplog { std::mem::take(&mut *self.read_ranges.lock().unwrap()) } - fn committed_length(&self) -> u64 { + pub(crate) fn committed_length(&self) -> u64 { self.state.lock().unwrap().committed.as_u64() } @@ -288,18 +293,32 @@ impl TestOplog { .cloned() .collect() } + + pub(crate) fn refuse_adds(&self, fence: crate::services::oplog::OplogFence) { + self.state.lock().unwrap().refused_adds = Some(fence); + } + + pub(crate) fn latch_fence(&self, fence: crate::services::oplog::OplogFence) { + self.state.lock().unwrap().fence = Some(fence); + } } #[async_trait] impl Oplog for TestOplog { - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + entry: OplogEntry, + ) -> Result { let mut state = self.state.lock().unwrap(); + if let Some(fence) = &state.refused_adds { + return Err(crate::services::oplog::OplogError::Fenced(fence.clone())); + } let index = state .entries .last_key_value() .map_or(OplogIndex::INITIAL, |(index, _)| index.next()); state.entries.insert(index, entry); - index + Ok(index) } fn enqueue_add(&self, entry: OplogEntry) -> OplogAddReceipt { @@ -309,7 +328,11 @@ impl Oplog for TestOplog { .last_key_value() .map_or(OplogIndex::INITIAL, |(index, _)| index.next()); state.entries.insert(index, entry); - Box::pin(async move { index }) + Box::pin(async move { Ok(index) }) + } + + fn fence(&self) -> Option { + self.state.lock().unwrap().fence.clone() } async fn drop_prefix(&self, last_dropped_id: OplogIndex) -> u64 { @@ -319,20 +342,23 @@ impl Oplog for TestOplog { (before - state.entries.len()) as u64 } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { let mut state = self.state.lock().unwrap(); let committed = state .entries .iter() .filter(|(index, _)| **index > state.committed) .map(|(index, entry)| (*index, entry.clone())) - .collect(); + .collect::>(); state.committed = state .entries .last_key_value() .map_or(state.committed, |(index, _)| *index); state.commit_count += 1; - committed + Ok(committed) } async fn current_oplog_index(&self) -> OplogIndex { @@ -470,9 +496,9 @@ impl Oplog for TestOplog { &self, serialized_request: Vec, build_start: Box Result + Send>, - ) -> Result { + ) -> Result { let entry = build_start(RawOplogPayload::SerializedInline(serialized_request))?; - let index = self.add(entry.clone()).await; + let index = self.add(entry.clone()).await?; Ok(OrderedOplogStart { index, entry, @@ -483,7 +509,7 @@ impl Oplog for TestOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let mut state = self.state.lock().unwrap(); let index = state .entries @@ -503,10 +529,10 @@ impl Oplog for TestOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { - let first = self.add(start).await; - let second = self.add(make_second(first)).await; - (first, second) + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { + let first = self.add(start).await?; + let second = self.add(make_second(first)).await?; + Ok((first, second)) } } @@ -514,7 +540,7 @@ impl Oplog for TestOplog { async fn test_oplog_read_exact_includes_uncommitted_entries() { let oplog = TestOplog::default(); let entry = OplogEntry::interrupted(); - let index = oplog.add(entry.clone()).await; + let index = oplog.add(entry.clone()).await.unwrap(); let entries = oplog.read_exact(index, 1).await; @@ -524,7 +550,7 @@ async fn test_oplog_read_exact_includes_uncommitted_entries() { #[test] async fn test_oplog_read_exact_rejects_incomplete_range() { let oplog = TestOplog::default(); - let index = oplog.add(OplogEntry::interrupted()).await; + let index = oplog.add(OplogEntry::interrupted()).await.unwrap(); let result = std::panic::AssertUnwindSafe(oplog.read_exact(index, 2)) .catch_unwind() @@ -861,7 +887,7 @@ async fn session_finish_holds_its_lock_and_reserves_terminal_batch_bytes() { let reached = reached.clone(); let release = release.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); reached.notify_one(); release.notified().await; if let Some(published) = published { @@ -1656,7 +1682,10 @@ async fn delayed_stream_records_retain_registration_entity_attribution() { request.entity_parent_start_index = entity_parent_start_index; let handle = live.register(None, request).await.unwrap().value; - oplog.add(OplogEntry::no_op(None)).await; + oplog + .add(OplogEntry::no_op(None)) + .await + .expect("oplog write"); live.write_items( None, handle.stream_id, @@ -1718,7 +1747,10 @@ async fn item_payloads_are_loaded_only_for_the_requested_batch() { .unwrap(); } for _ in 0..2050 { - oplog.add(OplogEntry::interrupted()).await; + oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } assert!( producer.index.lock().await.streams[&handle.stream_id] @@ -1949,7 +1981,10 @@ async fn cursor_validation_point_reads_without_historical_event_cache() { .unwrap() .value; for _ in 0..2050 { - oplog.add(OplogEntry::interrupted()).await; + oplog + .add(OplogEntry::interrupted()) + .await + .expect("oplog write"); } producer .end(None, handle.stream_id, 3, StreamEndResult::Ok) @@ -3509,7 +3544,7 @@ async fn session_record_commit_folds_a_pending_invocation_added_immediately_befo let oplog = oplog_for_commit.clone(); let batches = batches_for_commit.clone(); Box::pin(async move { - let committed_entries = oplog.commit(CommitLevel::Always).await; + let committed_entries = oplog.commit(CommitLevel::Always).await.unwrap(); batches .lock() .unwrap() @@ -3565,7 +3600,8 @@ async fn session_record_commit_folds_a_pending_invocation_added_immediately_befo Vec::new(), Vec::new(), )) - .await; + .await + .unwrap(); producer .append_session_record( None, @@ -4101,7 +4137,7 @@ async fn failed_commit_callbacks_fence_cached_reads_and_recover_committed_items( let oplog = oplog.clone(); let fail = fail.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert!( fail_after_receipt || !fail.load(Ordering::Acquire), "injected failure before durability receipt" @@ -4254,7 +4290,7 @@ async fn handle_read_hydrates_cancellation_committed_before_request_abort() { let block_commit = block_commit.clone(); let committed = committed.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(published) = published { let _ = published.send(()); } @@ -4336,7 +4372,7 @@ async fn external_append_retry_after_commit_cancellation_is_duplicate() { let block_commit = block_commit.clone(); let committed = committed.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(published) = published { let _ = published.send(()); } @@ -4438,7 +4474,7 @@ async fn external_append_survives_caller_abort_before_commit_receipt() { let blocked = blocked.clone(); let release = release.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if block_receipt.swap(false, Ordering::SeqCst) { blocked.notify_one(); release.notified().await; @@ -5436,7 +5472,7 @@ async fn prepared_input_registration_batch_recovers_without_duplicate_registrati let oplog = oplog.clone(); let commit_reached = commit_reached.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(committed) = committed { let _ = committed.send(()); } @@ -5611,7 +5647,7 @@ async fn prepared_foreign_inputs_recover_the_winning_invocation_and_topology() { let oplog = oplog.clone(); let commit_reached = commit_reached.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let _ = committed.unwrap().send(()); commit_reached.wait().await; std::future::pending::<()>().await; @@ -6299,7 +6335,7 @@ async fn malformed_history_is_rejected_while_rebuilding_the_index() { })) .await .unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); drop(producer); assert!(matches!( @@ -6457,7 +6493,7 @@ async fn history_rebuild_rejects_duplicate_nested_stream_ownership() { })) .await .unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); drop(producer); assert!( @@ -6512,7 +6548,7 @@ async fn history_rebuild_rejects_nested_registration_without_enclosing_item() { })) .await .unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); drop(producer); assert!(matches!( @@ -6964,6 +7000,184 @@ async fn session_control_batch_validates_before_appending_any_record() { } } +pub(crate) fn test_fence() -> crate::services::oplog::OplogFence { + crate::services::oplog::OplogFence { + agent_id: identity().agent_id, + expected_epoch: golem_common::model::ShardEpoch(3), + actual_epoch: Some(golem_common::model::ShardEpoch(4)), + writer_conflict: false, + } +} + +/// A fence has to reach the worker executor's boundaries as a fence: flattened into +/// `InvalidRequest` it is rejected as a validation failure, and the caller never reroutes. +#[test] +fn a_fenced_oplog_error_keeps_its_type_through_the_producer() { + use golem_service_base::error::worker_executor::WorkerExecutorError; + + let fenced = StreamStoreError::from(crate::services::oplog::OplogError::Fenced(test_fence())); + assert!(matches!(fenced, StreamStoreError::Fenced(_))); + assert!(matches!( + fenced.into_worker_executor_error(WorkerExecutorError::invalid_request), + WorkerExecutorError::OplogFenced { .. } + )); + + let storage = StreamStoreError::from(crate::services::oplog::OplogError::Storage( + "connection reset".to_string(), + )); + assert!(matches!(storage, StreamStoreError::Oplog(_))); + assert!(matches!( + storage.into_worker_executor_error(WorkerExecutorError::invalid_request), + WorkerExecutorError::InvalidRequest { .. } + )); +} + +#[test] +async fn a_fenced_session_record_append_is_reported_as_fenced() { + let identity = identity(); + let oplog = Arc::new(TestOplog::default()); + let producer = producer(oplog.clone(), &identity, None).await; + oplog.refuse_adds(test_fence()); + + let result = producer + .append_session_record( + None, + StreamSessionRecord::ConsumerDeleting(StreamConsumerDeletingRecord { + format_version: DURABLE_STREAM_FORMAT_VERSION, + consumer_environment_id: identity.environment_id, + consumer: identity.agent_id, + consumer_fingerprint: identity.fingerprint, + deleting_at_millis: 100, + }), + ) + .await; + + assert!( + matches!(result, Err(StreamStoreError::Fenced(_))), + "a refused session record append must stay a fence, got {result:?}" + ); +} + +/// A producer committing the way the worker does: a commit refused by a fence is swallowed, +/// and only the oplog's latch records it. +async fn producer_swallowing_fenced_commits( + oplog: Arc, + identity: &TestIdentity, +) -> Arc { + let commit_oplog = oplog.clone(); + let commit: DurableStreamCommit = Arc::new(move |committed| { + let oplog = commit_oplog.clone(); + Box::pin(async move { + if oplog.fence().is_some() { + return; + } + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); + if let Some(committed) = committed { + let _ = committed.send(()); + } + }) + }); + DurableStreamStore::load_with_commit( + oplog, + identity.environment_id, + identity.agent_id.clone(), + identity.fingerprint, + None, + commit, + ) + .await + .unwrap() +} + +/// Items whose commit was refused never reached the oplog: a live reader handed them would +/// journal offsets the shard's new owner replays without. +#[test] +async fn a_fenced_commit_neither_publishes_nor_indexes_stream_items() { + let identity = identity(); + let oplog = Arc::new(TestOplog::default()); + let producer = producer_swallowing_fenced_commits(oplog.clone(), &identity).await; + let handle = producer + .register(None, root_registration(&identity)) + .await + .unwrap() + .value; + let bus = producer.bus(handle.stream_id).unwrap(); + let mut subscription = bus.subscribe().await.unwrap(); + let committed_before = oplog.committed_length(); + oplog.latch_fence(test_fence()); + + let result = producer + .write_items( + None, + handle.stream_id, + 0, + StreamItemsPayload::PackedU8(vec![7]), + ) + .await; + + assert!( + matches!(result, Err(StreamStoreError::Fenced(_))), + "a write whose commit was refused must report the fence, got {result:?}" + ); + // The refused write poisons the store, which retires its buses: the reader may observe + // that retirement, but never an event. + let received = tokio::time::timeout(Duration::from_millis(200), subscription.recv()).await; + assert!( + !matches!(received, Ok(Ok(_))), + "nothing may be published for a write whose commit was refused" + ); + assert_eq!(oplog.committed_length(), committed_before); + assert_eq!( + producer.index.lock().await.streams[&handle.stream_id].next_sequence, + 0 + ); + // The latch outlives the refused write: a later write reports the fence as well, not a + // local recovery. + let retried = producer + .write_items( + None, + handle.stream_id, + 0, + StreamItemsPayload::PackedU8(vec![7]), + ) + .await; + assert!( + matches!(retried, Err(StreamStoreError::Fenced(_))), + "a write after a fence must report the fence, got {retried:?}" + ); +} + +#[test] +async fn a_fenced_commit_does_not_record_an_attachment_in_the_index() { + let identity = identity(); + let oplog = Arc::new(TestOplog::default()); + let producer = producer_swallowing_fenced_commits(oplog.clone(), &identity).await; + let handle = producer + .register(None, root_registration(&identity)) + .await + .unwrap() + .value; + let key = attachment_key(&identity, handle.stream_id); + oplog.latch_fence(test_fence()); + + let result = producer.prepare_attachment(key.clone(), 100).await; + + assert!( + matches!(result, Err(StreamStoreError::Fenced(_))), + "an attachment whose commit was refused must report the fence, got {result:?}" + ); + assert!( + producer + .inspect_attachments() + .await + .iter() + .all(|view| view.key != key) + ); +} + #[test] async fn malformed_session_record_is_rejected_at_the_write_boundary() { let identity = identity(); @@ -7181,7 +7395,7 @@ async fn restart_recovers_registration_committed_before_caller_observation() { let oplog = oplog.clone(); let commit_reached = commit_reached.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(committed) = committed { let _ = committed.send(()); } @@ -7332,7 +7546,7 @@ async fn session_notification_waits_for_status_fold_after_caller_cancellation() let live = live.clone(); async move { live.run_owned(None, 0, move |owner, context| async move { - owner.commit(&context).await; + owner.commit(&context).await.unwrap(); context.finish_durable_effect(); owner.notify_session_records_changed(Some(&context)); requested.send(()).unwrap(); @@ -7372,7 +7586,7 @@ async fn durable_activity_waits_for_callback_tails_but_not_abandoned_fanout() { let committed = committed.clone(); let release = release.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); if let Some(receipt) = receipt { let _ = receipt.send(()); } diff --git a/golem-worker-executor/src/durable_host/entity.rs b/golem-worker-executor/src/durable_host/entity.rs index 5935bab6d6..4389cd07a4 100644 --- a/golem-worker-executor/src/durable_host/entity.rs +++ b/golem-worker-executor/src/durable_host/entity.rs @@ -521,7 +521,7 @@ impl EntityInvocationDurability { Some(handle.start_index()), region.clone(), )) - .await; + .await?; } replay.register_entity_atomic_rollback(regions).await?; worker.reattach_worker_status().await; diff --git a/golem-worker-executor/src/durable_host/golem/retry_api.rs b/golem-worker-executor/src/durable_host/golem/retry_api.rs index c2a66d1ec0..286defca72 100644 --- a/golem-worker-executor/src/durable_host/golem/retry_api.rs +++ b/golem-worker-executor/src/durable_host/golem/retry_api.rs @@ -147,7 +147,7 @@ impl Host for DurableWorkerCtx { self.entity_parent_start_index(), Box::new(named_policy.clone()), )) - .await; + .await?; } self.state.apply_set_retry_policy(named_policy); @@ -176,7 +176,7 @@ impl Host for DurableWorkerCtx { self.entity_parent_start_index(), name.clone(), )) - .await; + .await?; } self.state.apply_remove_retry_policy(&name); diff --git a/golem-worker-executor/src/durable_host/golem/v1x.rs b/golem-worker-executor/src/durable_host/golem/v1x.rs index f4eb6b7af1..c816bbe1af 100644 --- a/golem-worker-executor/src/durable_host/golem/v1x.rs +++ b/golem-worker-executor/src/durable_host/golem/v1x.rs @@ -36,7 +36,7 @@ use crate::preview2::golem_api_1_x::host::{ use crate::preview2::golem_api_1_x::oplog::{ Host as OplogHost, HostGetOplog, HostSearchOplog, OplogReadError, SearchOplog, }; -use crate::services::oplog::CommitLevel; +use crate::services::oplog::{CommitLevel, OplogError}; use crate::services::promise::{PromiseHandle, PromiseService}; use crate::services::worker_proxy::WorkerProxyError; use crate::services::{HasOplogService, HasWorker}; @@ -668,6 +668,7 @@ impl Host for DurableWorkerCtx { .oplog .add(OplogEntry::no_op(self.entity_parent_start_index())) .await + .map_err(|error| anyhow!(WorkerExecutorError::from(error)))? { OplogIndex::NONE => self.state.current_oplog_index().await, index => index, @@ -740,7 +741,7 @@ impl Host for DurableWorkerCtx { self.public_state .worker() .add_and_commit_oplog(OplogEntry::jump(self.entity_parent_start_index(), jump)) - .await; + .await?; debug!("Interrupting live execution for jumping from {jump_source} to {jump_target}",); Err(InterruptKind::Jump.into()) @@ -758,7 +759,17 @@ impl Host for DurableWorkerCtx { debug!("Worker committing oplog to {replicas} replicas"); loop { // Applying a timeout to make sure the worker remains interruptible - if self.state.oplog.wait_for_replicas(replicas, timeout).await { + let committed = self.state.oplog.wait_for_replicas(replicas, timeout).await; + // The shard has a new owner, so nothing was committed and nothing can be. The + // fence surfaces as `ShardLost`, which gives the agent up without writing, instead + // of acknowledging a commit that did not happen or retrying one that never will: + // `check_interrupt` below has no interrupt to report for a latched fence. + if let Some(fence) = self.state.oplog.fence() { + return Err(anyhow!(WorkerExecutorError::from(OplogError::Fenced( + fence + )))); + } + if committed { debug!("Worker committed oplog to {replicas} replicas"); return Ok(()); } else { @@ -848,7 +859,7 @@ impl Host for DurableWorkerCtx { self.public_state .worker() .add_and_commit_oplog(OplogEntry::jump(None, deleted_region)) - .await; + .await?; // TODO: this recomputation should not be necessary. self.public_state.worker().reattach_worker_status().await; @@ -880,6 +891,7 @@ impl Host for DurableWorkerCtx { self.entity_parent_start_index(), )) .await + .map_err(|error| anyhow!(WorkerExecutorError::from(error)))? { OplogIndex::NONE => self.state.current_oplog_index().await, index => index, @@ -944,7 +956,8 @@ impl Host for DurableWorkerCtx { self.entity_parent_start_index(), begin_index, )) - .await; + .await + .map_err(|error| anyhow!(WorkerExecutorError::from(error)))?; } else { let (_, _) = get_oplog_entry!(self.state.replay_state, OplogEntry::EndAtomicRegion)?; } @@ -1663,7 +1676,7 @@ impl Host for DurableWorkerCtx { self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; let created_by = self.created_by(); let fork_result = loop { diff --git a/golem-worker-executor/src/durable_host/http/types.rs b/golem-worker-executor/src/durable_host/http/types.rs index 7ce9ee8a9e..cd6fcdebd4 100644 --- a/golem-worker-executor/src/durable_host/http/types.rs +++ b/golem-worker-executor/src/durable_host/http/types.rs @@ -21,7 +21,7 @@ use crate::durable_host::http::inline_retry::{ use crate::durable_host::http::{continue_http_request, end_http_request}; use crate::durable_host::{DurabilityHost, DurableWorkerCtx}; use crate::services::HasWorker; -use crate::services::oplog::{CommitLevel, OplogOps}; +use crate::services::oplog::{CommitLevel, OplogError, OplogOps}; use crate::workerctx::WorkerCtx; use golem_common::model::NamedRetryPolicy; use golem_common::model::oplog::host_functions::{ @@ -1161,7 +1161,7 @@ impl HostFutureIncomingResponse for DurableWorkerCtx { _ => None, }; } - persist_http_response(self, request, &serializable_response, begin_index).await; + persist_http_response(self, request, &serializable_response, begin_index).await?; if !is_pending && let Ok(Some(Ok(Ok(resource)))) = &response { let incoming_response_handle = resource.rep(); @@ -1422,7 +1422,7 @@ async fn persist_http_response( request: golem_common::model::oplog::HostRequestHttpRequest, serializable_response: &SerializableHttpResponse, begin_index: golem_common::model::oplog::OplogIndex, -) { +) -> Result<(), WorkerExecutorError> { if !ctx.state.durability_is_suppressed() { ctx.state .oplog @@ -1436,12 +1436,16 @@ async fn persist_http_response( Some(begin_index), ) .await - .unwrap_or_else(|err| panic!("failed to serialize http response: {err}")); + .map_err(|err| match err { + OplogError::Fenced(_) => err, + err => panic!("failed to serialize http response: {err}"), + })?; ctx.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; } + Ok(()) } /// Typed HTTP failure for retry classification, preserving the original `ErrorCode` diff --git a/golem-worker-executor/src/durable_host/logging/policy.rs b/golem-worker-executor/src/durable_host/logging/policy.rs index b02ee9872f..1c04b5e40a 100644 --- a/golem-worker-executor/src/durable_host/logging/policy.rs +++ b/golem-worker-executor/src/durable_host/logging/policy.rs @@ -112,7 +112,7 @@ pub async fn emit_log_event_with_state( if !replay_state.seen_log(*level, context, message).await { // haven't seen this log before public_state.event_service().emit_event(event.clone(), true); - public_state.worker().add_to_oplog(entry).await; + public_state.worker().add_to_oplog_or_give_up(entry).await; } else { // we have persisted emitting this log before, so we mark it as non-live and // remove the entry from the seen log set. @@ -129,7 +129,18 @@ pub async fn emit_log_event_with_state( public_state.event_service().emit_event(event.clone(), true); if is_live && !replay_state.seen_log(*level, context, message).await { - oplog.add(entry).await; + // Same contract as `Worker::add_to_oplog_or_give_up`, spelled out + // because this writes through the oplog handle passed in rather than the + // worker's own: a fence gives the agent up, anything else is fail-stop. + match oplog.add(entry).await { + Ok(_) => {} + Err(crate::services::oplog::OplogError::Fenced(fence)) => { + public_state.worker().mark_given_up( + crate::worker::GiveUpReason::Fenced(Some(Box::new(fence))), + ); + } + Err(error) => panic!("oplog write: {error}"), + } } } } diff --git a/golem-worker-executor/src/durable_host/mod.rs b/golem-worker-executor/src/durable_host/mod.rs index afd417ab69..6ee8ebb717 100644 --- a/golem-worker-executor/src/durable_host/mod.rs +++ b/golem-worker-executor/src/durable_host/mod.rs @@ -86,7 +86,7 @@ use crate::services::key_value::KeyValueService; use crate::services::linear_memory::{ LinearMemoryTracker, SHARED_LINEAR_MEMORY_ERROR, UnsharedMemoryGrowth, }; -use crate::services::oplog::{CommitLevel, Oplog, OplogOps, OplogService}; +use crate::services::oplog::{CommitLevel, Oplog, OplogError, OplogOps, OplogService}; use crate::services::promise::PromiseService; use crate::services::quota::QuotaService; use crate::services::rdbms::RdbmsService; @@ -700,6 +700,12 @@ fn validate_unshared_memory_growth( } impl DurableWorkerCtx { + /// `trap_type`, or `ShardLost` once this agent's oplog has latched a fence. For the invocation + /// loop, which reaches the oplog only through this context. + pub(crate) fn trap_type_under_latched_fence(&self, trap_type: TrapType) -> TrapType { + trap_type.under_latched_fence(self.state.oplog.fence().as_ref()) + } + #[cfg(feature = "test-utils")] pub(crate) fn test_should_skip_wall_clock_now_durability(&self) -> bool { self.owner_execution @@ -711,7 +717,8 @@ impl DurableWorkerCtx { self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await + .expect("test oplog commit was refused"); } #[cfg(feature = "test-utils")] @@ -2222,7 +2229,7 @@ impl DurableWorkerCtx { card_id, reason, )) - .await; + .await?; } Ok(Err(reason)) } else { @@ -2234,7 +2241,7 @@ impl DurableWorkerCtx { Box::new(card), self.state.wallet_generation, )) - .await; + .await?; Ok(Ok(())) } } @@ -2257,7 +2264,7 @@ impl DurableWorkerCtx { card_id, reason, )) - .await; + .await?; return Ok(Err(reason)); } @@ -2274,7 +2281,7 @@ impl DurableWorkerCtx { Box::new(card), self.state.wallet_generation, )) - .await; + .await?; Ok(Ok(())) } @@ -2306,7 +2313,7 @@ impl DurableWorkerCtx { card_id, self.state.wallet_generation, )) - .await; + .await?; } Ok(()) @@ -2352,9 +2359,12 @@ impl DurableWorkerCtx { local_wallet_generation: self.state.wallet_generation, }; if commit_immediately { - self.public_state.worker().add_and_commit_oplog(entry).await; + self.public_state + .worker() + .add_and_commit_oplog(entry) + .await?; } else { - self.public_state.worker().add_to_oplog(entry).await; + self.public_state.worker().add_to_oplog(entry).await?; } Ok(()) @@ -2393,7 +2403,7 @@ impl DurableWorkerCtx { card_id, wallet_generation, )) - .await; + .await?; } Ok(()) } @@ -2782,6 +2792,9 @@ impl DurableWorkerCtx { TrapType::Interrupt(InterruptKind::Suspend(ts)) => Some(RetryDecision::TryStop(*ts)), TrapType::Interrupt(InterruptKind::Restart) => Some(RetryDecision::Immediate), TrapType::Interrupt(InterruptKind::Jump) => Some(RetryDecision::Immediate), + // Never retried here: a retry in place would reopen the oplog with the same stale + // epoch. The worker service resumes the agent on the shard's owner. + TrapType::Interrupt(InterruptKind::ShardLost) => Some(RetryDecision::None), TrapType::Exit => Some(RetryDecision::None), TrapType::Error { error: AgentError::OutOfMemory, @@ -3091,7 +3104,15 @@ impl DurableWorkerCtx { request: None, durable_function_type: function_type.clone(), }; - let begin_index = self.public_state.worker().add_and_commit_oplog(entry).await; + // The scope's side effect runs as soon as this returns. A `Start` the storage + // refused has to stop it here: the shard's new owner has no record of the scope, + // so nothing would stop it running the effect a second time. + let begin_index = self + .public_state + .worker() + .add_and_commit_oplog(entry) + .await + .map_err(WorkerExecutorError::from)?; Ok(begin_index) } else { let scope_name = HostFunctionName::Custom("".to_string()); @@ -3182,7 +3203,7 @@ impl DurableWorkerCtx { self.entity_parent_start_index(), deleted_region, )) - .await; + .await?; // TODO: this recomputation should not be necessary. self.public_state.worker().reattach_worker_status().await; @@ -3227,6 +3248,25 @@ impl DurableWorkerCtx { self.state.current_retry_point = result; Ok(result) } else { + // No scope opens, so nothing is written before the side effect runs and a fence + // already latched would only surface at the commit after it - by which time the + // effect has happened and the shard's new owner, having no record of it, runs it + // again. Reading the latch costs no storage round trip, so a write whose effect is + // about to run is refused here instead. It does not close the window where the shard + // moves *during* the call: that one needs a round trip per call, which is exactly what + // an idempotent write is declared to avoid. + if self.state.is_live() + && matches!( + function_type, + DurableFunctionType::WriteRemote + | DurableFunctionType::WriteRemoteBatched(_) + | DurableFunctionType::WriteRemoteTransaction(_) + ) + && let Some(fence) = self.state.oplog.fence() + { + return Err(WorkerExecutorError::from(OplogError::Fenced(fence))); + } + // When there is no scope `Start` entry, the current retry point can only // point to the last written non-hint entry. Hint entries must be ignored // because they are nondeterministic. @@ -3273,7 +3313,7 @@ impl DurableWorkerCtx { response: None, forced_commit: true, }; - self.state.oplog.add(entry).await; + self.state.oplog.add(entry).await?; // The durable scope opened in `begin_function` is now closed. self.state.remove_durable_scope(begin_index)?; } else { @@ -3351,7 +3391,7 @@ impl DurableWorkerCtx { response: None, forced_commit: true, }) - .await; + .await?; } } } @@ -3413,11 +3453,16 @@ impl DurableWorkerCtx { scope_start, Box::new(move |_start_index| OplogEntry::begin_remote_transaction(tx_id, None)), ) - .await; + .await + .map_err(WorkerExecutorError::from)?; + // The pair is only buffered until this commit. If the storage refused it, the + // transaction must not be handed out, so `tx` is dropped here before any statement + // has run through it. self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await + .map_err(WorkerExecutorError::from)?; // The transaction scope is now open until commit/rollback; block checkpoints. Opened // live, so there is no recorded scope `End` to await on close. @@ -3579,13 +3624,16 @@ impl DurableWorkerCtx { end: pending.replay_target().next(), // skipping the Jump entry too }; + // Checked, like the begin below: a refused jump means the restart is not ours + // to run, and stopping here avoids opening a database transaction first. self.public_state .worker() .add_and_commit_oplog(OplogEntry::jump( self.entity_parent_start_index(), deleted_region, )) - .await; + .await + .map_err(WorkerExecutorError::from)?; // TODO: this recomputation should not be necessary. self.public_state.worker().reattach_worker_status().await; @@ -3593,14 +3641,16 @@ impl DurableWorkerCtx { self.finish_switch_to_live(pending).await?.require_live()?; let (tx_id, tx) = handler.create_new().await?; - let _ = self - .public_state + // The restarted transaction runs its statements once this returns, so a + // refused begin has to stop it; `tx` is dropped unused. + self.public_state .worker() .add_and_commit_oplog(OplogEntry::begin_remote_transaction( tx_id, Some(original_begin_index), )) - .await; + .await + .map_err(WorkerExecutorError::from)?; // Restarted live (jump + fresh `BeginRemoteTransaction`): the scope `End` will // be appended live by the transaction terminal, so do not store the (now @@ -3635,14 +3685,13 @@ impl DurableWorkerCtx { // make sure to write to the local oplog handle, but still commit to the parent for status consistency. self.state .oplog - .fallible_add(OplogEntry::pre_commit_remote_transaction(begin_index)) - .await - .map_err(WorkerExecutorError::runtime)?; + .add(OplogEntry::pre_commit_remote_transaction(begin_index)) + .await?; self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; Ok(()) } else { let (_, _) = crate::get_oplog_entry!( @@ -3664,14 +3713,13 @@ impl DurableWorkerCtx { // make sure to write to the local oplog handle, but still commit to the parent for status consistency. self.state .oplog - .fallible_add(OplogEntry::pre_rollback_remote_transaction(begin_index)) - .await - .map_err(WorkerExecutorError::runtime)?; + .add(OplogEntry::pre_rollback_remote_transaction(begin_index)) + .await?; self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; Ok(()) } else { let (_, _) = crate::get_oplog_entry!( @@ -3787,21 +3835,20 @@ impl DurableWorkerCtx { // successful append can never leave one without the other. self.state .oplog - .fallible_add_pair( + .add_pair( marker, - OplogEntry::End { + Box::new(move |_| OplogEntry::End { timestamp: Timestamp::now_utc(), start_index: begin_index, response: None, forced_commit: true, - }, + }), ) - .await - .map_err(WorkerExecutorError::runtime)?; + .await?; self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; self.state.remove_durable_scope(begin_index)?; Ok(()) } @@ -3875,7 +3922,7 @@ impl DurableWorkerCtx { "Manual update failed to lower load-snapshot invocation: {err}" )), ) - .await; + .await?; return Ok(Some(RetryDecision::Immediate)); } }; @@ -3898,7 +3945,7 @@ impl DurableWorkerCtx { "Manual update failed to install invocation context: {err}" )), ) - .await; + .await?; return Ok(Some(RetryDecision::Immediate)); } @@ -3968,7 +4015,7 @@ impl DurableWorkerCtx { .as_context_mut() .data_mut() .on_worker_update_failed(target_revision, Some(error)) - .await; + .await?; Ok(Some(RetryDecision::Immediate)) } else { let component_metadata = @@ -3994,7 +4041,7 @@ impl DurableWorkerCtx { }), ), ) - .await; + .await?; Ok(None) } } @@ -4006,7 +4053,7 @@ impl DurableWorkerCtx { target_revision, Some("Failed to find snapshot data for update".to_string()), ) - .await; + .await?; Ok(Some(RetryDecision::Immediate)) } Err(error) => { @@ -4014,7 +4061,7 @@ impl DurableWorkerCtx { .as_context_mut() .data_mut() .on_worker_update_failed(target_revision, Some(error)) - .await; + .await?; Ok(Some(RetryDecision::Immediate)) } } @@ -4735,7 +4782,7 @@ impl DurableWorkerCtx { target_revision, Some(stringified_error), ) - .await; + .await?; Err(error)? }; @@ -4755,7 +4802,7 @@ impl DurableWorkerCtx { }) }), ) - .await; + .await?; debug!("Finalizing automatic update to revision {target_revision}"); } @@ -4803,7 +4850,7 @@ impl DurableWorkerCtx { self.public_state .worker() .queue_card_revocations_locked(&revoked_card_ids) - .await; + .await?; Ok(()) } @@ -5115,18 +5162,24 @@ impl InvocationHooks for DurableWorkerCtx { }, ) .await - .unwrap_or_else(|err| { - panic!( + .map_err(|err| match err { + OplogError::Fenced(fence) => self.public_state.worker().given_up_by(fence), + err => panic!( "could not encode agent invocation on {}: {err}", self.agent_id() - ) - }); + ), + })?; self.primary_invocation_start_index = Some(start_index); + #[cfg(feature = "test-utils")] + self.owner_execution + .test_after_invocation_started_buffered() + .await; + self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; } Ok(()) } @@ -5140,13 +5193,42 @@ impl InvocationHooks for DurableWorkerCtx { full_function_name: &str, trap_type: &TrapType, ) -> RetryDecision { + // Covers the callers that hand over a trap and do not branch on it afterwards; the ones + // that do reclassify before this call, so they dispatch on the same kind. + let trap_type = &self.trap_type_under_latched_fence(trap_type.clone()); let current_idempotency_key = self.get_current_idempotency_key().await; + // Deliberately above the dropped-call drain: that drain appends `Cancelled` entries, and + // a given-up agent's oplog belongs to another executor now. Nothing further is + // written for it - not the drain, not an `Error` entry, not a status change - whatever + // the trap was: a revoke latches no fence, so the mark is all that says so, and the shard's + // new owner replays the invocation and records its outcome itself. + let worker = self.public_state.worker(); + let given_up = if matches!(trap_type, TrapType::Interrupt(InterruptKind::ShardLost)) { + worker.give_up_if_shard_lost(&WorkerExecutorError::Interrupted { + kind: InterruptKind::ShardLost, + }) + } else { + worker.is_given_up() + }; + if given_up { + return RetryDecision::None; + } + if self.state.is_live() && !self.state.snapshotting_mode && let Err(err) = concurrent::drain_queued_dropped_call_events(self).await { - error!("failed to drain dropped durable calls before invocation failure entry: {err}"); + error!( + error = %err, + "Failed to drain dropped durable calls before the invocation failure entry" + ); + // A `Cancelled` refused by a fence latched during the drain gives the agent up, so + // the stop that follows drops this generation instead of leaving it cached to be + // restarted in place. + self.public_state + .worker() + .give_up_if_shard_lost(&err.source); return RetryDecision::None; } @@ -5194,7 +5276,8 @@ impl InvocationHooks for DurableWorkerCtx { }, ) = (¤t_idempotency_key, trap_type) { - self.state + let denial_persisted = self + .state .oplog .add_pair( OplogEntry::cancel_pending_invocation(idempotency_key.clone()), @@ -5216,10 +5299,29 @@ impl InvocationHooks for DurableWorkerCtx { }), ) .await; - self.public_state + match denial_persisted { + Ok(_) => {} + Err(crate::services::oplog::OplogError::Fenced(fence)) => { + // The shard moved while this failure was being recorded. Give the agent up + // exactly as the `ShardLost` arm above does: nothing further may be written + // to an oplog that belongs to another executor now. + self.public_state + .worker() + .mark_given_up(crate::worker::GiveUpReason::Fenced(Some(Box::new(fence)))); + return RetryDecision::None; + } + Err(error) => panic!("oplog write: {error}"), + } + // Refused, like the add above: the agent has been given up. + if self + .public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await + .is_err() + { + return RetryDecision::None; + } true } else { false @@ -5230,6 +5332,8 @@ impl InvocationHooks for DurableWorkerCtx { TrapType::Interrupt(InterruptKind::Suspend(_)) => Some(OplogEntry::suspend()), TrapType::Interrupt(InterruptKind::Jump) => None, TrapType::Interrupt(InterruptKind::Restart) => None, + // The oplog is the new owner's; a stale writer must leave no trace in it. + TrapType::Interrupt(InterruptKind::ShardLost) => None, TrapType::Exit => Some(OplogEntry::exited()), TrapType::Error { error: AgentError::PermissionDenied(_), @@ -5253,8 +5357,17 @@ impl InvocationHooks for DurableWorkerCtx { )), }; - if let Some(entry) = oplog_entry { - self.public_state.worker().add_and_commit_oplog(entry).await; + // Refused, the agent has been given up: no failure is published for its invocation, which + // the shard's new owner runs. + if let Some(entry) = oplog_entry + && self + .public_state + .worker() + .add_and_commit_oplog(entry) + .await + .is_err() + { + return RetryDecision::None; }; let latest_status = self @@ -5425,9 +5538,12 @@ impl InvocationHooks for DurableWorkerCtx { component_revision, ) .await - .unwrap_or_else(|err| { - panic!("could not encode function result for {full_function_name}: {err}") - }); + .map_err(|err| match err { + OplogError::Fenced(fence) => self.public_state.worker().given_up_by(fence), + err => { + panic!("could not encode function result for {full_function_name}: {err}") + } + })?; let commit_level = match self.public_state.worker().agent_mode() { AgentMode::Durable => CommitLevel::Always, @@ -5436,7 +5552,7 @@ impl InvocationHooks for DurableWorkerCtx { self.public_state .worker() .commit_oplog_before_status_update(commit_level) - .await; + .await?; // Bump the read-only cache epoch after the // `AgentInvocationFinished` entry is committed, but *before* @@ -5508,7 +5624,10 @@ impl ResourceStore for DurableWorkerCtx { resource_id, name.clone(), ); - self.public_state.worker().add_to_oplog(entry).await; + self.public_state + .worker() + .add_to_oplog_or_give_up(entry) + .await; } id } @@ -5523,7 +5642,10 @@ impl ResourceStore for DurableWorkerCtx { id, resource_type_id.clone(), ); - self.public_state.worker().add_to_oplog(entry).await; + self.public_state + .worker() + .add_to_oplog_or_give_up(entry) + .await; } } result @@ -5565,15 +5687,22 @@ impl UpdateManagement for DurableWorkerCtx { &self, target_revision: ComponentRevision, details: Option, - ) { + ) -> Result<(), WorkerExecutorError> { + // A given-up agent's update is settled by the shard's new owner. A revoke latches no + // fence, so the storage would still accept this entry, and it would drop the update there. + let worker = self.public_state.worker(); + if worker.is_given_up() { + return Err(worker.give_up_error()); + } let entry = OplogEntry::failed_update(target_revision, details.clone()); - self.public_state.worker().add_and_commit_oplog(entry).await; + worker.add_and_commit_oplog(entry).await?; warn!( "Worker failed to update to {}: {}, update attempt aborted", target_revision, details.unwrap_or_else(|| "?".to_string()) ); + Ok(()) } async fn on_worker_update_succeeded( @@ -5583,9 +5712,13 @@ impl UpdateManagement for DurableWorkerCtx { new_active_plugins: HashSet< golem_common::base_model::environment_plugin_grant::EnvironmentPluginGrantId, >, - ) { + ) -> Result<(), WorkerExecutorError> { info!("Worker update to {} finished successfully", target_revision); let worker = self.public_state.worker(); + // As for a failed update: the outcome is the shard's new owner's to record. + if worker.is_given_up() { + return Err(worker.give_up_error()); + } worker .persist_successful_update( &self.linear_memory, @@ -5593,7 +5726,8 @@ impl UpdateManagement for DurableWorkerCtx { new_component_size, new_active_plugins, ) - .await; + .await?; + Ok(()) } } @@ -5688,7 +5822,7 @@ impl InvocationContextManagement for DurableWorkerCtx { linked_context_id: span.linked_context().map(|link| link.span_id().clone()), attributes: HashMap::from_iter(initial_attributes.iter().cloned()).into(), }) - .await; + .await?; } Ok(span) @@ -5732,7 +5866,7 @@ impl InvocationContextManagement for DurableWorkerCtx { self.entity_parent_start_index(), span_id.clone(), )) - .await; + .await?; } if &self.state.current_span_id == span_id { @@ -5783,7 +5917,7 @@ impl InvocationContextManagement for DurableWorkerCtx { key.to_string(), value, )) - .await; + .await?; } Ok(()) } @@ -6077,7 +6211,18 @@ impl ExternalOperations for DurableWorkerCtx { store.as_context().data().agent_mode(), )) } - }; + } + // Every arm below dispatches on the kind. Left an `Error`, a fence + // whose type was lost on the way would abandon the snapshot or break + // with `InvocationFailed` into a recovery failure, which unloads the + // agent as failed instead of giving it up. + .map(|trap_type| { + store + .as_context() + .data() + .durable_ctx() + .trap_type_under_latched_fence(trap_type) + }); let decision = match trap_type { // A recorded invocation that fails while its entries are still // being replayed after an automatic snapshot load most likely @@ -6116,7 +6261,14 @@ impl ExternalOperations for DurableWorkerCtx { error, stderr: store.as_context().data().get_public_state().event_service().get_last_invocation_errors(), }), - TrapType::Interrupt(kind) => Self::fixed_decision_for_trap_type(&TrapType::Interrupt(kind)), + TrapType::Interrupt(kind) => { + // `on_invocation_failure` is skipped on this path, + // so a lost shard is given up here: its `None` + // decision alone would stop the agent as an + // ordinary one, left cached to restart in place. + worker.give_up_if_shard_lost(&WorkerExecutorError::Interrupted { kind }); + Self::fixed_decision_for_trap_type(&TrapType::Interrupt(kind)) + } TrapType::Exit => break Err(WorkerExecutorError::runtime("Process exited during snapshot replay")), } } @@ -6133,7 +6285,16 @@ impl ExternalOperations for DurableWorkerCtx { // interrupted by a crash and cannot be retried. // A fresh interrupt is authoritative even before live // publication; already-finished sessions stay unchanged. + // Not for an agent given up here, however the loss + // reached the trap: the shard's new owner resumes that + // invocation. + let shard_lost = worker.is_given_up() + || matches!( + trap_type, + TrapType::Interrupt(InterruptKind::ShardLost) + ); if uses_streams + && !shard_lost && (matches!(trap_type, TrapType::Interrupt(_)) || store.as_context().data().durable_ctx().is_live()) { @@ -6256,7 +6417,7 @@ impl ExternalOperations for DurableWorkerCtx { .get_public_state() .oplog() .add(OplogEntry::restart()) - .await; + .await?; Ok(None) } else { @@ -6337,7 +6498,7 @@ impl ExternalOperations for DurableWorkerCtx { "Automatic update failed: {error}" )), ) - .await; + .await?; debug!( "Retrying prepare_instance after failed update attempt" @@ -6504,11 +6665,8 @@ impl ExternalOperations for DurableWorkerCtx { "failed to restart {owned_agent_id} during shard-assignment recovery: {error}" )); } - Err(error) => { - return Err(anyhow!( - "failed to restart {owned_agent_id} during shard-assignment recovery: {error}" - )); - } + // A shard that left the assignment mid-recovery is skipped, not fatal. + Err(error) => recovered_restart(&owned_agent_id, Err::<(), _>(error))?, } } } @@ -6918,6 +7076,31 @@ fn recovered_status( } } +/// The outcome of restarting one recovered agent, drawing the same line as [`recovered_status`]. +/// +/// `ShardingNotReady` means the agent's shard left this executor's assignment while recovery was +/// running: a later delivery revoked it, and the worker was refused an epoch rather than opened +/// unfenced. `OplogFenced` means another executor claimed the agent's oplog at a newer epoch +/// before this one wrote to it. Either way the agent belongs to the shard's new owner, which +/// recovers it there. This is a skip, like an oplog that is gone. Failing the whole assignment for +/// it would stop every other agent in the scan from being resumed. Any other restart failure still +/// fails the assignment. +fn recovered_restart( + owned_agent_id: &OwnedAgentId, + restarted: Result, +) -> Result<(), anyhow::Error> { + match restarted { + Ok(_) => Ok(()), + Err(WorkerExecutorError::ShardingNotReady | WorkerExecutorError::OplogFenced { .. }) => { + debug!(agent_id = %owned_agent_id, "Worker's shard left the assignment during shard-assignment recovery; skipping agent"); + Ok(()) + } + Err(error) => Err(anyhow!( + "failed to restart {owned_agent_id} during shard-assignment recovery: {error}" + )), + } +} + fn should_restart_after_shard_assignment_change(status: &AgentStatusRecord) -> bool { status.status != AgentStatus::Interrupted && (matches!( @@ -8919,6 +9102,43 @@ mod tests { ); } + /// A revoke that races recovery refuses the worker an epoch. That agent is skipped, so the rest + /// of the scan is still resumed. Any other restart failure still fails the assignment. + #[test] + fn shard_assignment_recovery_skips_a_worker_whose_shard_left_the_assignment() { + assert!(recovered_restart(&recovered_agent(), Ok(())).is_ok()); + assert!( + recovered_restart::<()>( + &recovered_agent(), + Err(WorkerExecutorError::ShardingNotReady) + ) + .is_ok() + ); + let agent = recovered_agent(); + assert!( + recovered_restart::<()>( + &agent, + Err(WorkerExecutorError::oplog_fenced( + agent.agent_id.clone(), + 2, + Some(3) + )) + ) + .is_ok() + ); + + let error = recovered_restart::<()>( + &recovered_agent(), + Err(WorkerExecutorError::runtime("instance failed to start")), + ) + .expect_err("a restart failure other than a lost shard was skipped"); + assert!( + error.to_string().contains("failed to restart") + && error.to_string().contains("instance failed to start"), + "{error}" + ); + } + fn open_region(regions: &mut Vec, begin: u64) -> OplogIndex { let begin_index = OplogIndex::from_u64(begin); regions.push(ActiveAtomicRegion::new(begin_index, begin_index.next())); diff --git a/golem-worker-executor/src/durable_host/p3/http/replay.rs b/golem-worker-executor/src/durable_host/p3/http/replay.rs index 23207b9d3d..bdd5758e93 100644 --- a/golem-worker-executor/src/durable_host/p3/http/replay.rs +++ b/golem-worker-executor/src/durable_host/p3/http/replay.rs @@ -37,7 +37,7 @@ use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; use std::task::Poll; use tokio::sync::{Notify, oneshot}; -use tracing::warn; +use tracing::{debug, warn}; use wasmtime::AsContextMut; use wasmtime::component::{Accessor, AccessorTask, Resource}; use wasmtime_wasi_http::p3::WasiHttp; @@ -556,6 +556,16 @@ async fn record_frame_or_warn( ) -> bool { match record_frame_entry(oplog.clone(), send_start_index, frame).await { Ok(_) => true, + // Not a recording fault: the storage turned the write away because the shard has a new + // owner, and the latch gives the agent up on its next durable write. + Err(error) if oplog.fence().is_some() => { + debug!( + send_start_index = %send_start_index, + error = %error, + "Request-body frame not recorded: the shard moved; the next durable write gives the agent up" + ); + false + } Err(error) => { warn!( send_start_index = %send_start_index, diff --git a/golem-worker-executor/src/durable_host/p3/http/request_body.rs b/golem-worker-executor/src/durable_host/p3/http/request_body.rs index b4c1672670..59a16fa7aa 100644 --- a/golem-worker-executor/src/durable_host/p3/http/request_body.rs +++ b/golem-worker-executor/src/durable_host/p3/http/request_body.rs @@ -464,13 +464,14 @@ pub(super) async fn record_frame_entry( let bytes = serialize(&request)?; let raw = oplog.upload_raw_payload(bytes).await?; let payload = raw.into_payload::()?; - Ok(oplog + oplog .add(OplogEntry::host_stream_frame( parent_start_index, HostStreamKind::P3HttpRequestBody, payload, )) - .await) + .await + .map_err(|error| error.to_string()) } /// Loads one recorded data/trailers frame back from its `HostStreamFrame` diff --git a/golem-worker-executor/src/durable_host/p3/http/response_body.rs b/golem-worker-executor/src/durable_host/p3/http/response_body.rs index 371379005f..7a38eac5ab 100644 --- a/golem-worker-executor/src/durable_host/p3/http/response_body.rs +++ b/golem-worker-executor/src/durable_host/p3/http/response_body.rs @@ -2001,7 +2001,8 @@ mod tests { )))), durable_function_type: DurableFunctionType::WriteRemoteBatched(None), }) - .await; + .await + .unwrap(); let child_start = oplog .add(OplogEntry::Start { timestamp: Timestamp::now_utc(), @@ -2016,7 +2017,8 @@ mod tests { OplogIndex::from_u64(1), )), }) - .await; + .await + .unwrap(); oplog .add(OplogEntry::End { timestamp: Timestamp::now_utc(), @@ -2030,7 +2032,8 @@ mod tests { ))), forced_commit: false, }) - .await; + .await + .unwrap(); child_start } diff --git a/golem-worker-executor/src/durable_host/p3/http/send.rs b/golem-worker-executor/src/durable_host/p3/http/send.rs index 64872fb3b2..61800a5113 100644 --- a/golem-worker-executor/src/durable_host/p3/http/send.rs +++ b/golem-worker-executor/src/durable_host/p3/http/send.rs @@ -34,6 +34,7 @@ use crate::durable_host::http::policy::{ use crate::durable_host::http::types::classify_serializable_http_error_code; use crate::durable_host::p3::{DurableP3, DurableP3View, durable_worker_ctx, wasi_http_view}; use crate::services::HasWorker; +use crate::services::oplog::{Oplog, OplogError}; use crate::workerctx::WorkerCtx; use anyhow::Context as _; use bytes::Bytes; @@ -856,6 +857,27 @@ where } Err(error_code) => { let _ = physical.final_transmission_tx.send(Err(error_code.clone())); + + // A write the storage refused because the shard moved latches the oplog, and nothing + // on this path sees the latch otherwise: a below-threshold frame add still succeeds, + // and so does this send's buffered `End`. Without this read the guest would be + // handed an HTTP error produced by, or racing, the lost shard instead of the + // ShardLost trap. `handle.trap` abandons the call exactly as the retry trap below + // does. + let latched = store.with(|mut access| { + latched_fence_error( + durable_worker_ctx::(access.data_mut()) + .state + .oplog + .as_ref(), + ) + }); + if let Some(error) = latched { + return Err(HttpError::trap(wasmtime::Error::from_anyhow( + handle.trap(error), + ))); + } + let serialized_error = serialize_error_code(&error_code); // Worker-level retry classification, mirroring the P2 @@ -948,6 +970,13 @@ where } } +/// The error a send traps with once the oplog has latched a fence, or `None` while it has not. +fn latched_fence_error(oplog: &dyn Oplog) -> Option { + oplog + .fence() + .map(|fence| WorkerExecutorError::from(OplogError::Fenced(fence))) +} + pub(super) struct PhysicalSendHttpError { error_code: ErrorCode, final_transmission_tx: oneshot::Sender>, @@ -1487,3 +1516,76 @@ pub(super) fn apply_headers_to_request_resource( .map_err(WorkerExecutorError::runtime) }) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::durable_host::durability::{DurableCallTrapContext, mark_durable_call_trap_context}; + use crate::durable_host::p3::http::test_support::*; + use crate::model::TrapType; + use crate::services::oplog::OplogFence; + use golem_common::model::agent::AgentMode; + use golem_common::model::component::ComponentId; + use golem_common::model::oplog::payload::types::SerializableP3HttpRequestBodyFrame; + use golem_common::model::{AgentId, ShardEpoch}; + use golem_service_base::error::worker_executor::InterruptKind; + use test_r::test; + + /// The send's gate cannot rely on the recording: once the fence has latched, a frame add that + /// stays below the commit threshold still succeeds. Only the latch says the shard moved, and + /// the error read from it has to keep its ShardLost classification through `handle.trap`. + /// + /// The gate inside `send_with_durability` needs a wasmtime `Accessor` and a live worker + /// context, so it is exercised here through the pieces it is built from. + #[test] + async fn a_send_failure_after_a_latched_fence_traps_as_shard_lost_even_when_frame_adds_succeed() + { + let oplog = FrameTestOplog::new(); + assert!( + latched_fence_error(oplog.as_ref()).is_none(), + "no fence has latched, so a send failure is a genuine HTTP error" + ); + + oplog.latch_fence(OplogFence { + agent_id: AgentId { + component_id: ComponentId::new(), + agent_id: "sender".to_string(), + }, + expected_epoch: ShardEpoch(8), + actual_epoch: Some(ShardEpoch(9)), + writer_conflict: false, + }); + record_frame_entry( + oplog.clone(), + OplogIndex::NONE, + SerializableP3HttpRequestBodyFrame::End, + ) + .await + .expect("a frame add below the commit threshold does not see the latch"); + + let error = latched_fence_error(oplog.as_ref()).expect("the latch must be read"); + assert!( + matches!(error, WorkerExecutorError::OplogFenced { .. }), + "expected a fenced error, got {error:?}" + ); + + let trapped = mark_durable_call_trap_context( + anyhow::Error::from(error), + DurableCallTrapContext { + retry_from: OplogIndex::INITIAL, + in_atomic_region: false, + }, + ); + let trap = TrapType::from_error::( + &trapped, + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + assert!( + matches!(trap, TrapType::Interrupt(InterruptKind::ShardLost)), + "the trapped send must give the agent up, got {trap:?}" + ); + } +} diff --git a/golem-worker-executor/src/durable_host/p3/http/test_support.rs b/golem-worker-executor/src/durable_host/p3/http/test_support.rs index 39b25fd6c1..5e9ca0669c 100644 --- a/golem-worker-executor/src/durable_host/p3/http/test_support.rs +++ b/golem-worker-executor/src/durable_host/p3/http/test_support.rs @@ -13,7 +13,7 @@ // limitations under the License. use super::*; -use crate::services::oplog::{CommitLevel, Oplog, OplogAddReceipt, OrderedOplogStart}; +use crate::services::oplog::{CommitLevel, Oplog, OplogAddReceipt, OplogFence, OrderedOplogStart}; use async_trait::async_trait; use bytes::Bytes; use golem_common::model::oplog::payload::types::{ @@ -43,6 +43,9 @@ use wasmtime_wasi_http::{FieldMap, WasiHttpCtx}; pub(super) struct FrameTestOplog { entries: std::sync::Mutex>, upload_gate: tokio::sync::Semaphore, + /// What `fence()` answers. `add` and `enqueue_add` keep succeeding while it is set, as the + /// primary oplog's below-threshold adds do after the fence has latched. + fence: std::sync::Mutex>, } impl FrameTestOplog { @@ -50,6 +53,7 @@ impl FrameTestOplog { Arc::new(Self { entries: std::sync::Mutex::new(Vec::new()), upload_gate: tokio::sync::Semaphore::new(tokio::sync::Semaphore::MAX_PERMITS), + fence: std::sync::Mutex::new(None), }) } @@ -59,9 +63,15 @@ impl FrameTestOplog { Arc::new(Self { entries: std::sync::Mutex::new(Vec::new()), upload_gate: tokio::sync::Semaphore::new(0), + fence: std::sync::Mutex::new(None), }) } + /// Latches `fence`, as a write the storage refused on another path would. + pub(super) fn latch_fence(&self, fence: OplogFence) { + *self.fence.lock().unwrap() = Some(fence); + } + pub(super) fn release_uploads(&self, n: usize) { self.upload_gate.add_permits(n); } @@ -119,44 +129,47 @@ impl FrameTestOplog { #[async_trait] impl Oplog for FrameTestOplog { - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + entry: OplogEntry, + ) -> Result { let mut entries = self.entries.lock().unwrap(); entries.push(entry); - OplogIndex::from_u64(entries.len() as u64) + Ok(OplogIndex::from_u64(entries.len() as u64)) } fn enqueue_add(&self, entry: OplogEntry) -> OplogAddReceipt { let mut entries = self.entries.lock().unwrap(); entries.push(entry); let index = OplogIndex::from_u64(entries.len() as u64); - Box::pin(async move { index }) + Box::pin(async move { Ok(index) }) } async fn add_pair( &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { let mut entries = self.entries.lock().unwrap(); entries.push(start); let first_idx = OplogIndex::from_u64(entries.len() as u64); entries.push(make_second(first_idx)); let second_idx = OplogIndex::from_u64(entries.len() as u64); - (first_idx, second_idx) + Ok((first_idx, second_idx)) } async fn add_start_with_reserved_raw_payload( &self, _serialized_request: Vec, _build_start: Box Result + Send>, - ) -> Result { + ) -> Result { unimplemented!() } async fn add_start_with_indexed_reserved_raw_payload( &self, _build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { unimplemented!() } @@ -164,8 +177,11 @@ impl Oplog for FrameTestOplog { 0 } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { - BTreeMap::new() + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { + Ok(BTreeMap::new()) } async fn current_oplog_index(&self) -> OplogIndex { @@ -204,6 +220,10 @@ impl Oplog for FrameTestOplog { self.entries.lock().unwrap().len() as u64 } + fn fence(&self) -> Option { + self.fence.lock().unwrap().clone() + } + async fn upload_raw_payload(&self, data: Vec) -> Result { let permit = self .upload_gate diff --git a/golem-worker-executor/src/durable_host/permissions/mod.rs b/golem-worker-executor/src/durable_host/permissions/mod.rs index 8910de66e7..ead6684d9a 100644 --- a/golem-worker-executor/src/durable_host/permissions/mod.rs +++ b/golem-worker-executor/src/durable_host/permissions/mod.rs @@ -981,7 +981,7 @@ where card: Box::new(created.clone()), wallet_generation, }) - .await; + .await?; } else { return Err(anyhow!( "replayed runtime permission-card creation {card_id} is missing its CardDerived audit event" @@ -1710,7 +1710,7 @@ async fn complete_source_card_transfer( installed_card.card_id(), target_holder, )) - .await; + .await?; Ok(()) } @@ -1751,7 +1751,7 @@ async fn ensure_source_card_transfer_started( target_holder.clone(), ctx.state.wallet_generation, )) - .await; + .await?; Ok(()) } @@ -1803,7 +1803,7 @@ async fn execute_source_card_transfer( card: Box::new(installed_card.clone()), wallet_generation: ctx.state.wallet_generation, }) - .await; + .await?; } ctx.public_state @@ -1817,7 +1817,7 @@ async fn execute_source_card_transfer( transfer.target_holder(), )), )) - .await; + .await?; } complete_source_card_transfer( @@ -2142,7 +2142,7 @@ pub(super) async fn complete_pending_source_card_transfers( agent_id: retry.target_agent_id, }), )) - .await; + .await?; } Ok(()) @@ -2798,7 +2798,7 @@ impl permissions_wallet::Host for DurableWorkerCtx { self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; } let start_index = handle.start_index(); diff --git a/golem-worker-executor/src/durable_host/rdbms/mod.rs b/golem-worker-executor/src/durable_host/rdbms/mod.rs index c796443708..6a5df3e14e 100644 --- a/golem-worker-executor/src/durable_host/rdbms/mod.rs +++ b/golem-worker-executor/src/durable_host/rdbms/mod.rs @@ -23,6 +23,7 @@ use crate::durable_host::{ DurabilityHost, DurableWorkerCtx, InternalRetryResult, LiveAuthorizationPermit, RemoteTransactionHandler, }; +use crate::services::oplog::OplogError; use crate::services::rdbms::{DbResult, DbRow, RdbmsType}; use crate::services::rdbms::{RdbmsError, RdbmsService, RdbmsTransactionStatus, RdbmsTypeService}; use crate::workerctx::WorkerCtx; @@ -37,6 +38,7 @@ use golem_common::model::oplog::{ }; use golem_common::model::retry_policy::RetryProperties; use golem_common::model::{AgentId, OplogIndex, RdbmsPoolKey, RetryContext, TransactionId}; +use golem_service_base::error::worker_executor::WorkerExecutorError; use std::marker::PhantomData; use std::ops::Deref; use std::sync::Arc; @@ -304,7 +306,17 @@ where let resource = ctx.as_wasi_view().table().push(entry)?; Ok(Ok(resource)) } - Err(error) => Ok(Err(error.into())), + Err(error) => { + // The handler's error type flattens the begin's own fence into an `RdbmsError`, so it + // is read from the oplog's latch. A refused begin is a lost shard, not a database + // failure the guest may catch and work around: it traps, and the agent is given up. + if let Some(fence) = ctx.state.oplog.fence() { + return Err(anyhow!(WorkerExecutorError::from(OplogError::Fenced( + fence + )))); + } + Ok(Err(error.into())) + } } } diff --git a/golem-worker-executor/src/durable_host/replay_state/tests.rs b/golem-worker-executor/src/durable_host/replay_state/tests.rs index 763246ed26..35e7a64d23 100644 --- a/golem-worker-executor/src/durable_host/replay_state/tests.rs +++ b/golem-worker-executor/src/durable_host/replay_state/tests.rs @@ -62,39 +62,42 @@ impl InMemoryOplog { #[async_trait] impl Oplog for InMemoryOplog { - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + entry: OplogEntry, + ) -> Result { let mut entries = self.entries.lock().unwrap(); entries.push(entry); - OplogIndex::from_u64(entries.len() as u64) + Ok(OplogIndex::from_u64(entries.len() as u64)) } fn enqueue_add(&self, entry: OplogEntry) -> OplogAddReceipt { let mut entries = self.entries.lock().unwrap(); entries.push(entry); let index = OplogIndex::from_u64(entries.len() as u64); - Box::pin(async move { index }) + Box::pin(async move { Ok(index) }) } async fn add_pair( &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { let mut entries = self.entries.lock().unwrap(); entries.push(start); let first_idx = OplogIndex::from_u64(entries.len() as u64); entries.push(make_second(first_idx)); let second_idx = OplogIndex::from_u64(entries.len() as u64); - (first_idx, second_idx) + Ok((first_idx, second_idx)) } async fn add_start_with_reserved_raw_payload( &self, serialized_request: Vec, build_start: Box Result + Send>, - ) -> Result { + ) -> Result { let entry = build_start(RawOplogPayload::SerializedInline(serialized_request))?; - let index = self.add(entry.clone()).await; + let index = self.add(entry.clone()).await?; Ok(OrderedOplogStart { index, entry, @@ -105,7 +108,7 @@ impl Oplog for InMemoryOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let mut entries = self.entries.lock().unwrap(); let index = OplogIndex::from_u64(entries.len() as u64 + 1); let (serialized_request, build_start) = build_request(index)?; @@ -122,8 +125,11 @@ impl Oplog for InMemoryOplog { 0 } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { - BTreeMap::new() + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { + Ok(BTreeMap::new()) } async fn current_oplog_index(&self) -> OplogIndex { @@ -1061,7 +1067,7 @@ fn fork_start() -> OplogEntry { async fn replay_state_over(entries: Vec) -> ReplayState { let oplog = Arc::new(InMemoryOplog::new()); for entry in entries { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; test_replay_state(test_agent_id(), oplog, DeletedRegions::default(), None) @@ -1093,9 +1099,9 @@ async fn held_completed_reconstruction() -> ( let parent = OplogIndex::from_u64(1); let (start, identity) = rejected_tool_reconstruction_start(parent); let oplog = Arc::new(InMemoryOplog::new()); - oplog.add(noop()).await; - oplog.add(start).await; - oplog.add(end_for(2, 1)).await; + oplog.add(noop()).await.unwrap(); + oplog.add(start).await.unwrap(); + oplog.add(end_for(2, 1)).await.unwrap(); let replay = test_replay_state( test_agent_id(), oplog.clone(), @@ -1119,7 +1125,7 @@ async fn held_completed_reconstruction() -> ( #[test] async fn growing_replay_target_revokes_published_live_state() { let oplog = Arc::new(InMemoryOplog::new()); - oplog.add(noop()).await; + oplog.add(noop()).await.unwrap(); let replay = test_replay_state( test_agent_id(), oplog.clone(), @@ -1130,7 +1136,7 @@ async fn growing_replay_target_revokes_published_live_state() { .expect("failed to build replay state"); assert!(replay.is_live_published()); - let new_target = oplog.add(noop()).await; + let new_target = oplog.add(noop()).await.unwrap(); replay .set_replay_target(new_target) .await @@ -1166,7 +1172,7 @@ async fn growing_replay_target_revokes_an_active_settling_transition() { .await .expect("primary transition did not enter settling"); - let new_target = oplog.add(noop()).await; + let new_target = oplog.add(noop()).await.unwrap(); replay .set_replay_target(new_target) .await @@ -1276,10 +1282,10 @@ async fn target_growth_does_not_misclassify_a_reconstruction_as_incomplete() { let (first_start, identity) = rejected_tool_reconstruction_start(parent); let (second_start, _) = rejected_tool_reconstruction_start(parent); let oplog = Arc::new(InMemoryOplog::new()); - oplog.add(noop()).await; - oplog.add(first_start).await; - oplog.add(second_start).await; - oplog.add(end_for(3, 2)).await; + oplog.add(noop()).await.unwrap(); + oplog.add(first_start).await.unwrap(); + oplog.add(second_start).await.unwrap(); + oplog.add(end_for(3, 2)).await.unwrap(); let replay = test_replay_state( test_agent_id(), oplog.clone(), @@ -1325,7 +1331,7 @@ async fn target_growth_does_not_misclassify_a_reconstruction_as_incomplete() { "the incomplete candidate bypassed the completed reconstruction fence" ); - let new_target = oplog.add(end_for(2, 1)).await; + let new_target = oplog.add(end_for(2, 1)).await.unwrap(); replay .set_replay_target(new_target) .await @@ -1400,7 +1406,7 @@ async fn concurrent_same_target_transitions_are_idempotent() { async fn old_settler_cannot_publish_a_grown_target() { let (replay, oplog, reconstruction) = held_completed_reconstruction().await; let old_target = replay.switch_cursor_to_live().await.unwrap(); - let new_target = oplog.add(noop()).await; + let new_target = oplog.add(noop()).await.unwrap(); replay .set_replay_target(new_target) .await @@ -1450,7 +1456,7 @@ async fn old_settler_cannot_publish_a_grown_target() { #[test] async fn owner_failure_wins_when_reconstruction_barrier_is_already_empty() { let oplog = Arc::new(InMemoryOplog::new()); - oplog.add(noop()).await; + oplog.add(noop()).await.unwrap(); let owner_operations = crate::durable_host::tool::operation::OwnerToolOperations::new(); let replay = ReplayState::new_for_owner( test_agent_id(), @@ -1549,7 +1555,7 @@ async fn permission_events_replay_after_invocation_wallet_pin() { }, start_now(), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let replay_state = test_replay_state(owned_agent_id, oplog, DeletedRegions::default(), None) @@ -1676,7 +1682,7 @@ async fn permission_events_are_recovered_from_skipped_regions() { }, start_now(), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion::from_range(2..=2)]); @@ -1712,7 +1718,7 @@ async fn snapshot_prefix_suppresses_replayed_permission_events() { }, start_now(), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion::from_range(2..=2)]); @@ -1839,7 +1845,7 @@ async fn missing_start_claim_remains_divergence_while_replaying() { async fn start_claim_reports_matching_deleted_region_while_replay_continues() { let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), start_with_parent(1)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion::from_range(2..=2)]); @@ -1869,7 +1875,7 @@ async fn assert_request_payload_failure_is_not_reclassified_as_deleted_region( start_now_with_request_payload(OplogPayload::Inline(Box::new(expected_request.clone()))), start_now_with_request_payload(failing_payload), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion::from_range(2..=2)]); @@ -1932,7 +1938,7 @@ async fn genuine_request_mismatch_still_reports_matching_deleted_region() { start_now_with_request_payload(OplogPayload::Inline(Box::new(expected_request.clone()))), start_now_with_request_payload(OplogPayload::Inline(Box::new(different_request))), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion::from_range(2..=2)]); @@ -1955,7 +1961,7 @@ async fn genuine_request_mismatch_still_reports_matching_deleted_region() { #[test] async fn request_matching_downloads_uncached_external_payloads() { let oplog = Arc::new(InMemoryOplog::new()); - oplog.add(noop()).await; + oplog.add(noop()).await.unwrap(); let first_request: HostRequest = HostRequestPollCount { count: 1 }.into(); let second_request: HostRequest = HostRequestPollCount { count: 2 }.into(); @@ -1973,7 +1979,8 @@ async fn request_matching_downloads_uncached_external_payloads() { request: Some(payload), durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); } let oplog: Arc = oplog; @@ -3536,7 +3543,7 @@ async fn marker_in_deleted_region_delivers_end_normally() { // the still-visible End must be delivered normally. let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), end_for(2, 42), discarded_for(2)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = @@ -3574,7 +3581,7 @@ async fn reverted_completion_marker_can_be_replaced_and_reconstructed() { for grow_target in [false, true] { let oplog: Arc = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), end_for(2, 42)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let dropped_region = OplogRegion { start: OplogIndex::from_u64(4), @@ -3591,7 +3598,7 @@ async fn reverted_completion_marker_can_be_replaced_and_reconstructed() { ]; if !grow_target { for entry in &suffix { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } } let rs = test_replay_state( @@ -3604,7 +3611,7 @@ async fn reverted_completion_marker_can_be_replaced_and_reconstructed() { .expect("a deleted marker must not conflict with its replacement"); if grow_target { for entry in suffix { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } rs.set_replay_target(OplogIndex::from_u64(6)) .await @@ -3656,7 +3663,7 @@ async fn delivered_marker_with_deleted_start_is_skipped_as_orphan() { delivered_for(2), noop(), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = @@ -3688,7 +3695,7 @@ async fn duplicate_completion_discarded_markers_fail_construction() { discarded_for(2), discarded_for(2), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let err = test_replay_state(test_agent_id(), oplog, DeletedRegions::default(), None) @@ -3710,7 +3717,7 @@ async fn conflicting_completion_markers_fail_construction() { delivered_for(2), discarded_for(2), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let err = test_replay_state(test_agent_id(), oplog, DeletedRegions::default(), None) @@ -3732,7 +3739,7 @@ async fn marker_recorded_at_runtime_is_visible_to_replay() { // already-recorded marker must be idempotent, not a duplicate-marker error. let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), end_for(2, 42)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let rs = test_replay_state( @@ -3743,7 +3750,7 @@ async fn marker_recorded_at_runtime_is_visible_to_replay() { ) .await .expect("failed to build replay state"); - let marker_idx = oplog.add(discarded_for(2)).await; + let marker_idx = oplog.add(discarded_for(2)).await.unwrap(); rs.record_discarded_completion(OplogIndex::from_u64(2), marker_idx); rs.set_replay_target(marker_idx) .await @@ -4756,7 +4763,7 @@ async fn entity_atomic_rollback_recovers_descendants_after_partial_jump_commit() OplogRegion::from_range(4..=4), ), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let rs = test_replay_state( test_agent_id(), @@ -4972,7 +4979,7 @@ async fn replay_finished_emitted_when_skipped_region_reaches_target() { // jumps the cursor over the deleted tail straight to the target (4). let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), log_entry(), log_entry()] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion { @@ -5539,7 +5546,7 @@ async fn orphan_end_with_deleted_start_is_skipped() { start_now(), end_for(4, 2), ] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion { @@ -5578,7 +5585,7 @@ async fn orphan_cancelled_with_deleted_start_is_skipped() { // [NoOp(1), Start(2), Cancelled(2→3)] with deleted region [2, 2]. let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), cancelled_for(2)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion { @@ -5605,7 +5612,7 @@ async fn positional_reader_skips_orphan_terminal() { // must consume the orphan End at 3 and return the NoOp at 4. let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), end_for(2, 1), noop()] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion { @@ -5629,7 +5636,7 @@ async fn deleted_terminal_reports_incomplete() { // [NoOp(1), Start(2), End(2→3)] with deleted region [3, 3]. let oplog = Arc::new(InMemoryOplog::new()); for entry in [noop(), start_now(), end_for(2, 1)] { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions([OplogRegion { @@ -5758,7 +5765,7 @@ async fn replay_skips_deleted_regions_fuzz() { let oplog = Arc::new(InMemoryOplog::new()); for entry in entries { - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } let oplog: Arc = oplog; let skipped = DeletedRegions::from_regions(regions.iter().map(|&(s, e)| OplogRegion { diff --git a/golem-worker-executor/src/durable_host/suspendable_wait.rs b/golem-worker-executor/src/durable_host/suspendable_wait.rs index 12ce9635f2..d5aa856fb9 100644 --- a/golem-worker-executor/src/durable_host/suspendable_wait.rs +++ b/golem-worker-executor/src/durable_host/suspendable_wait.rs @@ -418,7 +418,10 @@ mod tests { #[async_trait] impl Oplog for UnusedOplog { - async fn add(&self, _entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + _entry: OplogEntry, + ) -> Result { unreachable!("oplog is unused by promise waits") } @@ -430,7 +433,7 @@ mod tests { &self, _start: OplogEntry, _make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { unreachable!("oplog is unused by promise waits") } @@ -438,7 +441,10 @@ mod tests { unreachable!("oplog is unused by this test") } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { unreachable!("oplog is unused by this test") } @@ -486,14 +492,14 @@ mod tests { &self, _serialized_request: Vec, _build_start: Box Result + Send>, - ) -> Result { + ) -> Result { unreachable!("oplog is unused by this test") } async fn add_start_with_indexed_reserved_raw_payload( &self, _build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { unreachable!("oplog is unused by this test") } } @@ -670,7 +676,10 @@ mod tests { #[async_trait] impl Oplog for StubOplog { - async fn add(&self, _entry: OplogEntry) -> OplogIndex { + async fn add( + &self, + _entry: OplogEntry, + ) -> Result { unreachable!("oplog writes are unused by wakeup scheduling") } @@ -682,7 +691,7 @@ mod tests { &self, _start: OplogEntry, _make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { unreachable!("oplog writes are unused by wakeup scheduling") } @@ -690,7 +699,10 @@ mod tests { unreachable!("oplog is unused by this test") } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { unreachable!("oplog is unused by this test") } @@ -738,14 +750,14 @@ mod tests { &self, _serialized_request: Vec, _build_start: Box Result + Send>, - ) -> Result { + ) -> Result { unreachable!("oplog is unused by this test") } async fn add_start_with_indexed_reserved_raw_payload( &self, _build_request: crate::services::oplog::IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { unreachable!("oplog is unused by this test") } } diff --git a/golem-worker-executor/src/durable_host/tool/mcp.rs b/golem-worker-executor/src/durable_host/tool/mcp.rs index d98d961679..7fb2d3bbdb 100644 --- a/golem-worker-executor/src/durable_host/tool/mcp.rs +++ b/golem-worker-executor/src/durable_host/tool/mcp.rs @@ -132,7 +132,7 @@ async fn invoke_inner( ctx.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; let result = tokio::select! { biased; _ = async { @@ -144,7 +144,7 @@ async fn invoke_inner( let response = call.complete(ctx, HostResponseMcpToolCall { result: Err(SerializableToolRpcError::Cancelled), }).await?; - ctx.public_state.worker().commit_oplog_and_update_state(CommitLevel::DurableOnly).await; + ctx.public_state.worker().commit_oplog_and_update_state(CommitLevel::DurableOnly).await?; break 'response response; } result = live_call(ctx, &activation, &tool, &invocation.input, &auth, &key, &mut unauthorized_generation) => result, @@ -194,7 +194,7 @@ async fn invoke_inner( ctx.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; if let Some(generation) = unauthorized_generation { // Feedback never retries the tools/call whose response is already durable. let _ = ctx diff --git a/golem-worker-executor/src/durable_host/wasm_rpc/mod.rs b/golem-worker-executor/src/durable_host/wasm_rpc/mod.rs index c2c8087190..f89bb591f6 100644 --- a/golem-worker-executor/src/durable_host/wasm_rpc/mod.rs +++ b/golem-worker-executor/src/durable_host/wasm_rpc/mod.rs @@ -1564,7 +1564,7 @@ impl HostWasmRpc for DurableWorkerCtx { self.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; } let auth_ctx = handle.take_agent_auth_ctx(); @@ -2944,7 +2944,7 @@ async fn run_invoke_and_await( ctx.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; } let result = { @@ -3077,7 +3077,7 @@ async fn run_invoke( ctx.public_state .worker() .commit_oplog_and_update_state(CommitLevel::DurableOnly) - .await; + .await?; } let result = ctx diff --git a/golem-worker-executor/src/grpc/mod.rs b/golem-worker-executor/src/grpc/mod.rs index 13ec747c2d..cd7c9db309 100644 --- a/golem-worker-executor/src/grpc/mod.rs +++ b/golem-worker-executor/src/grpc/mod.rs @@ -33,11 +33,14 @@ use crate::services::worker_activator::{ }; use crate::services::worker_event::WorkerEventReceiver; use crate::services::{ - All, HasActiveAgents, HasAll, HasComponentService, HasConfig, HasEvents, HasOplogService, - HasPromiseService, HasRunningWorkerEnumerationService, HasShardManagerService, HasShardService, - HasWorkerEnumerationService, HasWorkerService, UsesAllDeps, + All, HasActiveAgents, HasAll, HasComponentService, HasConfig, HasEvents, HasOplog, + HasOplogService, HasPromiseService, HasRunningWorkerEnumerationService, HasShardManagerService, + HasShardService, HasWorkerEnumerationService, HasWorkerService, UsesAllDeps, +}; +use crate::worker::{ + ExportStreamControlResult as DomainExportResult, GiveUpReason, Worker, WorkerUpdateMode, + given_up_by_assignment, }; -use crate::worker::{ExportStreamControlResult as DomainExportResult, Worker, WorkerUpdateMode}; pub use crate::worker::{ PERMISSION_CARD_INSTALL_RECIPIENT_MISMATCH, PERMISSION_CARD_TRANSFER_PAYLOAD_CONFLICT, }; @@ -1233,35 +1236,42 @@ impl + UsesAllDeps + Send + Sync + let proto_shard_ids = request.shard_ids; let shard_ids = proto_shard_ids.into_iter().map(ShardId::from).collect(); - let revision = ShardLeaseRevision(request.revision); - - if let ShardDeliveryOutcome::Stale { delivered, applied } = - self.shard_service().revoke_shards(&shard_ids, revision)? - { - // A newer delivery has already been applied and its set is the - // authority; taking shards out of it would be acting on stale news. - tracing::warn!( - %delivered, - %applied, - "Ignoring a RevokeShards older than the last delivery applied" - ); - return Ok(()); - } - - for (agent_id, worker_details) in self.active_agents().snapshot().await { - if self.shard_service().check_worker(&agent_id).is_err() { - worker_details - .interrupt_and_retire(InterruptKind::Restart) - .await?; + let revision = ShardLeaseRevision::from_wire(&request.incarnation_id, request.revision); + + match self.shard_service().revoke_shards(&shard_ids, revision)? { + ShardDeliveryOutcome::Applied { .. } => {} + ShardDeliveryOutcome::Stale { delivered, applied } => { + // A newer delivery has already been applied and its set is the + // authority; taking shards out of it would be acting on stale news. + tracing::warn!( + %delivered, + %applied, + "Ignoring a RevokeShards older than the last delivery applied" + ); + return Ok(()); + } + ShardDeliveryOutcome::FromAnotherManager { delivered, applied } => { + self.renew_after_a_push_from_another_manager("RevokeShards", delivered, applied); + return Ok(()); } } + // Given up, not restarted: a restart in place would reopen each agent's oplog with the + // epoch this executor no longer holds. They are dropped from here and recovered by the + // shards' new owners. + let shard_service = self.shard_service(); + self.active_agents() + .give_up_matching(GiveUpReason::ShardRevoked, |agent_id| { + shard_service.check_worker(agent_id).is_err() + }) + .await; + Ok(()) } /// Full replace: the request carries this executor's complete shard set /// with epochs and the cluster's shard count. Anything absent from the - /// set is dropped, and any agent whose shard went away is restarted. + /// set is dropped, and any agent whose shard went away is given up. async fn assign_shards_internal( &self, request: golem::workerexecutor::v1::AssignShardsRequest, @@ -1282,28 +1292,55 @@ impl + UsesAllDeps + Send + Sync + )); } - let revision = ShardLeaseRevision(request.revision); - if let ShardDeliveryOutcome::Stale { delivered, applied } = self + let revision = ShardLeaseRevision::from_wire(&request.incarnation_id, request.revision); + match self .shard_service() .assign_shards(number_of_shards, &shard_epochs, revision)? { - // Crossed on the network with a newer delivery, which has already - // been applied; applying this one would put the older set back. - tracing::warn!( - %delivered, - %applied, - "Ignoring an AssignShards push older than the last delivery applied" - ); - return Ok(()); + ShardDeliveryOutcome::Applied { .. } => {} + ShardDeliveryOutcome::Stale { delivered, applied } => { + // Crossed on the network with a newer delivery, which has already + // been applied; applying this one would put the older set back. + tracing::warn!( + %delivered, + %applied, + "Ignoring an AssignShards push older than the last delivery applied" + ); + return Ok(()); + } + ShardDeliveryOutcome::FromAnotherManager { delivered, applied } => { + self.renew_after_a_push_from_another_manager("AssignShards", delivered, applied); + return Ok(()); + } } Self::apply_shard_assignment_effects(self).await?; Ok(()) } + /// A push from a shard manager process this executor does not follow is either a deposed + /// manager still sending, or a new one this executor has not heard a reply from yet. The + /// push cannot say which, so it is ignored either way and the renewal asks: its answer names + /// the process in charge and carries that process's set. + fn renew_after_a_push_from_another_manager( + &self, + push: &'static str, + delivered: ShardLeaseRevision, + applied: ShardLeaseRevision, + ) { + tracing::warn!( + push, + %delivered, + %applied, + "Ignoring a push from a shard manager process other than the one followed; renewing the lease to hear from the one in charge" + ); + self.shard_manager_service().renew_now(); + } + /// The one receipt path for a delivered shard set, whichever way it came: /// a registration, an `AssignShards` push, or a renewal reply that - /// corrected the set. Sweeps the agents whose shard went away, then hands + /// corrected the set. Sweeps the agents whose shard went away or came back + /// at a higher epoch, then hands /// the executor the new set to recover agents for. The sweep runs for /// every path, because a renewal can narrow the set as well as widen it: /// a path without it would leave agents running on shards this executor @@ -1325,16 +1362,37 @@ impl + UsesAllDeps + Send + Sync + T: HasAll + Send + Sync + 'static, { let ticket = this.shard_manager_service().recovery_deferred(); - - // Pure set membership on purpose: a lapsed lease must not restart every - // running agent: a lapsed lease refuses new work and leaves running work alone. - for (agent_id, worker_details) in this.active_agents().snapshot().await { - if this.shard_service().check_worker(&agent_id).is_err() { - worker_details - .interrupt_and_retire(InterruptKind::Restart) - .await?; - } - } + // Membership and epochs, never the lease: a lapsed lease must not give up every running + // agent - a lapsed lease refuses new work and leaves running work alone. + // + // Given up rather than restarted: a narrowing delivery means these shards have another + // owner now, and a restart in place would reopen their oplogs at the stale epoch. A + // delivery that raises the epoch of a shard this executor kept means the shard left and + // came back, so another executor may have written to its agents. Those are given up the + // same way, and the recovery below or their next invocation reopens them at the new epoch. + // + // The epochs come from one snapshot and the assignment from one read, both taken just + // before the sweep selects. An agent created after the snapshot read its epoch from the + // delivered assignment, so only membership applies to it. An agent given up and reopened + // at the new epoch between the snapshot and the selection is given up once more, which + // the same reopen repairs. + let held_epochs: HashMap> = this + .active_agents() + .snapshot() + .await + .into_iter() + .map(|(agent_id, worker)| (agent_id, worker.oplog().shard_epoch())) + .collect(); + let assignment = this.shard_service().try_get_current_assignment(); + this.active_agents() + .give_up_matching(GiveUpReason::ShardNotAssigned, |agent_id| { + given_up_by_assignment( + assignment.as_ref(), + agent_id, + held_epochs.get(agent_id).copied().flatten(), + ) + }) + .await; if !this.shard_service().is_ready() { tracing::info!( diff --git a/golem-worker-executor/src/lib.rs b/golem-worker-executor/src/lib.rs index 124843f8d7..ab3660fbbe 100644 --- a/golem-worker-executor/src/lib.rs +++ b/golem-worker-executor/src/lib.rs @@ -880,30 +880,24 @@ pub async fn create_worker_executor_impl< let sweep_archives = oplog_archives.clone(); let oplog_archives = NEVec::try_from_vec(oplog_archives); + // Built once for both shapes, so neither can be left without the observer: without it a + // refused write never reaches the shard manager, and a manager whose state lost history goes + // on minting below the oplog rows that refuse it. + let primary_oplog_service = PrimaryOplogService::new( + indexed_storage.clone(), + blob_storage.clone(), + golem_config.oplog.max_operations_before_commit, + golem_config.oplog.max_operations_before_commit_ephemeral, + golem_config.oplog.max_payload_size, + golem_config.indexed_storage_retry.clone(), + ) + .await + .with_fence_observer(shard_service.clone()); + let base_oplog_service: Arc = match oplog_archives { - None => Arc::new( - PrimaryOplogService::new( - indexed_storage.clone(), - blob_storage.clone(), - golem_config.oplog.max_operations_before_commit, - golem_config.oplog.max_operations_before_commit_ephemeral, - golem_config.oplog.max_payload_size, - golem_config.indexed_storage_retry.clone(), - ) - .await, - ), + None => Arc::new(primary_oplog_service), Some(oplog_archives) => { - let primary = Arc::new( - PrimaryOplogService::new( - indexed_storage.clone(), - blob_storage.clone(), - golem_config.oplog.max_operations_before_commit, - golem_config.oplog.max_operations_before_commit_ephemeral, - golem_config.oplog.max_payload_size, - golem_config.indexed_storage_retry.clone(), - ) - .await, - ); + let primary = Arc::new(primary_oplog_service); Arc::new(MultiLayerOplogService::new( primary, @@ -1128,7 +1122,11 @@ pub async fn create_worker_executor_impl< /// Derives a `DbSqliteConfig` for a module that should live in a separate /// SQLite DB file next to a base one (used by `KVStoreSqlite` to give the /// indexed storage its own DB and migration table). -fn derive_disjoint_sqlite_config(base: &DbSqliteConfig, suffix: &str) -> DbSqliteConfig { +/// +/// Public so test utilities can open the same file an executor uses without copying the naming +/// rule. +#[doc(hidden)] +pub fn derive_disjoint_sqlite_config(base: &DbSqliteConfig, suffix: &str) -> DbSqliteConfig { let database = match base.database.strip_suffix(".db") { Some(stem) => format!("{stem}-{suffix}.db"), None => format!("{}-{suffix}", base.database), diff --git a/golem-worker-executor/src/metrics.rs b/golem-worker-executor/src/metrics.rs index 57ec3af0f5..4f9a2264f4 100644 --- a/golem-worker-executor/src/metrics.rs +++ b/golem-worker-executor/src/metrics.rs @@ -1295,6 +1295,13 @@ pub mod oplog { &["op"] ) .unwrap(); + static ref OPLOG_EPOCH_FENCE_TOTAL: CounterVec = register_counter_vec!( + "oplog_epoch_fence_total", + "Oplog operations checked against the shard epoch: `op` is `record` for an open \ + recording its epoch and `append` for a write, `outcome` is `accepted` or `refused`", + &["op", "outcome"] + ) + .unwrap(); } pub fn record_oplog_call(api_name: &'static str) { @@ -1313,6 +1320,12 @@ pub mod oplog { .inc(); } + pub fn record_oplog_epoch_fence(op: &'static str, refused: bool) { + OPLOG_EPOCH_FENCE_TOTAL + .with_label_values(&[op, if refused { "refused" } else { "accepted" }]) + .inc(); + } + pub fn record_oplog_sweep_outcome(route: &str, outcome: &'static str, count: u64) { if count > 0 { OPLOG_SWEEP_OUTCOME_TOTAL diff --git a/golem-worker-executor/src/model/mod.rs b/golem-worker-executor/src/model/mod.rs index 4b1d1828e7..8e2137d79b 100644 --- a/golem-worker-executor/src/model/mod.rs +++ b/golem-worker-executor/src/model/mod.rs @@ -290,6 +290,22 @@ pub enum TrapType { } impl TrapType { + /// `ShardLost` once the agent's oplog has latched a fence, whatever the trap was. + /// + /// A latched oplog refuses every later write, so giving the agent up is the only outcome + /// left. It also catches a fence that crossed a `String` boundary on its way to the trap and + /// no longer classifies as `ShardLost` by itself. + pub fn under_latched_fence( + self, + latched: Option<&crate::services::oplog::OplogFence>, + ) -> TrapType { + if latched.is_some() { + TrapType::Interrupt(InterruptKind::ShardLost) + } else { + self + } + } + pub fn from_worker_executor_error( error: WorkerExecutorError, fallback_retry_from: OplogIndex, @@ -479,6 +495,13 @@ impl TrapType { Some(WorkerExecutorError::PermissionDenied { details }) => { make_error(AgentError::PermissionDenied(details.clone())) } + // Not a failure of the invocation: the storage refused the write + // because the shard has a new owner. Classified as an interrupt so + // the loop stops the agent without appending an `Error` entry to an + // oplog that is no longer this executor's to write. + Some(WorkerExecutorError::OplogFenced { .. }) => { + TrapType::Interrupt(InterruptKind::ShardLost) + } Some(WorkerExecutorError::ParamTypeMismatch { details }) => { make_error(AgentError::InvalidRequest(details.clone())) } @@ -497,7 +520,7 @@ impl TrapType { // // `WorkerExecutorError::Runtime` is intentionally NOT // mapped here: it is also used as a generic transient - // error wrapper (e.g. for `Oplog::fallible_add` + // error wrapper (e.g. for an `OplogError::Storage` // failures) and must remain retriable via the default // policy path (`AgentError::Unknown`). Some(WorkerExecutorError::UnexpectedOplogEntry { expected, got }) => { @@ -506,8 +529,19 @@ impl TrapType { ))) } _ => { - // Search the full error chain for ClassifiedHostError - if let Some(classified) = error + // A bare `?` on an oplog write inside an anyhow host function + // carries the `OplogError` itself, not its `WorkerExecutorError` + // form, so the fence is looked for along the chain as well. A + // storage error stays a retriable `Unknown`. After that, search + // the full error chain for ClassifiedHostError. + if error.chain().any(|cause| { + matches!( + cause.downcast_ref::(), + Some(crate::services::oplog::OplogError::Fenced(_)) + ) + }) { + TrapType::Interrupt(InterruptKind::ShardLost) + } else if let Some(classified) = error .chain() .find_map(|e| e.downcast_ref::()) { @@ -535,6 +569,10 @@ impl TrapType { TrapType::Interrupt(InterruptKind::Interrupt(_)) => Some(WorkerExecutorError::runtime( "Interrupted via the Golem API", )), + // What a caller can act on: refresh the routing table and retry on the owner. + TrapType::Interrupt(InterruptKind::ShardLost) => { + Some(WorkerExecutorError::ShardingNotReady) + } TrapType::Error { error, .. } => match error { AgentError::InvalidRequest(msg) => { Some(WorkerExecutorError::invalid_request(msg.clone())) @@ -903,6 +941,158 @@ mod tests { )); } + /// The contract every fenced host-call site depends on: a fence that escapes a host function + /// as an `anyhow` error must classify as `ShardLost`, so the loop gives the agent up instead + /// of appending an `Error` entry to the very oplog that refused the write. + #[test] + fn a_fenced_oplog_write_escaping_a_host_call_classifies_as_shard_lost() { + let fence = crate::services::oplog::OplogFence { + agent_id: golem_common::model::AgentId { + component_id: ComponentId::new(), + agent_id: "fenced-host-call".to_string(), + }, + expected_epoch: golem_common::model::ShardEpoch(7), + actual_epoch: Some(golem_common::model::ShardEpoch(8)), + writer_conflict: false, + }; + + let trap = TrapType::from_error::( + &anyhow::anyhow!(WorkerExecutorError::from( + crate::services::oplog::OplogError::Fenced(fence) + )), + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + + assert!( + matches!(trap, TrapType::Interrupt(InterruptKind::ShardLost)), + "a fenced write must be an interrupt, got {trap:?}" + ); + } + + /// The deliberate other half: a transient storage failure is not a fence and must stay a + /// retriable failure. Classifying it as `ShardLost` would hand an agent to another executor + /// over a blip that retrying would have cleared. + #[test] + fn a_transient_oplog_storage_failure_does_not_give_up_the_agent() { + let trap = TrapType::from_error::( + &anyhow::anyhow!(WorkerExecutorError::from( + crate::services::oplog::OplogError::Storage("connection reset".to_string()) + )), + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + + assert!( + !matches!(trap, TrapType::Interrupt(InterruptKind::ShardLost)), + "a transient storage failure must not be treated as a lost shard, got {trap:?}" + ); + } + + fn fence_for(agent_id: &str) -> crate::services::oplog::OplogFence { + crate::services::oplog::OplogFence { + agent_id: golem_common::model::AgentId { + component_id: ComponentId::new(), + agent_id: agent_id.to_string(), + }, + expected_epoch: golem_common::model::ShardEpoch(7), + actual_epoch: Some(golem_common::model::ShardEpoch(8)), + writer_conflict: false, + } + } + + /// The same contract for a host function that puts a bare `?` on an oplog write: the error is + /// the `OplogError` itself, possibly under context, and still has to read as a lost shard. + #[test] + fn a_bare_fenced_oplog_error_escaping_a_host_call_classifies_as_shard_lost() { + let bare = anyhow::Error::from(crate::services::oplog::OplogError::Fenced(fence_for( + "bare-fenced-host-call", + ))); + let with_context = anyhow::Error::from(crate::services::oplog::OplogError::Fenced( + fence_for("bare-fenced-host-call"), + )) + .context("ending atomic region"); + + for error in [bare, with_context] { + let trap = TrapType::from_error::( + &error, + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + assert!( + matches!(trap, TrapType::Interrupt(InterruptKind::ShardLost)), + "a bare fenced oplog error must be an interrupt, got {trap:?}" + ); + } + } + + #[test] + fn a_bare_oplog_storage_error_is_not_shard_lost() { + let trap = TrapType::from_error::( + &anyhow::Error::from(crate::services::oplog::OplogError::Storage( + "connection reset".to_string(), + )), + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + + assert!( + !matches!(trap, TrapType::Interrupt(InterruptKind::ShardLost)), + "a bare storage failure must stay a retriable failure, got {trap:?}" + ); + } + + /// Once the oplog has latched a fence nothing more can be written for the agent, so no trap + /// may lead to a retry, an exit record or a jump - only to giving the agent up. + #[test] + fn a_latched_fence_turns_every_trap_into_shard_lost() { + let fence = fence_for("latched"); + let unknown_error = || TrapType::Error { + error: AgentError::Unknown("fence flattened into text".to_string()), + retry_from: OplogIndex::INITIAL, + in_atomic_region: false, + atomic_region_had_side_effects: false, + semantic_trap_retry_override: None, + }; + + for trap in [ + unknown_error(), + TrapType::Exit, + TrapType::Interrupt(InterruptKind::Jump), + TrapType::Interrupt(InterruptKind::Suspend(Timestamp::now_utc())), + ] { + let reclassified = trap.clone().under_latched_fence(Some(&fence)); + assert!( + matches!(reclassified, TrapType::Interrupt(InterruptKind::ShardLost)), + "{trap:?} under a latched fence must be a lost shard, got {reclassified:?}" + ); + let decision = crate::durable_host::DurableWorkerCtx::< + crate::workerctx::default::Context, + >::fixed_decision_for_trap_type(&reclassified); + assert_eq!(decision, Some(RetryDecision::None)); + } + + assert!(matches!( + unknown_error().under_latched_fence(None), + TrapType::Error { + error: AgentError::Unknown(_), + .. + } + )); + assert!(matches!( + TrapType::Interrupt(InterruptKind::Jump).under_latched_fence(None), + TrapType::Interrupt(InterruptKind::Jump) + )); + } + #[test] fn semantic_trap_retry_override_carries_retry_point() { use crate::durable_host::durability::{ @@ -980,6 +1170,44 @@ mod tests { assert_eq!(decision, Some(RetryDecision::None)); } + #[test] + fn a_fenced_oplog_write_is_a_lost_shard_and_is_never_retried() { + let agent_id = AgentId { + component_id: golem_common::model::component::ComponentId::new(), + agent_id: "fenced".to_string(), + }; + let trap = TrapType::from_worker_executor_error::( + golem_service_base::error::worker_executor::WorkerExecutorError::oplog_fenced( + agent_id, + 3, + Some(4), + ), + OplogIndex::INITIAL, + false, + false, + AgentMode::Durable, + ); + + // An interrupt, not an error: no `Error` entry may be appended to an oplog that belongs + // to another executor now. + assert!(matches!( + trap, + TrapType::Interrupt(InterruptKind::ShardLost) + )); + + // Callers are told what they can act on, which is the same thing as for a lapsed lease. + assert!(matches!( + trap.as_golem_error(""), + Some(WorkerExecutorError::ShardingNotReady) + )); + + // And it is never retried in place - that would reopen the oplog at the stale epoch. + let decision = crate::durable_host::DurableWorkerCtx::< + crate::workerctx::default::Context, + >::fixed_decision_for_trap_type(&trap); + assert_eq!(decision, Some(RetryDecision::None)); + } + #[test] fn permission_denied_is_a_non_retriable_invocation_rejection() { let trap = TrapType::from_worker_executor_error::( @@ -1016,7 +1244,7 @@ mod tests { #[test] fn runtime_error_falls_back_to_unknown_and_is_policy_retriable() { // `WorkerExecutorError::Runtime` is a generic transient-error wrapper - // (used e.g. for `Oplog::fallible_add` failures). It must not be + // (used e.g. for `OplogError::Storage` failures). It must not be // classified as `InternalError` (non-retriable); it must fall through // to `AgentError::Unknown` so the configured retry policy applies. let trap = TrapType::from_error::( diff --git a/golem-worker-executor/src/model/public_oplog/tests.rs b/golem-worker-executor/src/model/public_oplog/tests.rs index b79a1a94fc..12851efa12 100644 --- a/golem-worker-executor/src/model/public_oplog/tests.rs +++ b/golem-worker-executor/src/model/public_oplog/tests.rs @@ -311,6 +311,7 @@ async fn public_oplog_zero_start_reads_from_initial_index() { make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let timestamp = Timestamp::now_utc(); @@ -320,10 +321,11 @@ async fn public_oplog_zero_start_reads_from_initial_index() { timestamp, entity_parent_start_index: None, }) - .await, + .await + .unwrap(), OplogIndex::INITIAL ); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let chunk = get_public_oplog_chunk( Arc::new(PanicComponentService), @@ -375,10 +377,11 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; - let agent_entry = oplog.add(OplogEntry::no_op(None)).await; + let agent_entry = oplog.add(OplogEntry::no_op(None)).await.unwrap(); let observational_owner = oplog .add(OplogEntry::Start { timestamp: Timestamp::now_utc(), @@ -389,7 +392,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: None, durable_function_type: DurableFunctionType::WriteLocal, }) - .await; + .await + .unwrap(); let middleware_entity = AgentEntity::ToolMiddleware(ToolMiddlewareName::try_from("audit").unwrap()); let middleware_input = "middleware-input" @@ -413,9 +417,10 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: Some(OplogPayload::Inline(Box::new(middleware_request))), durable_function_type: DurableFunctionType::WriteLocal, }) - .await; + .await + .unwrap(); - let interleaved_agent_entry = oplog.add(OplogEntry::no_op(None)).await; + let interleaved_agent_entry = oplog.add(OplogEntry::no_op(None)).await.unwrap(); let child_start = oplog .add(OplogEntry::Start { timestamp: Timestamp::now_utc(), @@ -426,7 +431,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: Some(OplogPayload::Inline(Box::new(HostRequestNoInput {}.into()))), durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); let tool_entity = AgentEntity::Tool(ToolName::try_from("lookup").unwrap()); let secret_id = Uuid::from_u128(1); @@ -475,7 +481,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: Some(OplogPayload::Inline(Box::new(tool_request))), durable_function_type: DurableFunctionType::WriteLocal, }) - .await; + .await + .unwrap(); let entity_retry_error = oplog .add(OplogEntry::error( Some(tool_start), @@ -485,8 +492,12 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { false, None, )) - .await; - let entity_marker = oplog.add(OplogEntry::no_op(Some(tool_start))).await; + .await + .unwrap(); + let entity_marker = oplog + .add(OplogEntry::no_op(Some(tool_start))) + .await + .unwrap(); let log_index = oplog .add(OplogEntry::Log { timestamp: Timestamp::now_utc(), @@ -495,7 +506,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { context: "tool".to_string(), message: "entity-attribution-needle".to_string(), }) - .await; + .await + .unwrap(); let span_id = SpanId::generate(); let span_index = oplog .add(OplogEntry::StartSpan { @@ -506,7 +518,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { linked_context_id: None, attributes: AttributeMap(HashMap::new()), }) - .await; + .await + .unwrap(); let stream_frame_index = oplog .add(OplogEntry::HostStreamFrame { timestamp: Timestamp::now_utc(), @@ -514,7 +527,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { kind: HostStreamKind::P3HttpRequestBody, payload: OplogPayload::Inline(Box::new(HostRequestNoInput {}.into())), }) - .await; + .await + .unwrap(); let reveal_secret_id = Uuid::from_u128(2); let reveal_request = HostRequestSecretReveal { @@ -564,7 +578,8 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: None, durable_function_type: DurableFunctionType::ReadLocal, }) - .await; + .await + .unwrap(); let observational_log = oplog .add(OplogEntry::Log { timestamp: Timestamp::now_utc(), @@ -573,10 +588,12 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { context: "custom".to_string(), message: "agent-owned observation".to_string(), }) - .await; + .await + .unwrap(); let observational_end = oplog .add(OplogEntry::end(observational_start, None, false)) - .await; + .await + .unwrap(); let transaction_start = oplog .add(OplogEntry::Start { @@ -588,24 +605,31 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: None, durable_function_type: DurableFunctionType::WriteRemoteTransaction(None), }) - .await; + .await + .unwrap(); let transaction_begin = oplog .add(OplogEntry::BeginRemoteTransaction { timestamp: Timestamp::now_utc(), transaction_id: TransactionId::new("entity-transaction".to_string()), original_begin_index: None, }) - .await; + .await + .unwrap(); let transaction_commit = oplog .add(OplogEntry::CommittedRemoteTransaction { timestamp: Timestamp::now_utc(), begin_index: transaction_start, }) - .await; + .await + .unwrap(); let transaction_end = oplog .add(OplogEntry::end(transaction_start, None, false)) - .await; - let child_end = oplog.add(OplogEntry::end(child_start, None, false)).await; + .await + .unwrap(); + let child_end = oplog + .add(OplogEntry::end(child_start, None, false)) + .await + .unwrap(); let tool_terminal = SerializableToolOperationTerminal { body_execution: SerializableEntityBodyExecution::Executed, result: Ok(SerializableToolStructuredResult { result: None }), @@ -622,10 +646,12 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { Some(OplogPayload::Inline(Box::new(tool_response))), false, )) - .await; + .await + .unwrap(); let completion = oplog .add(OplogEntry::completion_delivered(tool_start)) - .await; + .await + .unwrap(); let rejected_request: HostRequest = HostRequestGolemToolInvocationRejected { attempt_ordinal: 0, @@ -649,13 +675,16 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { request: Some(OplogPayload::Inline(Box::new(rejected_request))), durable_function_type: DurableFunctionType::WriteLocal, }) - .await; + .await + .unwrap(); let rejected_end = oplog .add(OplogEntry::end(rejected_start, None, false)) - .await; + .await + .unwrap(); let middleware_end = oplog .add(OplogEntry::end(middleware_start, None, false)) - .await; + .await + .unwrap(); let final_log = oplog .add(OplogEntry::Log { timestamp: Timestamp::now_utc(), @@ -664,8 +693,9 @@ async fn entity_attribution_is_nested_page_independent_and_order_preserving() { context: "tool".to_string(), message: "last-entity-attribution-needle".to_string(), }) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let components: Arc = Arc::new(PanicComponentService); let chunk = get_public_oplog_chunk( @@ -909,10 +939,11 @@ async fn explicit_entity_attribution_rejects_non_causal_and_non_entity_anchors() make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; - let non_start = oplog.add(OplogEntry::no_op(None)).await; + let non_start = oplog.add(OplogEntry::no_op(None)).await.unwrap(); let non_entity_start = oplog .add(OplogEntry::Start { timestamp: Timestamp::now_utc(), @@ -923,7 +954,8 @@ async fn explicit_entity_attribution_rejects_non_causal_and_non_entity_anchors() request: None, durable_function_type: DurableFunctionType::WriteLocal, }) - .await; + .await + .unwrap(); let entity_request = test_entity_request( &owned_agent_id, AgentEntity::Tool(ToolName::try_from("valid").unwrap()), @@ -941,16 +973,24 @@ async fn explicit_entity_attribution_rejects_non_causal_and_non_entity_anchors() request: Some(OplogPayload::Inline(Box::new(entity_request))), durable_function_type: DurableFunctionType::WriteLocal, }) - .await; - let valid = oplog.add(OplogEntry::no_op(Some(entity_start))).await; - let invalid_non_start = oplog.add(OplogEntry::no_op(Some(non_start))).await; - let invalid_non_entity = oplog.add(OplogEntry::no_op(Some(non_entity_start))).await; + .await + .unwrap(); + let valid = oplog + .add(OplogEntry::no_op(Some(entity_start))) + .await + .unwrap(); + let invalid_non_start = oplog.add(OplogEntry::no_op(Some(non_start))).await.unwrap(); + let invalid_non_entity = oplog + .add(OplogEntry::no_op(Some(non_entity_start))) + .await + .unwrap(); let invalid_forward_index = invalid_non_entity.next(); let future_entity_start = invalid_forward_index.next(); assert_eq!( oplog .add(OplogEntry::no_op(Some(future_entity_start))) - .await, + .await + .unwrap(), invalid_forward_index ); assert_eq!( @@ -964,10 +1004,11 @@ async fn explicit_entity_attribution_rejects_non_causal_and_non_entity_anchors() request: None, durable_function_type: DurableFunctionType::WriteLocal, }) - .await, + .await + .unwrap(), future_entity_start ); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let components: Arc = Arc::new(PanicComponentService); let valid_chunk = get_public_oplog_chunk( @@ -1053,6 +1094,7 @@ async fn p3_payloads_render_through_public_oplog_api_and_wit() { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -1308,7 +1350,8 @@ async fn p3_payloads_render_through_public_oplog_api_and_wit() { request: Some(cancelled_request_payload), durable_function_type: DurableFunctionType::WriteRemote, }) - .await; + .await + .unwrap(); expected_starts.insert( cancelled_start_index, ( @@ -1331,8 +1374,9 @@ async fn p3_payloads_render_through_public_oplog_api_and_wit() { cancelled_start_index, Some(partial_payload), )) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let last_index = oplog_service .get_last_index(&owned_agent_id, AgentMode::Durable) diff --git a/golem-worker-executor/src/services/active_agents/memory_probe.rs b/golem-worker-executor/src/services/active_agents/memory_probe.rs index 6b150d4d68..42ae368846 100644 --- a/golem-worker-executor/src/services/active_agents/memory_probe.rs +++ b/golem-worker-executor/src/services/active_agents/memory_probe.rs @@ -383,6 +383,12 @@ mod tests { } } + /// These tests wait for a refresh that runs on `spawn_blocking`, so what they are really + /// bounded by is a blocking-pool slot becoming free, not the refresh being quick. Under the + /// full parallel lib suite that can take seconds. The deadline exists to fail rather than + /// hang; no assertion depends on its value. + const REFRESH_COMPLETION_TIMEOUT: Duration = Duration::from_secs(30); + #[derive(Debug)] struct PanickingProbe { reads: Arc, @@ -437,7 +443,7 @@ mod tests { current_bytes.store(42, Ordering::Relaxed); assert_eq!(probe.snapshot().current_bytes, 1); - tokio::time::timeout(Duration::from_secs(1), async { + tokio::time::timeout(REFRESH_COMPLETION_TIMEOUT, async { while reads.load(Ordering::Acquire) != 2 { tokio::task::yield_now().await; } @@ -455,7 +461,7 @@ mod tests { *refresh_gate.0.lock().unwrap() = true; refresh_gate.1.notify_one(); - tokio::time::timeout(Duration::from_secs(1), async { + tokio::time::timeout(REFRESH_COMPLETION_TIMEOUT, async { while probe.snapshot().limit_bytes != 50 || probe.snapshot().current_bytes != 42 { tokio::task::yield_now().await; } @@ -485,7 +491,7 @@ mod tests { tokio::time::sleep(refresh_interval).await; assert_eq!(probe.snapshot().current_bytes, 1); - tokio::time::timeout(Duration::from_secs(1), async { + tokio::time::timeout(REFRESH_COMPLETION_TIMEOUT, async { while reads.load(Ordering::Acquire) != 2 { tokio::task::yield_now().await; } @@ -494,7 +500,7 @@ mod tests { .unwrap(); tokio::time::sleep(refresh_interval).await; - tokio::time::timeout(Duration::from_secs(1), async { + tokio::time::timeout(REFRESH_COMPLETION_TIMEOUT, async { while probe.snapshot().current_bytes != 42 { tokio::task::yield_now().await; } @@ -519,7 +525,7 @@ mod tests { ); assert_eq!(probe.snapshot().current_bytes, 1); - tokio::time::timeout(Duration::from_secs(1), async { + tokio::time::timeout(REFRESH_COMPLETION_TIMEOUT, async { while probe.inner.refresh_in_progress.load(Ordering::Acquire) { tokio::task::yield_now().await; } diff --git a/golem-worker-executor/src/services/active_agents/mod.rs b/golem-worker-executor/src/services/active_agents/mod.rs index 3d4cff8a73..9514500492 100644 --- a/golem-worker-executor/src/services/active_agents/mod.rs +++ b/golem-worker-executor/src/services/active_agents/mod.rs @@ -38,7 +38,7 @@ use tokio::task::JoinHandle; use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; -use tracing::{Instrument, debug}; +use tracing::{Instrument, debug, info}; use crate::durable_host::tool::operation::OwnerFailureWinner; use crate::services::HasAll; @@ -63,7 +63,7 @@ use crate::worker::instance::{ use crate::worker::owner_lane::{EntityCallMode, OwnerInvocationId, OwnerInvocationTicket}; use crate::worker::status_flusher::AgentStatusFlushQueue; use crate::worker::{ - EvictionClass, EvictionStopOutcome, FilesystemPressureEligibility, UnloadRequest, + EvictionClass, EvictionStopOutcome, FilesystemPressureEligibility, GiveUpReason, UnloadRequest, }; use crate::worker::{Worker, WorkerCreationMode}; use crate::workerctx::WorkerCtx; @@ -1117,6 +1117,23 @@ impl ActiveAgents { /// Removes only the cache generation owned by `expected`. Bookkeeping is cleared only when /// that exact generation was still authoritative at the point of removal. pub async fn remove_worker(&self, expected: &Arc>, deletion_owner: bool) -> bool { + self.remove_worker_with( + expected, + deletion_owner, + OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt(Timestamp::now_utc())), + ) + .await + } + + /// [`Self::remove_worker`] with an explicit reason for tearing the agent's entity bodies down. + /// A given-up agent must not report itself as interrupted through the Golem API: it was + /// not, its shard moved. A deletion owner tears nothing down here, whatever the reason. + pub(crate) async fn remove_worker_with( + &self, + expected: &Arc>, + deletion_owner: bool, + owner_failure: OwnerFailureWinner, + ) -> bool { let owned_agent_id = expected.owned_agent_id().clone(); let Some(active_agent) = self.agents.get(&owned_agent_id).await else { return false; @@ -1132,11 +1149,7 @@ impl ActiveAgents { return false; }; if !deletion_owner { - active_agent - .fence_entity_bodies(OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt( - Timestamp::now_utc(), - ))) - .await; + active_agent.fence_entity_bodies(owner_failure).await; } let expected_active = active_agent.clone(); let expected_worker = expected.clone(); @@ -1158,6 +1171,71 @@ impl ActiveAgents { removed } + /// The worker cached for `owned_agent_id`, without waiting on a creation still in progress. + /// + /// For callers acting on one particular generation: a pending or still-unresolved entry is a + /// newer generation being created, never the one they hold, so waiting on it could only delay + /// them. + pub(crate) async fn try_get_cached( + &self, + owned_agent_id: &OwnedAgentId, + ) -> Option>> { + self.agents + .try_get(owned_agent_id) + .await + .and_then(|active_agent| active_agent.resolved_primary()) + } + + /// Whether `worker` is the generation cached for its agent right now. + pub(crate) async fn is_cached_generation(&self, worker: &Worker) -> bool { + self.try_get_cached(worker.owned_agent_id()) + .await + .is_some_and(|cached| std::ptr::eq(Arc::as_ptr(&cached), worker)) + } + + /// [`Self::remove_worker_with`] for a caller holding the generation by reference: tears the + /// entry down and drops it only while it still holds `worker`. Returns whether it did. + /// + /// A given-up agent reaches its removal more than once - from its own loop's stop, again + /// from the give-up that waited for it, or from a stop through a handle kept past its + /// generation - and by then a newer generation may be cached under the same id. Keyed by id + /// alone, such a pass evicts that generation and fences its entity bodies while its loop keeps + /// running. + /// + /// A removal refused while this generation is still cached is retried, unless a deletion owns + /// its retirement and removes it itself. The only other refusal is the retirement marker held + /// by a concurrent attempt - an idle expiry, or another pass of this removal - which ends with + /// the generation removed or the marker rolled back. Without the retry, an agent given up + /// while an idle expiry happened to be checking it would stay cached here, and a later + /// re-grant of its shard would find this given-up generation instead of opening the oplog at + /// the new epoch. + pub(crate) async fn remove_generation( + &self, + worker: &Worker, + owner_failure: OwnerFailureWinner, + ) -> bool { + loop { + let Some(cached) = self.try_get_cached(worker.owned_agent_id()).await else { + return false; + }; + if !std::ptr::eq(Arc::as_ptr(&cached), worker) { + return false; + } + if self + .remove_worker_with(&cached, false, owner_failure.clone()) + .await + { + return true; + } + if cached.deletion_owns_retirement().await { + return false; + } + drop(cached); + // The concurrent attempt may be draining entity bodies; poll rather than spin. + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + pub async fn tracked_card_ids(&self) -> Vec { self.card_interest_index.tracked_card_ids().await } @@ -1201,7 +1279,8 @@ impl ActiveAgents { if scope.bind(&worker.tasks).is_err() { continue; } - scope + // A refusal has already given the agent up; the other agents are still notified. + let _ = scope .run(worker.queue_card_revocations(&affected_card_ids)) .await; } @@ -1220,6 +1299,48 @@ impl ActiveAgents { .collect() } + /// Gives up every agent the predicate selects: stops each one here and drops it from this + /// executor, so the shard's new owner recovers it. + /// + /// Concurrent rather than sequential, unlike [`Self::unload_environment`]: a revoke can name + /// many agents and each stop waits for that agent's invocation loop to exit. No acknowledgement + /// channel is awaited either - [`Worker::give_up`] never subscribes to one - so an agent + /// that is already stopping cannot panic the sweep, which is what the old + /// `set_interrupting(..).recv().await.unwrap()` shape risked. + /// + /// The snapshot includes suspended, loading and already-stopping agents; the stop state + /// machine has an arm for each, so none is skipped. An agent still being resolved is not in + /// it: one that read the assignment before the shard left opens its oplog at the epoch it + /// was granted, and checks the assignment again once it is published - see + /// `Worker::give_up_if_shard_left_during_construction`. + pub(crate) async fn give_up_matching( + &self, + reason: GiveUpReason, + select: impl Fn(&AgentId) -> bool, + ) { + let selected: Vec>> = self + .snapshot() + .await + .into_iter() + .filter(|(agent_id, _)| select(agent_id)) + .map(|(_, worker)| worker) + .collect(); + + if !selected.is_empty() { + info!( + ?reason, + agents = selected.len(), + "Giving up agents whose shard has moved" + ); + } + + futures::future::join_all(selected.into_iter().map(|worker| { + let reason = reason.clone(); + async move { worker.give_up(reason).await } + })) + .await; + } + /// Interrupts and unloads all in-memory workers whose environment matches /// `environment_id`. Called when the environment is deleted so that /// running workers stop promptly. diff --git a/golem-worker-executor/src/services/golem_config.rs b/golem-worker-executor/src/services/golem_config.rs index 954176563b..1fdebd0cad 100644 --- a/golem-worker-executor/src/services/golem_config.rs +++ b/golem-worker-executor/src/services/golem_config.rs @@ -2151,7 +2151,11 @@ impl KeyValueStorageConfig { impl Default for IndexedStorageConfig { fn default() -> Self { - Self::KVStoreRedis(IndexedStorageKVStoreRedisConfig {}) + Self::Sqlite(DbSqliteConfig { + database: "../data/worker-executor-indexed-storage.db".to_string(), + max_connections: 10, + foreign_keys: false, + }) } } diff --git a/golem-worker-executor/src/services/oplog/compressed.rs b/golem-worker-executor/src/services/oplog/compressed.rs index 55daed216d..d093079a2f 100644 --- a/golem-worker-executor/src/services/oplog/compressed.rs +++ b/golem-worker-executor/src/services/oplog/compressed.rs @@ -80,6 +80,92 @@ where } } +/// Appends one already-serialized compressed chunk, retrying transient failures like +/// [`retry_storage_op`]. A permanent failure is reconciled against storage before it is treated +/// as fatal, because this append is not protected by a shard epoch (it is driven by whichever +/// executor's transfer fiber is running, primary-owner or not) and its background task can be +/// aborted between steps - including after this append lands but before the `drop_source_prefix` +/// that would have advanced the source past it. The owner's next transfer then chunks from the +/// same unadvanced point, so a chunk it writes under an id that already exists holds the identical +/// entries and bytes. A duplicate-id failure whose stored content matches what this attempt would +/// have written is that resumed transfer catching up, not corruption, and is treated as success +/// rather than panicking on the storage's key conflict. +async fn append_compressed_chunk( + retry_config: &RetryConfig, + indexed_storage: &(dyn IndexedStorage + Send + Sync), + namespace: &IndexedStorageNamespace, + key: &str, + id: u64, + value: Vec, +) { + let mut attempts = 0u32; + loop { + attempts += 1; + let error = match indexed_storage + .with_entity("compressed_oplog", "append", "compressed_entry") + .append_raw(namespace.clone(), key, id, value.clone(), None) + .await + { + Ok(()) => return, + Err(error) => error, + }; + + if let IndexedStorageError::Transient(msg) = &error { + if let Some(delay) = get_delay(retry_config, attempts) { + record_oplog_storage_retry("compressed_append"); + warn!( + op = "compressed_append", + key = key, + attempt = attempts, + delay_ms = delay.as_millis() as u64, + "Transient indexed storage error, retrying: {msg}" + ); + tokio::time::sleep(delay).await; + continue; + } + panic!( + "Indexed storage operation 'compressed_append' failed for key '{key}' after {attempts} attempts: Transient storage error: {msg}" + ); + } + + if stored_chunk_matches(retry_config, indexed_storage, namespace, key, id, &value).await { + return; + } + panic!("Indexed storage operation 'compressed_append' failed for key '{key}': {error}"); + } +} + +/// Reads back the chunk stored at `id` and compares it byte-for-byte with `expected`. Used only to +/// tell a resumed transfer's harmless repeat write apart from a genuine conflict - see +/// [`append_compressed_chunk`]. +async fn stored_chunk_matches( + retry_config: &RetryConfig, + indexed_storage: &(dyn IndexedStorage + Send + Sync), + namespace: &IndexedStorageNamespace, + key: &str, + id: u64, + expected: &[u8], +) -> bool { + let actual = retry_storage_op(retry_config, "compressed_append_reconcile", key, || { + let namespace = namespace.clone(); + async move { + indexed_storage + .with_entity( + "compressed_oplog", + "compressed_append_reconcile", + "compressed_entry", + ) + .read_raw(namespace, key, id, id) + .await + } + }) + .await; + actual + .into_iter() + .find(|(actual_id, _)| *actual_id == id) + .is_some_and(|(_, bytes)| bytes == expected) +} + #[derive(Debug)] pub struct CompressedOplogArchiveService { indexed_storage: Arc, @@ -497,27 +583,22 @@ impl OplogArchive for CompressedOplogArchive { total_bytes += compressed_chunk.compressed_data.len() as u64; { - let is = self.indexed_storage.clone(); - let agent_id_clone = self.agent_id.clone(); - let agent_mode = self.agent_mode; - let level = self.level; - let key = self.key.clone(); + let ns = IndexedStorageNamespace::CompressedOpLog { + agent_id: self.agent_id.clone(), + agent_mode: self.agent_mode, + level: self.level, + }; let last_id_val: u64 = last_id.into(); - retry_storage_op(&self.retry_config, "compressed_append", &key, || { - let is = is.clone(); - let ns = IndexedStorageNamespace::CompressedOpLog { - agent_id: agent_id_clone.clone(), - agent_mode, - level, - }; - let key = key.clone(); - let chunk = compressed_chunk.clone(); - async move { - is.with_entity("compressed_oplog", "append", "compressed_entry") - .append(ns, &key, last_id_val, &chunk) - .await - } - }) + let value = serialize(&compressed_chunk) + .unwrap_or_else(|err| panic!("failed to serialize oplog chunk: {err}")); + append_compressed_chunk( + &self.retry_config, + self.indexed_storage.as_ref(), + &ns, + &self.key, + last_id_val, + value, + ) .await; } } diff --git a/golem-worker-executor/src/services/oplog/ephemeral.rs b/golem-worker-executor/src/services/oplog/ephemeral.rs index 8a2031a189..0dd9f5dce0 100644 --- a/golem-worker-executor/src/services/oplog/ephemeral.rs +++ b/golem-worker-executor/src/services/oplog/ephemeral.rs @@ -22,8 +22,8 @@ use crate::services::oplog::reader::{ }; use crate::services::oplog::{ CommitLevel, DurableStreamBatchBuilder, IndexedReservedStartBuilder, Oplog, OplogAddReceipt, - OplogCloseCompletion, OplogService, OrderedOplogStart, PendingUpload, ReservedRawStartBuilder, - downcast_oplog, + OplogCloseCompletion, OplogError, OplogService, OrderedOplogStart, PendingUpload, + ReservedRawStartBuilder, downcast_oplog, }; use async_trait::async_trait; use futures::FutureExt; @@ -800,18 +800,18 @@ impl Oplog for EphemeralOplog { } let owned_agent_id = self.owned_agent_id.clone(); Box::pin(async move { - done_rx.await.unwrap_or_else(|_| { + Ok(done_rx.await.unwrap_or_else(|_| { panic!( "Ephemeral oplog actor for {owned_agent_id:?} dropped an add request without replying" ) - }) + })) }) } async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { record_oplog_call("add_durable_stream_batch"); Ok(self .run_job(|done| EphemeralJob::AddDurableStreamBatch { make_batch, done }) @@ -822,21 +822,22 @@ impl Oplog for EphemeralOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { record_oplog_call("add_pair"); - self.run_job(|done| EphemeralJob::AddPair { - start, - make_second, - done, - }) - .await + Ok(self + .run_job(|done| EphemeralJob::AddPair { + start, + make_second, + done, + }) + .await) } async fn add_start_with_reserved_raw_payload( &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { record_oplog_call("add_start_with_reserved_raw_payload"); // Ephemeral oplogs are never replayed, so cross-call `Start` ordering need not be // deterministic and there is no deferred-upload/commit-barrier machinery here. Upload the @@ -860,13 +861,14 @@ impl Oplog for EphemeralOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { record_oplog_call("add_start_with_indexed_reserved_raw_payload"); self.run_job(|done| EphemeralJob::AddIndexedStart { build_request, done, }) .await + .map_err(OplogError::from) } async fn drop_prefix(&self, last_dropped_id: OplogIndex) -> u64 { @@ -878,24 +880,25 @@ impl Oplog for EphemeralOplog { dropped } - async fn commit(&self, level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError> { record_oplog_call("commit"); match level { - CommitLevel::Always => { - self.run_job(|done| EphemeralJob::Commit { + CommitLevel::Always => Ok(self + .run_job(|done| EphemeralJob::Commit { wait_for_storage: true, done, }) - .await - } - CommitLevel::Deferred => { - self.run_job(|done| EphemeralJob::Commit { + .await), + CommitLevel::Deferred => Ok(self + .run_job(|done| EphemeralJob::Commit { wait_for_storage: false, done, }) - .await - } - CommitLevel::DurableOnly => BTreeMap::new(), + .await), + CommitLevel::DurableOnly => Ok(BTreeMap::new()), } } diff --git a/golem-worker-executor/src/services/oplog/ephemeral/tests.rs b/golem-worker-executor/src/services/oplog/ephemeral/tests.rs index 9f31f87655..7abb83a2e1 100644 --- a/golem-worker-executor/src/services/oplog/ephemeral/tests.rs +++ b/golem-worker-executor/src/services/oplog/ephemeral/tests.rs @@ -241,24 +241,30 @@ fn entry(n: usize) -> OplogEntry { #[timeout("30s")] async fn threshold_flush_does_not_block_add_and_deferred_returns_receipt() { let fixture = fixture(1).await; - assert_eq!(fixture.oplog.add(entry(0)).await, OplogIndex::INITIAL); - assert_eq!(fixture.oplog.add(entry(1)).await, OplogIndex::from_u64(2)); + assert_eq!( + fixture.oplog.add(entry(0)).await.unwrap(), + OplogIndex::INITIAL + ); + assert_eq!( + fixture.oplog.add(entry(1)).await.unwrap(), + OplogIndex::from_u64(2) + ); fixture.archive.wait_for_appends(1).await; - let receipt = fixture.oplog.commit(CommitLevel::Deferred).await; + let receipt = fixture.oplog.commit(CommitLevel::Deferred).await.unwrap(); assert_eq!( receipt.keys().copied().collect::>(), vec![OplogIndex::INITIAL, OplogIndex::from_u64(2)] ); fixture.archive.release(1); - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); } #[test] #[timeout("30s")] async fn always_commit_waits_for_prior_write_even_with_empty_residual_batch() { let fixture = fixture(0).await; - fixture.oplog.add(entry(0)).await; + fixture.oplog.add(entry(0)).await.unwrap(); fixture.archive.wait_for_appends(1).await; let commit = fixture.oplog.commit(CommitLevel::Always); tokio::pin!(commit); @@ -268,7 +274,7 @@ async fn always_commit_waits_for_prior_write_even_with_empty_residual_batch() { .is_err() ); fixture.archive.release(1); - let receipt = commit.await; + let receipt = commit.await.unwrap(); assert_eq!(receipt.len(), 1); assert_eq!(fixture.archive.append_calls.load(Ordering::Relaxed), 1); } @@ -280,11 +286,11 @@ async fn read_exact_combines_persisted_handed_off_and_buffer_without_reading_buf let expected: Vec<_> = (0..5).map(entry).collect(); fixture.archive.release(1); for entry in &expected[..2] { - fixture.oplog.add(entry.clone()).await; + fixture.oplog.add(entry.clone()).await.unwrap(); } - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); for entry in &expected[2..] { - fixture.oplog.add(entry.clone()).await; + fixture.oplog.add(entry.clone()).await.unwrap(); } fixture.archive.wait_for_appends(2).await; @@ -312,7 +318,7 @@ async fn read_exact_combines_persisted_handed_off_and_buffer_without_reading_buf expected[2..] ); fixture.archive.release(2); - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); } #[test] @@ -320,7 +326,7 @@ async fn read_exact_combines_persisted_handed_off_and_buffer_without_reading_buf async fn bounded_writer_queue_backpressures_fourth_threshold_flush_and_close_drains() { let fixture = fixture(0).await; for n in 0..3 { - fixture.oplog.add(entry(n)).await; + fixture.oplog.add(entry(n)).await.unwrap(); } fixture.archive.wait_for_appends(1).await; @@ -332,7 +338,7 @@ async fn bounded_writer_queue_backpressures_fourth_threshold_flush_and_close_dra .is_err() ); fixture.archive.release(1); - assert_eq!(fourth.await, OplogIndex::from_u64(4)); + assert_eq!(fourth.await.unwrap(), OplogIndex::from_u64(4)); let closed = fixture.oplog.closed(); fixture.oplog.retire(); @@ -349,16 +355,16 @@ async fn receipt_overflow_retains_a_detectable_gap_and_storage_barrier_covers_it fixture.archive.release(count); let expected: Vec<_> = (0..count).map(entry).collect(); for entry in &expected { - fixture.oplog.add(entry.clone()).await; + fixture.oplog.add(entry.clone()).await.unwrap(); } - let receipts = fixture.oplog.commit(CommitLevel::Deferred).await; + let receipts = fixture.oplog.commit(CommitLevel::Deferred).await.unwrap(); assert_eq!(receipts.len(), MAX_RETAINED_RECEIPT_BATCHES); assert_eq!(*receipts.first_key_value().unwrap().0, OplogIndex::INITIAL); assert_eq!(receipts.last_key_value().unwrap().0.as_u64(), count as u64); assert!(!receipts.contains_key(&OplogIndex::from_u64(2))); assert_eq!(fixture.archive.read_calls.load(Ordering::Relaxed), 0); - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( fixture .archive @@ -376,7 +382,7 @@ async fn receipt_overflow_retains_a_detectable_gap_and_storage_barrier_covers_it #[timeout("30s")] async fn failed_writer_is_joined_and_reported_by_close() { let fixture = fixture(0).await; - fixture.oplog.add(entry(0)).await; + fixture.oplog.add(entry(0)).await.unwrap(); fixture.archive.wait_for_appends(1).await; fixture.archive.append_permits.close(); fixture.oplog.retire(); diff --git a/golem-worker-executor/src/services/oplog/mod.rs b/golem-worker-executor/src/services/oplog/mod.rs index 34b4d8f720..1013d9751a 100644 --- a/golem-worker-executor/src/services/oplog/mod.rs +++ b/golem-worker-executor/src/services/oplog/mod.rs @@ -40,7 +40,7 @@ use golem_common::model::oplog::{ }; use golem_common::model::{ AgentId, AgentInvocation, AgentInvocationResult, AgentMetadata, AgentStatusRecord, - DurableStreamSessionStatus, OwnedAgentId, ScanCursor, Timestamp, + DurableStreamSessionStatus, OwnedAgentId, ScanCursor, ShardEpoch, Timestamp, }; use golem_common::read_only_lock; use golem_common::retries::get_delay; @@ -53,7 +53,7 @@ pub use multilayer::{MultiLayerOplog, MultiLayerOplogService, OplogArchive, Oplo pub use primary::PrimaryOplogService; use std::any::{Any, TypeId}; use std::collections::BTreeMap; -use std::fmt::{Debug, Formatter}; +use std::fmt::{Debug, Display, Formatter}; use std::marker::PhantomData; use std::ops::Deref; use std::sync::{Arc, Weak}; @@ -167,6 +167,7 @@ pub trait OplogService: Debug + Send + Sync { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc; /// Creates an oplog whose absence has already been established by the caller. @@ -183,6 +184,7 @@ pub trait OplogService: Debug + Send + Sync { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc; /// Opens an existing oplog for the given worker. @@ -203,6 +205,7 @@ pub trait OplogService: Debug + Send + Sync { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc; async fn get_last_index( @@ -211,12 +214,17 @@ pub trait OplogService: Debug + Send + Sync { agent_mode: AgentMode, ) -> OplogIndex; + /// Deletes the agent's oplog, in every layer. With `expected_epoch` - the epoch the caller's + /// own handle asserts - only while that is still the epoch recorded for the oplog and this + /// executor recorded it: otherwise nothing is deleted and the delete is refused with + /// [`OplogError::Fenced`], as a write at that epoch would be. `None` deletes unconditionally. async fn delete( &self, lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, - ); + expected_epoch: Option, + ) -> Result<(), OplogError>; /// Reads exactly `n` contiguous entries starting at `idx`. async fn read_exact( @@ -601,10 +609,86 @@ pub type ReservedRawStartBuilder = pub type IndexedReservedStartBuilder = Box Result<(Vec, ReservedRawStartBuilder), String> + Send>; +/// Why an oplog write was refused by the storage: the shard epoch this executor asserted is +/// behind the one recorded for the oplog, because another executor owns the shard now. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct OplogFence { + pub agent_id: AgentId, + pub expected_epoch: ShardEpoch, + pub actual_epoch: Option, + /// The stored epoch is the one this executor asserted, but another process recorded it. The + /// epoch alone therefore says nothing about who may write, and the shard manager has to mint + /// past it rather than leave two holders on one generation. + pub writer_conflict: bool, +} + +/// Told of every refusal the storage returns, carrying the epoch recorded on the oplog. +/// +/// That epoch is evidence of a generation somebody held for the agent's shard, which a shard +/// manager whose state lost history no longer knows about. The same refusal can be reported more +/// than once - a refused create, and then the refused open behind it - so an observer merges what +/// it is told rather than counting it. +pub trait OplogFenceObserver: Send + Sync { + fn fenced(&self, fence: &OplogFence); +} + +/// The one way an oplog write can fail without taking the executor down. +/// +/// A `Fenced` write is not a storage failure - the storage is healthy and refused the write on +/// purpose - so it is returned rather than retried or panicked on, and the worker that hit it is +/// stopped and left to the shard's new owner. Every other storage failure keeps its fail-stop +/// semantics inside the oplog implementation; `Storage` exists so that test doubles and payload +/// helpers that already return a `String` can flow through the same `Result` without a second +/// error type at every call site. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum OplogError { + Fenced(OplogFence), + Storage(String), +} + +impl From for OplogError { + fn from(details: String) -> Self { + OplogError::Storage(details) + } +} + +impl From for WorkerExecutorError { + fn from(error: OplogError) -> Self { + match error { + OplogError::Fenced(fence) => WorkerExecutorError::oplog_fenced( + fence.agent_id, + fence.expected_epoch.0, + fence.actual_epoch.map(|epoch| epoch.0), + ), + OplogError::Storage(details) => WorkerExecutorError::runtime(details), + } + } +} + +impl Display for OplogError { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + match self { + OplogError::Fenced(fence) => write!( + f, + "oplog write for {} fenced: asserted shard epoch {}, stored {}", + fence.agent_id, + fence.expected_epoch, + fence + .actual_epoch + .map(|epoch| epoch.to_string()) + .unwrap_or_else(|| "none".to_string()) + ), + OplogError::Storage(details) => write!(f, "oplog storage error: {details}"), + } + } +} + +impl std::error::Error for OplogError {} + /// A single oplog append that has already been synchronously enqueued in the oplog's ordering /// domain. Creating this receipt reserves the entry's position; awaiting it returns the assigned /// index after the append finishes. -pub type OplogAddReceipt = BoxFuture<'static, OplogIndex>; +pub type OplogAddReceipt = BoxFuture<'static, Result>; #[derive(Clone, Debug, PartialEq, Eq)] pub struct RawDurableStreamSessionStatus { @@ -657,7 +741,7 @@ pub trait Oplog: Any + Debug + Send + Sync { } /// Adds a single entry to the oplog (possibly buffered), and returns its index - async fn add(&self, entry: OplogEntry) -> OplogIndex { + async fn add(&self, entry: OplogEntry) -> Result { self.enqueue_add(entry).await } @@ -677,7 +761,7 @@ pub trait Oplog: Any + Debug + Send + Sync { async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { let first_index = self.current_oplog_index().await.next(); let records = make_batch(first_index); let mut result = Vec::with_capacity(records.len()); @@ -688,7 +772,7 @@ pub trait Oplog: Any + Debug + Send + Sync { index.next() }); let entry = record.into_inline_entry(); - let index = self.add(entry.clone()).await; + let index = self.add(entry.clone()).await?; assert_eq!( index, expected_index, "oplog add_durable_stream_batch default observed a concurrent writer" @@ -698,12 +782,6 @@ pub trait Oplog: Any + Debug + Send + Sync { Ok(result) } - /// A variant of add that can inject failures in tests. TO BE REMOVED - async fn fallible_add(&self, entry: OplogEntry) -> Result<(), String> { - self.add(entry).await; - Ok(()) - } - /// Drop a chunk of entries from the beginning of the oplog /// /// This should only be called _after_ `append` succeeded in the layer below this one @@ -712,7 +790,10 @@ pub trait Oplog: Any + Debug + Send + Sync { async fn drop_prefix(&self, last_dropped_id: OplogIndex) -> u64; /// Commits the buffered entries to the oplog - async fn commit(&self, level: CommitLevel) -> BTreeMap; + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError>; /// Returns the current oplog index async fn current_oplog_index(&self) -> OplogIndex; @@ -766,10 +847,10 @@ pub trait Oplog: Any + Debug + Send + Sync { async fn length(&self) -> u64; /// Adds an entry to the oplog and immediately commits it - async fn add_and_commit(&self, entry: OplogEntry) -> OplogIndex { - let index = self.add(entry).await; - self.commit(CommitLevel::Always).await; - index + async fn add_and_commit(&self, entry: OplogEntry) -> Result { + let index = self.add(entry).await?; + self.commit(CommitLevel::Always).await?; + Ok(index) } /// Uploads a big oplog payload and returns a reference to it @@ -820,7 +901,7 @@ pub trait Oplog: Any + Debug + Send + Sync { &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result; + ) -> Result; /// Like [`Self::add_start_with_reserved_raw_payload`], but builds the request after the leaf /// oplog has assigned the exact `Start` index. The leaf must invoke `build_request` and append @@ -829,7 +910,7 @@ pub trait Oplog: Any + Debug + Send + Sync { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result; + ) -> Result; /// Atomically appends a `Start` entry and a second entry (its `End` or /// `Cancelled`) that references the `Start`'s `OplogIndex`. @@ -850,19 +931,21 @@ pub trait Oplog: Any + Debug + Send + Sync { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex); + ) -> Result<(OplogIndex, OplogIndex), OplogError>; - /// Like [`add_pair`](Self::add_pair) but for two already-built entries, returning a - /// `Result` so test wrappers can inject a write failure on either entry. The default - /// delegates to `add_pair`, inheriting its atomic buffering, so the two entries are - /// never split by a commit-threshold check or a crash boundary. - async fn fallible_add_pair( - &self, - first: OplogEntry, - second: OplogEntry, - ) -> Result<(OplogIndex, OplogIndex), String> { - let (first_idx, second_idx) = self.add_pair(first, Box::new(move |_| second)).await; - Ok((first_idx, second_idx)) + /// The shard epoch this oplog's writes assert, or `None` for an oplog nothing fences - one + /// opened without an ownership claim, or an ephemeral one. + /// + /// Only the primary oplog knows it, so a wrapper answers from the oplog it wraps. + fn shard_epoch(&self) -> Option { + self.inner().and_then(|inner| inner.shard_epoch()) + } + + /// The refusal this oplog has latched, if the storage has turned one of its writes away: + /// every later write fails on it, so the handle is finished. Answered without a round trip, + /// so the open-oplog cache can decline to hand a finished handle to a new opener. + fn fence(&self) -> Option { + self.inner().and_then(|inner| inner.fence()) } /// Returns the inner oplog wrapped by this implementation, if any. @@ -967,7 +1050,7 @@ pub trait OplogOps: Oplog { &self, request: T, build_start: impl FnOnce(OplogPayload) -> OplogEntry + Send + 'static, - ) -> Result<(OplogIndex, PendingUpload), String> + ) -> Result<(OplogIndex, PendingUpload), OplogError> where T: BinaryCodec + Debug + Clone + PartialEq + Send + Sync + 'static, { @@ -992,7 +1075,7 @@ pub trait OplogOps: Oplog { &self, build_request: impl FnOnce(OplogIndex) -> Result + Send + 'static, build_start: impl FnOnce(OplogPayload) -> OplogEntry + Send + 'static, - ) -> Result<(OplogIndex, PendingUpload), String> + ) -> Result<(OplogIndex, PendingUpload), OplogError> where T: BinaryCodec + Debug + Clone + PartialEq + Send + Sync + 'static, { @@ -1039,7 +1122,7 @@ pub trait OplogOps: Oplog { response: &HostResponse, function_type: DurableFunctionType, parent_start_index: Option, - ) -> Result<(OplogIndex, OplogIndex), String> { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { let request_payload: OplogPayload = self.upload_payload(request).await?; let response_payload: OplogPayload = self.upload_payload(response).await?; let now = Timestamp::now_utc(); @@ -1062,7 +1145,7 @@ pub trait OplogOps: Oplog { forced_commit: false, }), ) - .await; + .await?; Ok((start_idx, end_idx)) } @@ -1070,11 +1153,11 @@ pub trait OplogOps: Oplog { &self, invocation: AgentInvocation, wallet_pin: InvocationWalletPin, - ) -> Result { + ) -> Result { let entry = self .agent_invocation_started_entry(invocation, wallet_pin) .await?; - self.add(entry.clone()).await; + self.add(entry.clone()).await?; Ok(entry) } @@ -1082,11 +1165,11 @@ pub trait OplogOps: Oplog { &self, invocation: AgentInvocation, wallet_pin: InvocationWalletPin, - ) -> Result { + ) -> Result { let entry = self .agent_invocation_started_entry(invocation, wallet_pin) .await?; - Ok(self.add(entry).await) + self.add(entry).await } async fn agent_invocation_started_entry( @@ -1114,7 +1197,7 @@ pub trait OplogOps: Oplog { method_name: Option, consumed_fuel: u64, component_revision: ComponentRevision, - ) -> Result { + ) -> Result { let consumed_fuel = if consumed_fuel > i64::MAX as u64 { i64::MAX } else { @@ -1129,7 +1212,7 @@ pub trait OplogOps: Oplog { consumed_fuel, component_revision, }; - Ok(self.add(entry).await) + self.add(entry).await } async fn create_snapshot_based_update_description( @@ -1223,6 +1306,8 @@ pub type OplogCloseCompletion = Shared>>; struct OpenOplogEntry { oplog: Weak, closed: OplogCloseCompletion, + /// The epoch the opener that constructed this handle asked it to assert. + requested_epoch: Option, } type OplogSlot = Arc>>; @@ -1304,6 +1389,7 @@ impl OpenOplogs { constructor: impl OplogConstructor, ) -> Arc { lifecycle.assert_agent(agent_id); + let requested_epoch = constructor.shard_epoch(); let slot = self.slot(agent_id).await; let is_primary = Arc::ptr_eq( &slot, @@ -1321,16 +1407,25 @@ impl OpenOplogs { } else { lifecycle.slot.as_ref().unwrap().as_ref() }; - if let Some(oplog) = cached.and_then(|entry| entry.oplog.upgrade()) { - if !oplog.is_retired() { - return oplog; + match cached.and_then(|entry| entry.oplog.upgrade()) { + Some(oplog) if !oplog.is_retired() => { + let opened_with = cached.and_then(|entry| entry.requested_epoch); + if can_reuse(&*oplog, opened_with, requested_epoch) { + return oplog; + } + // Replaced without waiting for it to close: see `can_reuse`. + } + live => { + if let Some(oplog) = live { + oplog.retire(); + } + if let Some(cached) = cached { + // Completion, including an error, proves the old layer no longer owns running + // work. The new attempt reloads persisted state rather than inheriting the old + // error. + let _ = cached.closed.clone().await; + } } - oplog.retire(); - } - if let Some(cached) = cached { - // Completion, including an error, proves the old layer no longer owns running work. - // The new attempt reloads persisted state rather than inheriting the old error. - let _ = cached.closed.clone().await; } let owner = self.clone(); let close_agent_id = agent_id.clone(); @@ -1341,6 +1436,7 @@ impl OpenOplogs { let entry = Some(OpenOplogEntry { oplog: Arc::downgrade(&oplog), closed: closed.clone(), + requested_epoch, }); if let Some(wrapper) = &mut wrapper_slot { **wrapper = entry; @@ -1360,6 +1456,27 @@ impl OpenOplogs { } } +/// Whether a live cached handle can be handed to an opener asking for `requested`. It cannot, +/// and is replaced, when: +/// - it is fenced: the storage refused one of its writes, so every later one is refused too; +/// - or it belongs to an older ownership generation: `requested` is newer than the epoch it was +/// opened with (`None`, opened without a claim, is older than any epoch) and it really asserts +/// that epoch. An ephemeral handle opened with an epoch asserts none, so it is reused at any +/// epoch; one opened without a claim is replaced by any open that makes one. +/// +/// A replaced handle may still be held by a worker that is stopping, so nobody waits for it to +/// close; it keeps any background work, such as an archive transfer, until its holder drops it. +/// An equal or older request gets the cached handle, so no two live handles assert one epoch. +fn can_reuse( + oplog: &dyn Oplog, + opened_with: Option, + requested: Option, +) -> bool { + let fenced = oplog.fence().is_some(); + let older_generation = requested > opened_with && oplog.shard_epoch() == opened_with; + !fenced && !older_generation +} + impl Debug for OpenOplogs { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("OpenOplogs").finish() @@ -1373,4 +1490,10 @@ pub trait OplogConstructor: Send { lifecycle: &mut OplogLifecycleGuard, close: Box, ) -> Arc; + + /// The epoch the oplog this constructor builds is asked to assert, or `None` for one opened + /// without an ownership claim. The open-oplog cache compares it with the epoch a cached + /// handle was opened with, so it has no default: a layer that left it out would hand an + /// older generation's handle to every newer opener. + fn shard_epoch(&self) -> Option; } diff --git a/golem-worker-executor/src/services/oplog/multilayer.rs b/golem-worker-executor/src/services/oplog/multilayer.rs index b3b0aa31db..0a4895577f 100644 --- a/golem-worker-executor/src/services/oplog/multilayer.rs +++ b/golem-worker-executor/src/services/oplog/multilayer.rs @@ -24,13 +24,14 @@ use crate::services::oplog::multilayer::BackgroundTransferMessage::{ use crate::services::oplog::reader::{OplogRead, OplogReadError, OplogReadSource, fail_stop}; use crate::services::oplog::{ CommitLevel, DurableStreamBatchBuilder, IndexedReservedStartBuilder, OpenOplogs, Oplog, - OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogLifecycleGuard, OplogService, - OrderedOplogStart, ReservedRawStartBuilder, decode_scan_cursor, downcast_oplog, + OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogError, OplogLifecycleGuard, + OplogService, OrderedOplogStart, ReservedRawStartBuilder, decode_scan_cursor, downcast_oplog, first_scan_cursor, }; use crate::storage::indexed::IndexedStorageMetaNamespace; use async_trait::async_trait; use futures::FutureExt; +use golem_common::model::ShardEpoch; use golem_common::model::account::AccountId; use golem_common::model::agent::AgentMode; use golem_common::model::component::ComponentId; @@ -423,6 +424,7 @@ struct CreateOplogConstructor { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, } impl CreateOplogConstructor { @@ -438,6 +440,7 @@ impl CreateOplogConstructor { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Self { Self { owned_agent_id, @@ -450,12 +453,17 @@ impl CreateOplogConstructor { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, } } } #[async_trait] impl OplogConstructor for CreateOplogConstructor { + fn shard_epoch(&self) -> Option { + self.shard_epoch + } + async fn create_oplog( self, lifecycle: &mut OplogLifecycleGuard, @@ -487,6 +495,7 @@ impl OplogConstructor for CreateOplogConstructor { self.initial_worker_metadata, self.last_known_status, self.execution_status, + self.shard_epoch, ) .await } else { @@ -499,6 +508,7 @@ impl OplogConstructor for CreateOplogConstructor { self.initial_worker_metadata, self.last_known_status, self.execution_status, + self.shard_epoch, ) .await } @@ -512,6 +522,7 @@ impl OplogConstructor for CreateOplogConstructor { self.initial_worker_metadata, self.last_known_status, self.execution_status, + self.shard_epoch, ) .await }; @@ -653,6 +664,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -669,6 +681,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ), ) .await @@ -683,6 +696,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -699,6 +713,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ), ) .await @@ -713,6 +728,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -729,6 +745,7 @@ impl OplogService for MultiLayerOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ), ) .await @@ -760,15 +777,19 @@ impl OplogService for MultiLayerOplogService { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, - ) { + expected_epoch: Option, + ) -> Result<(), OplogError> { lifecycle.assert_agent(&owned_agent_id.agent_id); self.abort_transfer(&owned_agent_id.agent_id).await; + // The primary decides: the archive layers carry no epoch of their own, and a refused + // delete means they now hold the new owner's history. self.primary - .delete(lifecycle, owned_agent_id, agent_mode) - .await; + .delete(lifecycle, owned_agent_id, agent_mode, expected_epoch) + .await?; for layer in &self.lower { layer.delete(owned_agent_id, agent_mode).await } + Ok(()) } async fn read_exact( @@ -1127,6 +1148,29 @@ impl MultiLayerOplog { Some(Self::archive(this, true).await) } + /// Ends this handle's background transfer for good and returns once nothing the transfer + /// started is still running. Does nothing for an oplog without archive layers. + /// + /// Dropping the handle is not enough: a transfer under way holds its own reference to it, and + /// goes on to drop the primary's prefix, deleting the primary oplog when that empties it, no + /// matter who has opened the agent's oplog since. The entries it did not move stay in the + /// primary layer, for the next handle to archive. + /// + /// Built on the same `retire`/`closed` mechanism the open-oplog cache uses to evict a stale + /// handle, rather than a second, competing shutdown path: `retire` unregisters and aborts the + /// transfer fiber, and `closed` (`MultiLayerOplogService::transfer_closed`) is its completion. + pub async fn try_abort_transfer(this: &Arc) { + let Some(this) = downcast_oplog::(this) else { + return; + }; + this.retire(); + let _ = this.closed().await; + // A transfer cancelled while it waited for its `drop_prefix` reply has already handed the + // job to the primary's actor, which runs it regardless. The actor serves jobs in the order + // they were sent, so a reply to a job sent after it means that job has finished. + this.primary.current_oplog_index().await; + } + async fn archive(this: Arc, blocking: bool) -> bool { let (done_tx, done_rx) = if blocking { let (done_tx, done_rx) = tokio::sync::oneshot::channel(); @@ -1261,7 +1305,7 @@ impl Oplog for MultiLayerOplog { async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { self.primary.add_durable_stream_batch(make_batch).await } @@ -1271,8 +1315,11 @@ impl Oplog for MultiLayerOplog { dropped_entries } - async fn commit(&self, level: CommitLevel) -> BTreeMap { - let result = self.primary.commit(level).await; + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError> { + let result = self.primary.commit(level).await?; if let Some(index) = result.keys().next_back() { self.last_reported_commit_index.max(*index); @@ -1292,7 +1339,7 @@ impl Oplog for MultiLayerOplog { }); self.last_transfer_point.max(last_committed_idx); } - result + Ok(result) } async fn current_oplog_index(&self) -> OplogIndex { @@ -1362,7 +1409,7 @@ impl Oplog for MultiLayerOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { self.primary.add_pair(start, make_second).await } @@ -1370,7 +1417,7 @@ impl Oplog for MultiLayerOplog { &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { self.primary .add_start_with_reserved_raw_payload(serialized_request, build_start) .await @@ -1379,7 +1426,7 @@ impl Oplog for MultiLayerOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { self.primary .add_start_with_indexed_reserved_raw_payload(build_request) .await @@ -1415,6 +1462,22 @@ trait BackgroundTransfer { async fn verify_target(&self, entries: &[(OplogIndex, OplogEntry)]); async fn drop_source_prefix(&self, last_dropped_id: OplogIndex); + /// `try_abort_transfer` can land between any two `.await`s here, including between + /// `append_target` and `drop_source_prefix`: a `JoinHandle::abort` takes effect at whichever + /// suspension point the task is next parked at, not at a step boundary this trait controls. + /// So a target chunk can already be durable while the source that fed it has not yet been + /// trimmed. That is only safe because the target append is written to be replayed: the next + /// transfer starts from the same untrimmed source position and chunks from there, so every + /// chunk it shares an id with has the identical bytes, and the archive backing `append_target` + /// (the compressed layer; a blob-backed one overwrites by path and is idempotent by + /// construction) reconciles a duplicate-id write against what is already stored instead of + /// treating it as a conflict. A next transfer that covers more entries ends the aborted run's + /// trailing partial chunk at a later id instead: the two overlap with identical entries, which + /// reads tolerate, and the earlier one goes with the layer's next `drop_prefix` past it. Sequencing the steps + /// behind a cooperative, checked-between-steps cancellation instead of a hard abort would + /// also close this window, but the archive already has to tolerate a replayed append for + /// other reasons (retried indeterminate writes), so leaning on that here avoids a second + /// cancellation mechanism. async fn run(&self) { let entries = self.read_source().await; match entries.last() { @@ -1750,6 +1813,7 @@ mod transfer_lifecycle_tests { metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let entries = (1..=archived + 10) @@ -1768,9 +1832,9 @@ mod transfer_lifecycle_tests { .collect::>(); let captured = archived + 2; for entry in &entries[..captured as usize] { - writer.add(entry.clone()).await; + writer.add(entry.clone()).await.unwrap(); } - writer.commit(CommitLevel::Always).await; + writer.commit(CommitLevel::Always).await.unwrap(); let deep_archive = deepest.open(&owned, AgentMode::Durable).await; if archived > 0 { let prefix = entries[..archived as usize] @@ -1790,6 +1854,7 @@ mod transfer_lifecycle_tests { metadata, default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_eq!( @@ -1797,9 +1862,9 @@ mod transfer_lifecycle_tests { OplogIndex::from_u64(captured) ); for entry in &entries[captured as usize..] { - writer.add(entry.clone()).await; + writer.add(entry.clone()).await.unwrap(); } - writer.commit(CommitLevel::Always).await; + writer.commit(CommitLevel::Always).await.unwrap(); assert_eq!(observer.length().await, 10); let service = MultiLayerOplogService::new(observer_service, nev![first, deepest], 100, 100); diff --git a/golem-worker-executor/src/services/oplog/plugin.rs b/golem-worker-executor/src/services/oplog/plugin.rs index 7db95aff13..cb410605cf 100644 --- a/golem-worker-executor/src/services/oplog/plugin.rs +++ b/golem-worker-executor/src/services/oplog/plugin.rs @@ -18,8 +18,8 @@ use crate::services::activity::{ActivityGate, ActivityGuard}; use crate::services::component::ComponentService; use crate::services::oplog::{ CommitLevel, DurableStreamBatchBuilder, IndexedReservedStartBuilder, OpenOplogs, Oplog, - OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogLifecycleGuard, OplogService, - OrderedOplogStart, ReservedRawStartBuilder, + OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogError, OplogFence, + OplogLifecycleGuard, OplogService, OrderedOplogStart, ReservedRawStartBuilder, }; use crate::services::shard::ShardService; use crate::services::worker_activator::WorkerActivator; @@ -33,6 +33,7 @@ use anyhow::anyhow; use async_lock::{RwLock, RwLockUpgradableReadGuard}; use async_trait::async_trait; use futures::FutureExt; +use golem_common::model::ShardEpoch; use golem_common::model::account::AccountId; use golem_common::model::agent::{AgentMode, ParsedAgentId, Principal}; use golem_common::model::component::{ComponentId, ComponentRevision, InstalledPlugin}; @@ -533,6 +534,7 @@ struct CreateOplogConstructor { execution_status: read_only_lock::std::ReadOnlyLock, plugin_max_commit_count: usize, plugin_max_elapsed_time: Duration, + shard_epoch: Option, } impl CreateOplogConstructor { @@ -551,6 +553,7 @@ impl CreateOplogConstructor { execution_status: read_only_lock::std::ReadOnlyLock, plugin_max_commit_count: usize, plugin_max_elapsed_time: Duration, + shard_epoch: Option, ) -> Self { Self { owned_agent_id, @@ -566,25 +569,22 @@ impl CreateOplogConstructor { execution_status, plugin_max_commit_count, plugin_max_elapsed_time, + shard_epoch, } } } #[async_trait] impl OplogConstructor for CreateOplogConstructor { + fn shard_epoch(&self) -> Option { + self.shard_epoch + } + async fn create_oplog( self, lifecycle: &mut OplogLifecycleGuard, close: Box, ) -> Arc { - let last_oplog_index = match self.last_oplog_index { - Some(idx) => idx, - None => { - self.inner - .get_last_index(&self.owned_agent_id, self.agent_mode) - .await - } - }; let inner = if let Some(initial_entry) = self.initial_entry { if self.fresh { self.inner @@ -596,6 +596,7 @@ impl OplogConstructor for CreateOplogConstructor { self.initial_worker_metadata.clone(), self.last_known_status.clone(), self.execution_status.clone(), + self.shard_epoch, ) .await } else { @@ -608,6 +609,7 @@ impl OplogConstructor for CreateOplogConstructor { self.initial_worker_metadata.clone(), self.last_known_status.clone(), self.execution_status.clone(), + self.shard_epoch, ) .await } @@ -617,13 +619,21 @@ impl OplogConstructor for CreateOplogConstructor { lifecycle, &self.owned_agent_id, self.agent_mode, - Some(last_oplog_index), + self.last_oplog_index, self.initial_worker_metadata.clone(), self.last_known_status.clone(), self.execution_status.clone(), + self.shard_epoch, ) .await }; + // Taken from the opened inner oplog, not read up front: the inner open claims the shard + // epoch, so an index read before it could be behind a losing executor's last commit, and + // the forwarding buffer would then label its entries with indexes already in use. + let last_oplog_index = match self.last_oplog_index { + Some(idx) => idx, + None => inner.current_oplog_index().await, + }; Arc::new( ForwardingOplog::new( @@ -751,6 +761,7 @@ impl OplogService for ForwardingOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -770,6 +781,7 @@ impl OplogService for ForwardingOplogService { execution_status, self.plugin_max_commit_count, self.plugin_max_elapsed_time, + shard_epoch, ), ) .await @@ -784,6 +796,7 @@ impl OplogService for ForwardingOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -803,6 +816,7 @@ impl OplogService for ForwardingOplogService { execution_status, self.plugin_max_commit_count, self.plugin_max_elapsed_time, + shard_epoch, ), ) .await @@ -817,6 +831,7 @@ impl OplogService for ForwardingOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { self.oplogs .get_or_open( @@ -836,6 +851,7 @@ impl OplogService for ForwardingOplogService { execution_status, self.plugin_max_commit_count, self.plugin_max_elapsed_time, + shard_epoch, ), ) .await @@ -854,9 +870,10 @@ impl OplogService for ForwardingOplogService { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, - ) { + expected_epoch: Option, + ) -> Result<(), OplogError> { self.inner - .delete(lifecycle, owned_agent_id, agent_mode) + .delete(lifecycle, owned_agent_id, agent_mode, expected_epoch) .await } @@ -942,6 +959,8 @@ impl OplogService for ForwardingOplogService { pub struct ForwardingOplog { inner: Arc, jobs: tokio::sync::mpsc::UnboundedSender, + /// Completion of the actor task; also used to wait for a + /// cooperative shutdown without needing to take the `JoinHandle` out of a shared reference. closed: OplogCloseCompletion, retired: AtomicBool, timer: Option>, @@ -952,32 +971,36 @@ pub struct ForwardingOplog { /// A request processed by the [`ForwardingOplog`] actor task, which exclusively owns the /// [`ForwardingOplogState`]. enum ForwardingJob { + /// Requests a graceful shutdown: sent by `Drop` and `retire`. + /// Drains no further jobs after this one, so the actor exits once every job already queued + /// ahead of it - including a stray `Tick` the timer enqueued in the instant before it was + /// stopped - has been processed. Close, Add { entry: OplogEntry, - done: tokio::sync::oneshot::Sender, + done: tokio::sync::oneshot::Sender>, }, AddDurableStreamBatch { make_batch: DurableStreamBatchBuilder, - done: tokio::sync::oneshot::Sender, String>>, + done: tokio::sync::oneshot::Sender, OplogError>>, }, AddPair { start: OplogEntry, make_second: Box OplogEntry + Send>, - done: tokio::sync::oneshot::Sender<(OplogIndex, OplogIndex)>, + done: tokio::sync::oneshot::Sender>, }, AddStart { serialized_request: Vec, build_start: ReservedRawStartBuilder, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender>, }, AddIndexedStart { build_request: IndexedReservedStartBuilder, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender>, }, Commit { level: CommitLevel, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender, OplogError>>, }, SetWorkerEventService { service: Arc, @@ -1082,13 +1105,17 @@ impl ForwardingOplog { ForwardingJob::Add { entry, done } => { let cache_entries = state.cache_is_required(); let cached_entry = cache_entries.then(|| entry.clone()); - let idx = state.inner.add(entry).await; - if let Some(entry) = cached_entry { - state.record_cached([(idx, entry)]); - } else { - state.record_uncached([idx]); + let result = state.inner.add(entry).await; + // A refused write is recorded nowhere: the inner oplog appended + // nothing, so the index it would have taken has to stay free. + if let Ok(idx) = &result { + if let Some(entry) = cached_entry { + state.record_cached([(*idx, entry)]); + } else { + state.record_uncached([*idx]); + } } - let _ = done.send(idx); + let _ = done.send(result); } ForwardingJob::AddDurableStreamBatch { make_batch, done } => { let cache_entries = state.cache_is_required(); @@ -1117,10 +1144,15 @@ impl ForwardingOplog { cache_entries.then(|| (start.clone(), second.clone())); let result = state.inner.add_pair(start, Box::new(move |_| second)).await; - if let Some((start, second)) = cached_entries { - state.record_cached([(result.0, start), (result.1, second)]); - } else { - state.record_uncached([result.0, result.1]); + if let Ok((start_idx, second_idx)) = &result { + if let Some((start, second)) = cached_entries { + state.record_cached([ + (*start_idx, start), + (*second_idx, second), + ]); + } else { + state.record_uncached([*start_idx, *second_idx]); + } } let _ = done.send(result); } @@ -1168,41 +1200,49 @@ impl ForwardingOplog { let _ = done.send(result); } ForwardingJob::Commit { level, done } => { - let mut result = state.inner.commit(level).await; - // Update last_committed_idx from committed entries - if let Some(max_idx) = result.keys().max() - && *max_idx > state.last_committed_idx - { - state.last_committed_idx = *max_idx; - } - state.commit_count += 1; - if state.commit_count >= max_commit_count { - // Spanned inside the threshold check, not around the commit: - // this arm runs per oplog commit, the flush only every - // `max_commit_count` of them. The actor has no ambient span, - // so without this the flush would be untraceable. - // - // Named apart from the periodic `oplog_forwarding_flush` so the - // two triggers stay distinguishable in a trace backend. The link - // points at the worker's startup rather than at the commit that - // tripped the threshold: the actor receives commits over a - // channel, so the committing invocation's context is not - // available here. - state - .try_flush() - .instrument(related_span!( - flush_origin, - tracing::Level::INFO, - "oplog_forwarding_threshold_flush", - agent_id = %agent_id - )) - .await; + match state.inner.commit(level).await { + Err(error) => { + let _ = done.send(Err(error)); + } + Ok(mut result) => { + // Update last_committed_idx from committed entries + if let Some(max_idx) = result.keys().max() + && *max_idx > state.last_committed_idx + { + state.last_committed_idx = *max_idx; + } + state.commit_count += 1; + if state.commit_count >= max_commit_count { + // Spanned inside the threshold check, not around the + // commit: this arm runs per oplog commit, the flush only + // every `max_commit_count` of them. The actor has no + // ambient span, so without this the flush would be + // untraceable. + // + // Named apart from the periodic + // `oplog_forwarding_flush` so the two triggers stay + // distinguishable in a trace backend. The link points at + // the worker's startup rather than at the commit that + // tripped the threshold: the actor receives commits over + // a channel, so the committing invocation's context is + // not available here. + state + .try_flush() + .instrument(related_span!( + flush_origin, + tracing::Level::INFO, + "oplog_forwarding_threshold_flush", + agent_id = %agent_id + )) + .await; + } + // Merge entries committed directly to inner during flush + // so the Worker folds them into AgentStatusRecord + result.append(&mut state.pending_direct_commits); + state.pending_checkpoint_activity.take(); + let _ = done.send(Ok(result)); + } } - // Merge entries committed directly to inner during flush - // so the Worker folds them into AgentStatusRecord - result.append(&mut state.pending_direct_commits); - state.pending_checkpoint_activity.take(); - let _ = done.send(result); } ForwardingJob::SetWorkerEventService { service, done } => { state.worker_event_service = Some(service); @@ -1313,8 +1353,11 @@ impl ForwardingOplog { /// Enqueues a job for the actor task and awaits its reply. /// - /// A missing reply means the actor failed or this handle was used after retirement. - /// Orderly shutdown drains jobs queued before Close. + /// A missing reply means the actor failed, or this handle was used after retirement or after + /// `retire` stopped it - both cooperative shutdowns that run only once no + /// caller can still be in flight, and both drain every job queued before `Close`, so a + /// missing reply otherwise means the actor itself panicked and the oplog's state is no + /// longer trustworthy. async fn run_job( &self, make_job: impl FnOnce(tokio::sync::oneshot::Sender) -> ForwardingJob, @@ -1345,6 +1388,11 @@ impl Drop for ForwardingOplog { if let Some(timer) = self.timer.take() { timer.abort(); } + // In-flight `Oplog` calls borrow `self`, so at this point no caller can be awaiting a + // job reply anymore. Requesting a graceful stop (rather than aborting the actor outright) + // lets it drain whatever is already queued ahead of `Close` - including a stray `Tick` + // the timer enqueued in the instant before it was aborted - before its background monitor + // tasks are joined and it exits. let _ = self.jobs.send(ForwardingJob::Close); } } @@ -1400,7 +1448,7 @@ impl Oplog for ForwardingOplog { async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { self.run_job(|done| ForwardingJob::AddDurableStreamBatch { make_batch, done }) .await } @@ -1409,7 +1457,10 @@ impl Oplog for ForwardingOplog { self.inner.drop_prefix(last_dropped_id).await } - async fn commit(&self, level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError> { self.run_job(|done| ForwardingJob::Commit { level, done }) .await } @@ -1475,7 +1526,7 @@ impl Oplog for ForwardingOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { self.run_job(|done| ForwardingJob::AddPair { start, make_second, @@ -1488,7 +1539,7 @@ impl Oplog for ForwardingOplog { &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { self.run_job(|done| ForwardingJob::AddStart { serialized_request, build_start, @@ -1500,7 +1551,7 @@ impl Oplog for ForwardingOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { self.run_job(|done| ForwardingJob::AddIndexedStart { build_request, done, @@ -1613,6 +1664,14 @@ impl ForwardingOplogState { if self.forwarding_retired.is_cancelled() { return; } + // A fenced oplog is finished: nothing may be sent or checkpointed from it again. Without + // this, every tick and threshold flush picks the same batch and writes its checkpoint + // again, taking an index on an oplog that refuses the commit. The bookkeeping below is + // skipped as well, so the mirrored buffer is no longer pruned and the commit count no + // longer reset; both last only until the fenced worker is given up. + if self.inner.fence().is_some() { + return; + } let status = self.last_known_status.get(); let flush_set = self.reconcile_plugin_state(&status); @@ -1773,14 +1832,19 @@ impl ForwardingOplogState { target_agent = %id, "Oplog processor: resolved target plugin worker" ); - self.write_checkpoint( - grant_id, - &id, - live.confirmed_up_to, - live.confirmed_up_to, - live.last_batch_start, - ) - .await; + if self + .write_checkpoint( + grant_id, + &id, + live.confirmed_up_to, + live.confirmed_up_to, + live.last_batch_start, + ) + .await + .is_err() + { + return; + } if let Some(s) = self.plugin_state.get_mut(&grant_id) { s.target_agent_id = Some(id.clone()); } @@ -1828,14 +1892,19 @@ impl ForwardingOplogState { } if !is_retry { - self.write_checkpoint( - grant_id, - &target_agent_id, - live.confirmed_up_to, - batch_end, - batch_start, - ) - .await; + if self + .write_checkpoint( + grant_id, + &target_agent_id, + live.confirmed_up_to, + batch_end, + batch_start, + ) + .await + .is_err() + { + return; + } if let Some(s) = self.plugin_state.get_mut(&grant_id) { s.sending_up_to = batch_end; } @@ -1880,14 +1949,19 @@ impl ForwardingOplogState { "Oplog processor: batch enqueued successfully" ); // Enqueue succeeded — immediately confirm - self.write_checkpoint( - grant_id, - &target_agent_id, - batch_end, - batch_end, - batch_start, - ) - .await; + if self + .write_checkpoint( + grant_id, + &target_agent_id, + batch_end, + batch_end, + batch_start, + ) + .await + .is_err() + { + return; + } if let Some(s) = self.plugin_state.get_mut(&grant_id) { s.confirmed_up_to = batch_end; s.sending_up_to = batch_end; @@ -2052,6 +2126,13 @@ impl ForwardingOplogState { } /// Write an OplogProcessorCheckpoint entry, commit it, and update index tracking. + /// + /// `Err` means the agent's oplog was fenced: its shard has a new owner, so nothing more may be + /// written and forwarding stops. Giving the agent up is deliberately not this layer's call - a + /// storage decorator has no business stopping agents - and it does not need to be: the fence + /// latches on the oplog, so the worker's own commit path is refused too and gives it up. + /// + /// Any other storage failure keeps the fail-stop behaviour it has always had. async fn write_checkpoint( &mut self, grant_id: EnvironmentPluginGrantId, @@ -2059,7 +2140,13 @@ impl ForwardingOplogState { confirmed_up_to: OplogIndex, sending_up_to: OplogIndex, last_batch_start: OplogIndex, - ) { + ) -> Result<(), OplogFence> { + // Checked here as well as when the flush starts: the worker's own write can latch the + // fence while the flush awaits a send or a lookup, and a checkpoint added after that + // would still take an index for an entry that is never committed. + if let Some(fence) = self.inner.fence() { + return Err(fence); + } if self.pending_checkpoint_activity.is_none() { self.pending_checkpoint_activity = Some( self.forwarding_activity @@ -2077,19 +2164,40 @@ impl ForwardingOplogState { }; let cache_entries = self.cache_is_required(); let cached_checkpoint = cache_entries.then(|| checkpoint.clone()); - let idx = self.inner.add(checkpoint).await; + let idx = match self.inner.add(checkpoint).await { + Ok(idx) => idx, + Err(OplogError::Fenced(fence)) => return Err(self.stop_forwarding(fence)), + Err(error) => panic!("oplog write: {error}"), + }; if let Some(checkpoint) = cached_checkpoint { self.record_cached([(idx, checkpoint)]); } else { self.record_uncached([idx]); } - let committed = self.inner.commit(CommitLevel::Always).await; + let committed = match self.inner.commit(CommitLevel::Always).await { + Ok(committed) => committed, + Err(OplogError::Fenced(fence)) => return Err(self.stop_forwarding(fence)), + Err(error) => panic!("oplog write: {error}"), + }; if let Some(max_idx) = committed.keys().max().copied() { self.last_committed_idx = self.last_committed_idx.max(max_idx); } // Track all directly committed entries so ForwardingOplog::commit() // can surface them to the Worker for status folding self.pending_direct_commits.extend(committed); + Ok(()) + } + + /// Logs the checkpoint that found the fence and hands the fence back to the caller, which + /// stops forwarding for this agent. Every flush after it sees the latched fence and returns + /// before writing anything. + fn stop_forwarding(&self, fence: OplogFence) -> OplogFence { + tracing::info!( + source_agent = %self.initial_worker_metadata.agent_id, + expected_epoch = fence.expected_epoch.0, + "Oplog processor: checkpoint fenced, the shard has a new owner - forwarding stopped" + ); + fence } /// Prune buffer: drain entries that ALL active/in-flight plugins have confirmed past. @@ -2147,6 +2255,10 @@ impl ForwardingOplogState { if self.forwarding_retired.is_cancelled() { return; } + // A migration is recorded with a checkpoint, which a fenced oplog no longer takes. + if self.inner.fence().is_some() { + return; + } let status = self.last_known_status.get(); // Ensure plugin_state is reconciled with current status self.reconcile_plugin_state(&status); @@ -2322,14 +2434,19 @@ impl ForwardingOplogState { } } - self.write_checkpoint( - grant_id, - &new_target, - confirmed, - confirmed, - last_batch_start, - ) - .await; + if self + .write_checkpoint( + grant_id, + &new_target, + confirmed, + confirmed, + last_batch_start, + ) + .await + .is_err() + { + return; + } if let Some(s) = self.plugin_state.get_mut(&grant_id) { s.target_agent_id = Some(new_target.clone()); } @@ -2642,7 +2759,7 @@ mod tests { ) .await, ); - oplog.add(OplogEntry::no_op(None)).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); let commit = tokio::spawn({ let oplog = oplog.clone(); async move { oplog.commit(CommitLevel::Always).await } @@ -2650,17 +2767,17 @@ mod tests { entered.notified().await; oplog.fence_forwarding(); oplog.drain_forwarding().await; - let committed = commit.await.unwrap(); + let committed = commit.await.unwrap().unwrap(); assert_eq!(committed.len(), 3); assert!(matches!(committed.get(&OplogIndex::from_u64(3)), Some(OplogEntry::OplogProcessorCheckpoint { confirmed_up_to, sending_up_to, .. }) if *confirmed_up_to == OplogIndex::NONE && *sending_up_to == OplogIndex::INITIAL)); oplog.jobs.send(ForwardingJob::Tick).unwrap(); assert_eq!( - oplog.add(OplogEntry::no_op(None)).await, + oplog.add(OplogEntry::no_op(None)).await.unwrap(), OplogIndex::from_u64(4) ); - let cleanup = oplog.commit(CommitLevel::Always).await; + let cleanup = oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!(cleanup.len(), 1); assert_eq!(plugin.send_count().await, 1); assert_eq!(oplog.current_oplog_index().await, OplogIndex::from_u64(4)); @@ -2696,8 +2813,8 @@ mod tests { Duration::from_secs(3600), ) .await; - oplog.add(OplogEntry::no_op(None)).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); oplog.jobs.send(ForwardingJob::Tick).unwrap(); entered.notified().await; oplog.fence_forwarding(); @@ -2705,7 +2822,7 @@ mod tests { assert!(futures::poll!(drain.as_mut()).is_pending()); release.notify_one(); drain.await; - let committed = oplog.commit(CommitLevel::Always).await; + let committed = oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!(committed.len(), 1); assert!(matches!( committed.get(&OplogIndex::from_u64(2)), @@ -2739,8 +2856,8 @@ mod tests { ) .await; oplog.drain_forwarding().await; - oplog.add(OplogEntry::no_op(None)).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!(plugin.send_count().await, 1); } @@ -2882,6 +2999,10 @@ mod tests { std::sync::Mutex, Arc)>>, read_exact_count: std::sync::atomic::AtomicUsize, read_exact_requests: std::sync::Mutex>, + /// Refuses the next `commit` as fenced once set. The refusal latches, as it does on the + /// primary oplog, so every commit after it is refused and `fence` reports it. + armed_fence: std::sync::Mutex>, + latched_fence: std::sync::OnceLock, } #[allow(dead_code)] @@ -2895,9 +3016,15 @@ mod tests { checkpoint_commit_gate: std::sync::Mutex::new(None), read_exact_count: std::sync::atomic::AtomicUsize::new(0), read_exact_requests: std::sync::Mutex::new(Vec::new()), + armed_fence: std::sync::Mutex::new(None), + latched_fence: std::sync::OnceLock::new(), } } + fn arm_fence(&self, fence: OplogFence) { + *self.armed_fence.lock().unwrap() = Some(fence); + } + fn read_exact_count(&self) -> usize { self.read_exact_count .load(std::sync::atomic::Ordering::Relaxed) @@ -2932,7 +3059,7 @@ mod tests { entered.notify_one(); release.notified().await; } - result + Ok(result) }) } @@ -2940,7 +3067,7 @@ mod tests { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), crate::services::oplog::OplogError> { let mut entries = self.entries.lock().unwrap(); let mut idx = self.current_idx.lock().unwrap(); *idx = idx.next(); @@ -2949,13 +3076,13 @@ mod tests { *idx = idx.next(); let second_idx = *idx; entries.push(make_second(first_idx)); - (first_idx, second_idx) + Ok((first_idx, second_idx)) } async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { let mut entries = self.entries.lock().unwrap(); let mut idx = self.current_idx.lock().unwrap(); let records = make_batch(idx.next()); @@ -2973,9 +3100,9 @@ mod tests { &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { let entry = build_start(RawOplogPayload::SerializedInline(serialized_request))?; - let index = self.add(entry.clone()).await; + let index = self.add(entry.clone()).await?; Ok(OrderedOplogStart { index, entry, @@ -2986,7 +3113,7 @@ mod tests { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let mut entries = self.entries.lock().unwrap(); let mut idx = self.current_idx.lock().unwrap(); let index = idx.next(); @@ -3005,7 +3132,17 @@ mod tests { 0 } - async fn commit(&self, _level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + _level: CommitLevel, + ) -> Result, crate::services::oplog::OplogError> { + if let Some(fence) = self.latched_fence.get() { + return Err(OplogError::Fenced(fence.clone())); + } + if let Some(fence) = self.armed_fence.lock().unwrap().take() { + let _ = self.latched_fence.set(fence.clone()); + return Err(OplogError::Fenced(fence)); + } let gate = if matches!( self.entries.lock().unwrap().last(), Some(OplogEntry::OplogProcessorCheckpoint { .. }) @@ -3028,7 +3165,7 @@ mod tests { result.insert(idx, entries[(idx.as_u64() - 1) as usize].clone()); } *committed = current; - result + Ok(result) } async fn current_oplog_index(&self) -> OplogIndex { @@ -3084,6 +3221,10 @@ mod tests { ) -> Result, String> { unimplemented!() } + + fn fence(&self) -> Option { + self.latched_fence.get().cloned() + } } fn test_worker_metadata( @@ -3195,6 +3336,91 @@ mod tests { ); } + // -------------------------------------------------------------------------- + // A fenced checkpoint ends forwarding: the flushes after it write and send nothing + // -------------------------------------------------------------------------- + + #[test] + async fn a_fenced_checkpoint_stops_later_flushes_before_they_write() { + let grant_id = EnvironmentPluginGrantId::new(); + let (metadata, status_lock) = test_worker_metadata(HashSet::from([grant_id])); + let recording_plugin = Arc::new(RecordingOplogProcessorPlugin::new()); + let components: Arc = Arc::new( + FakeComponentService::with_one_oplog_processor_plugin(grant_id), + ); + let in_memory = Arc::new(InMemoryOplog::new()); + in_memory.add(grow_memory(1)).await.unwrap(); + in_memory.add(grow_memory(2)).await.unwrap(); + in_memory.commit(CommitLevel::Always).await.unwrap(); + in_memory.arm_fence(OplogFence { + agent_id: metadata.agent_id.clone(), + expected_epoch: ShardEpoch(1), + actual_epoch: Some(ShardEpoch(2)), + writer_conflict: false, + }); + let inner: Arc = in_memory.clone(); + + // No target yet, so the first write is the checkpoint recording the resolved target. It + // is written before the grant is marked as sending, so nothing but the fence stops the + // next flush from resolving and writing it again. + let mut state = ForwardingOplogState { + forwarding_retired: CancellationToken::new(), + forwarding_activity: ActivityGate::new(), + pending_checkpoint_activity: None, + buffer: None, + buffer_start_idx: OplogIndex::from_u64(3), + commit_count: 0, + last_send: Instant::now(), + oplog_plugins: recording_plugin.clone(), + initial_worker_metadata: metadata, + last_known_status: status_lock, + last_oplog_idx: OplogIndex::from_u64(2), + last_committed_idx: OplogIndex::from_u64(2), + components, + inner, + plugin_state: HashMap::from([( + grant_id, + LivePluginState { + target_agent_id: None, + confirmed_up_to: OplogIndex::NONE, + sending_up_to: OplogIndex::NONE, + send_in_progress: false, + last_batch_start: OplogIndex::NONE, + }, + )]), + pending_direct_commits: BTreeMap::new(), + worker_event_service: None, + monitor_tasks: Vec::new(), + }; + + state.try_flush().await; + assert_eq!(recording_plugin.send_count().await, 0); + assert_eq!( + in_memory.length().await, + 3, + "the refused checkpoint is the only entry the first flush adds" + ); + let last_oplog_idx = state.last_oplog_idx; + + state.try_flush().await; + state.try_flush().await; + + assert_eq!( + in_memory.length().await, + 3, + "a flush after the fence must not add another checkpoint" + ); + assert_eq!(state.last_oplog_idx, last_oplog_idx); + assert_eq!(recording_plugin.send_count().await, 0); + assert_eq!( + recording_plugin + .resolve_count + .load(std::sync::atomic::Ordering::Relaxed), + 1, + "a flush after the fence must not resolve the target again" + ); + } + // -------------------------------------------------------------------------- // U5 (partial): No active plugins → no send even with entries in buffer // -------------------------------------------------------------------------- @@ -3262,8 +3488,8 @@ mod tests { timestamp: Timestamp::now_utc(), delta: 200, }; - inner.add(entry1.clone()).await; - inner.add(entry2.clone()).await; + inner.add(entry1.clone()).await.unwrap(); + inner.add(entry2.clone()).await.unwrap(); let mut state = ForwardingOplogState { forwarding_retired: CancellationToken::new(), @@ -3317,7 +3543,7 @@ mod tests { timestamp: Timestamp::now_utc(), delta: 100, }; - inner.add(entry.clone()).await; + inner.add(entry.clone()).await.unwrap(); let mut state = ForwardingOplogState { forwarding_retired: CancellationToken::new(), @@ -3396,9 +3622,10 @@ mod tests { timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); - assert_eq!(first.await, OplogIndex::INITIAL); + assert_eq!(first.await.unwrap(), OplogIndex::INITIAL); assert_eq!(second, OplogIndex::INITIAL.next()); } @@ -3469,7 +3696,7 @@ mod tests { second_pending.wait().await.unwrap(); // With max_commit_count = 1 the first commit triggers a flush to the plugin. - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let sends = recording_plugin.sends().await; assert_eq!(sends.len(), 1, "Expected exactly one batch"); @@ -3556,10 +3783,11 @@ mod tests { let (oplog, inner, recording_plugin, status_writer) = test_forwarding_oplog(HashSet::new(), grant_id, 1).await; - oplog.add(grow_memory(1)).await; + oplog.add(grow_memory(1)).await.unwrap(); oplog .add_pair(grow_memory(2), Box::new(|_| grow_memory(3))) - .await; + .await + .unwrap(); oplog .add_start_with_reserved_raw_payload(Vec::new(), Box::new(|_| Ok(grow_memory(4)))) .await @@ -3594,13 +3822,13 @@ mod tests { assert_eq!(uncached.buffer_start_idx, uncached.last_oplog_idx.next()); publish_status(&status_writer, HashSet::from([grant_id]), None); - oplog.add(grow_memory(8)).await; + oplog.add(grow_memory(8)).await.unwrap(); let cached = oplog.inspect_state().await; assert_eq!(cached.buffer_len, Some(1)); assert_eq!(cached.buffer_start_idx, OplogIndex::from_u64(8)); assert_eq!(cached.last_oplog_idx, OplogIndex::from_u64(8)); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let sends = recording_plugin.sends().await; assert_eq!(sends.len(), 1); assert_eq!(sends[0].initial_oplog_index, OplogIndex::INITIAL); @@ -3624,7 +3852,7 @@ mod tests { test_forwarding_oplog(HashSet::new(), grant_id, usize::MAX).await; let started = std::time::Instant::now(); for delta in 0..APPENDS { - uncached.add(grow_memory(delta)).await; + uncached.add(grow_memory(delta)).await.unwrap(); } let uncached_elapsed = started.elapsed(); assert_eq!(uncached.inspect_state().await.buffer_len, None); @@ -3633,7 +3861,7 @@ mod tests { test_forwarding_oplog(HashSet::from([grant_id]), grant_id, usize::MAX).await; let started = std::time::Instant::now(); for delta in 0..APPENDS { - cached.add(grow_memory(delta)).await; + cached.add(grow_memory(delta)).await.unwrap(); } let cached_elapsed = started.elapsed(); assert_eq!( @@ -3653,8 +3881,8 @@ mod tests { let grant_id = EnvironmentPluginGrantId::new(); let (oplog, inner, recording_plugin, status_writer) = test_forwarding_oplog(HashSet::new(), grant_id, 1).await; - oplog.add(grow_memory(1)).await; - oplog.add(grow_memory(2)).await; + oplog.add(grow_memory(1)).await.unwrap(); + oplog.add(grow_memory(2)).await.unwrap(); let target = recording_plugin.target_agent_id.clone(); publish_status( @@ -3670,8 +3898,8 @@ mod tests { }, )), ); - oplog.add(grow_memory(3)).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(grow_memory(3)).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let sends = recording_plugin.sends().await; assert_eq!(sends.len(), 1); @@ -3685,19 +3913,19 @@ mod tests { let grant_id = EnvironmentPluginGrantId::new(); let (oplog, inner, recording_plugin, status_writer) = test_forwarding_oplog(HashSet::from([grant_id]), grant_id, 1).await; - oplog.add(grow_memory(1)).await; + oplog.add(grow_memory(1)).await.unwrap(); publish_status(&status_writer, HashSet::new(), None); - oplog.add(grow_memory(2)).await; + oplog.add(grow_memory(2)).await.unwrap(); let skipped = oplog.inspect_state().await; assert_eq!(skipped.buffer_len, None); assert_eq!(skipped.buffer_start_idx, skipped.last_oplog_idx.next()); publish_status(&status_writer, HashSet::from([grant_id]), None); - oplog.add(grow_memory(3)).await; + oplog.add(grow_memory(3)).await.unwrap(); let resumed = oplog.inspect_state().await; assert_eq!(resumed.buffer_len, Some(1)); assert_eq!(resumed.buffer_start_idx, OplogIndex::from_u64(3)); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let sends = recording_plugin.sends().await; assert_eq!(sends.len(), 1); @@ -3712,7 +3940,7 @@ mod tests { let grant_id = EnvironmentPluginGrantId::new(); let (oplog, _inner, _recording_plugin, status_writer) = test_forwarding_oplog(HashSet::from([grant_id]), grant_id, usize::MAX).await; - oplog.add(grow_memory(1)).await; + oplog.add(grow_memory(1)).await.unwrap(); assert_eq!(oplog.inspect_state().await.buffer_len, Some(1)); publish_status(&status_writer, HashSet::new(), None); @@ -3765,13 +3993,13 @@ mod tests { let (oplog, _inner, recording_plugin, status_writer) = test_forwarding_oplog(HashSet::from([grant_id]), grant_id, 1).await; recording_plugin.fail_next_send_from_remote_target(); - oplog.add(grow_memory(1)).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(grow_memory(1)).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); publish_status(&status_writer, HashSet::new(), None); - oplog.add(grow_memory(2)).await; + oplog.add(grow_memory(2)).await.unwrap(); assert!(oplog.inspect_state().await.buffer_len.is_some()); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let sends = recording_plugin.sends().await; assert_eq!(sends.len(), 2); @@ -3796,8 +4024,8 @@ mod tests { let grant_id = EnvironmentPluginGrantId::new(); let (oplog, _inner, _recording_plugin, _status_writer) = test_forwarding_oplog(HashSet::from([grant_id]), grant_id, usize::MAX).await; - oplog.add(grow_memory(1)).await; - let initial_commit = oplog.commit(CommitLevel::Always).await; + oplog.add(grow_memory(1)).await.unwrap(); + let initial_commit = oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!(initial_commit.len(), 1); oplog.tick().await; @@ -3806,7 +4034,7 @@ mod tests { assert_eq!(pruned.buffer_len, None); assert_eq!(pruned.buffer_start_idx, pruned.last_oplog_idx.next()); - let checkpoint_commit = oplog.commit(CommitLevel::Always).await; + let checkpoint_commit = oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( checkpoint_commit .values() @@ -3814,6 +4042,6 @@ mod tests { .count(), 3 ); - assert!(oplog.commit(CommitLevel::Always).await.is_empty()); + assert!(oplog.commit(CommitLevel::Always).await.unwrap().is_empty()); } } diff --git a/golem-worker-executor/src/services/oplog/primary.rs b/golem-worker-executor/src/services/oplog/primary.rs index 5671eb73d8..5ab7e6ae4e 100644 --- a/golem-worker-executor/src/services/oplog/primary.rs +++ b/golem-worker-executor/src/services/oplog/primary.rs @@ -12,7 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::metrics::oplog::{record_oplog_call, record_oplog_storage_retry}; +use crate::metrics::oplog::{ + record_oplog_call, record_oplog_epoch_fence, record_oplog_storage_retry, +}; use crate::metrics::storage::{ STORAGE_TYPE_OPLOG, record_storage_bytes_written, record_storage_objects_deleted, record_storage_objects_written, @@ -23,9 +25,10 @@ use crate::services::oplog::reader::{ }; use crate::services::oplog::{ CommitLevel, DurableStreamBatchBuilder, IndexedReservedStartBuilder, OpenOplogs, Oplog, - OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogLifecycleGuard, OplogService, - OrderedOplogStart, PendingUpload, ReservedPayload, ReservedRawStartBuilder, decode_scan_cursor, - next_scan_cursor, retry_scan_storage_op, + OplogAddReceipt, OplogCloseCompletion, OplogConstructor, OplogError, OplogFence, + OplogFenceObserver, OplogLifecycleGuard, OplogService, OrderedOplogStart, PendingUpload, + ReservedPayload, ReservedRawStartBuilder, decode_scan_cursor, next_scan_cursor, + retry_scan_storage_op, }; use crate::storage::indexed::{ IndexedStorage, IndexedStorageError, IndexedStorageLabelledApi, IndexedStorageMetaNamespace, @@ -35,6 +38,7 @@ use async_trait::async_trait; use bytes::Bytes; use futures::FutureExt; use golem_common::model::RetryConfig; +use golem_common::model::ShardEpoch; use golem_common::model::account::AccountId; use golem_common::model::agent::AgentMode; use golem_common::model::component::ComponentId; @@ -60,12 +64,39 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; use tracing::{error, warn}; +/// Runs a storage operation under the retry policy, panicking on anything it cannot retry away. +/// +/// Reads, deletions and prefix drops keep this shape: a permanent failure there is a broken +/// deployment, and failing fast is the long-standing contract. async fn retry_storage_op( retry_config: &RetryConfig, op_name: &str, key: &str, - mut op: F, + op: F, ) -> T +where + F: FnMut() -> Fut, + Fut: std::future::Future>, +{ + match retry_storage_op_fenceable(retry_config, op_name, key, op).await { + Ok(val) => val, + Err(err) => panic!("Indexed storage operation '{op_name}' failed for key '{key}': {err}"), + } +} + +/// As [`retry_storage_op`], but hands a fence back instead of panicking on it. +/// +/// A fenced write is not a storage failure: the storage is healthy and refused the write on +/// purpose, because this executor no longer owns the agent's shard. Retrying cannot change that, +/// and panicking would take the whole executor down over one agent that simply moved. Every other +/// permanent failure still panics, so the fail-stop contract is unchanged for everything else - +/// including the primary-key collision that has always been the crude fence. +async fn retry_storage_op_fenceable( + retry_config: &RetryConfig, + op_name: &str, + key: &str, + mut op: F, +) -> Result where F: FnMut() -> Fut, Fut: std::future::Future>, @@ -74,7 +105,8 @@ where loop { attempts += 1; match op().await { - Ok(val) => return val, + Ok(val) => return Ok(val), + Err(err @ IndexedStorageError::Fenced { .. }) => return Err(err), Err(IndexedStorageError::Transient(msg)) => { if let Some(delay) = get_delay(retry_config, attempts) { record_oplog_storage_retry(op_name); @@ -156,17 +188,18 @@ impl SerializedOplogAppend { namespace: &IndexedStorageNamespace, api_name: &'static str, key: &str, + shard_epoch: Option, ) -> Result<(), IndexedStorageError> { let storage = indexed_storage.with_entity("oplog", api_name, "entry"); match self { Self::Entry((id, value)) => { storage - .append_raw(namespace.clone(), key, *id, value.to_vec()) + .append_raw(namespace.clone(), key, *id, value.to_vec(), shard_epoch) .await } Self::Batch(entries) => { storage - .append_many_raw(namespace, key, entries.clone()) + .append_many_raw(namespace, key, entries.clone(), shard_epoch) .await } } @@ -181,19 +214,27 @@ async fn retry_oplog_append( api_name: &'static str, key: &str, append: SerializedOplogAppend, -) { + shard_epoch: Option, +) -> Result<(), IndexedStorageError> { let mut attempts = 0u32; let mut write_may_have_committed = false; loop { attempts += 1; let error = match append - .write(indexed_storage, namespace, api_name, key) + .write(indexed_storage, namespace, api_name, key, shard_epoch) .await { - Ok(()) => return, + Ok(()) => return Ok(()), Err(error) => error, }; + // The storage refused the write because the shard has a new owner. Not a transient + // failure to retry and not an indeterminate one to reconcile: it is a deliberate refusal, + // so hand it back and let the caller give up this one agent instead of aborting. + if matches!(error, IndexedStorageError::Fenced { .. }) { + return Err(error); + } + let retryable = match &error { IndexedStorageError::Indeterminate(_) => { write_may_have_committed = true; @@ -214,6 +255,8 @@ async fn retry_oplog_append( } false } + // Returned above; named here only because the guard does not make this exhaustive. + IndexedStorageError::Fenced { .. } => unreachable!("a fence returns before this match"), }; if write_may_have_committed { @@ -226,10 +269,26 @@ async fn retry_oplog_append( ) .await { - Some(true) => return, - Some(false) => panic!( - "Indexed storage operation '{op_name}' failed for key '{key}' and the indeterminate write did not match storage: {error}" - ), + Some(true) => return Ok(()), + Some(false) => { + // The stored content differs from what this attempt sent - the only + // legitimate way that happens is a new owner having already written those + // same indices. Repeat the write once as a probe: the backends check the + // epoch inside the same transaction as the insert, so a shard that has + // moved on is fenced before the insert is even attempted. A same-epoch + // conflict instead fails the probe's insert (still fatal, below) - the + // mismatch is unexplained and not safe to paper over. + if let Some(epoch) = shard_epoch + && let Err(fenced @ IndexedStorageError::Fenced { .. }) = append + .write(indexed_storage, namespace, api_name, key, Some(epoch)) + .await + { + return Err(fenced); + } + panic!( + "Indexed storage operation '{op_name}' failed for key '{key}' and the indeterminate write did not match storage: {error}" + ) + } None => {} } } @@ -260,6 +319,81 @@ async fn retry_oplog_append( } } +/// Records the epoch this executor is allowed to write `key` with, and reports the fence when +/// the stored record is already ahead of it. +/// +/// Monotonic on the storage side once a record exists, so a re-grant at a higher epoch takes the +/// oplog over while an executor holding a stale one cannot claim it back. An oplog with no record +/// (new, deleted, or from before the record existed) is claimed by whichever epoch opens it +/// first. Written before the oplog's first entry - +/// an absent record fences too, which is what closes the window between creating an oplog and +/// recording who owns it. +/// +/// A refusal is also handed to `fence_observer`, because the epoch it carries is what a shard +/// manager whose state lost history has to mint above. +/// Whether an open still has to record the epoch it asserts, or its caller did so already. +#[derive(Clone)] +enum EpochRecord { + /// The open records it, and takes the refusal it gets. + Pending, + /// A create recorded it before the first entry went in, with this refusal. + Recorded(Option), +} + +async fn record_owning_epoch( + indexed_storage: &(dyn IndexedStorage + Send + Sync), + retry_config: &RetryConfig, + owned_agent_id: &OwnedAgentId, + agent_mode: AgentMode, + key: &str, + shard_epoch: ShardEpoch, + fence_observer: Option<&dyn OplogFenceObserver>, +) -> Option { + let outcome = retry_storage_op_fenceable(retry_config, "set_key_epoch", key, || { + let ns = IndexedStorageNamespace::OpLog { + agent_id: owned_agent_id.agent_id(), + agent_mode, + }; + async move { + indexed_storage + .set_key_epoch("oplog", "set_key_epoch", ns, key, shard_epoch) + .await + } + }) + .await; + + record_oplog_epoch_fence("record", outcome.is_err()); + match outcome { + Ok(()) => None, + Err(IndexedStorageError::Fenced { + expected, + actual, + writer_conflict, + .. + }) => { + warn!( + agent_id = %owned_agent_id, + expected_epoch = expected.0, + actual_epoch = ?actual.map(|epoch| epoch.0), + writer_conflict, + "Oplog opened at a stale shard epoch: the shard has a new owner" + ); + let fence = OplogFence { + agent_id: owned_agent_id.agent_id(), + expected_epoch: expected, + actual_epoch: actual, + writer_conflict, + }; + if let Some(observer) = fence_observer { + observer.fenced(&fence); + } + Some(fence) + } + // `retry_storage_op_fenceable` panics on every other permanent failure. + Err(other) => unreachable!("unexpected storage error: {other}"), + } +} + async fn read_persisted_oplog_entries( indexed_storage: Arc, namespace: IndexedStorageNamespace, @@ -289,7 +423,7 @@ async fn read_persisted_oplog_entries( /// /// Stores and retrieves individual oplog entries from the `IndexedStorage` implementation configured for /// the executor. -#[derive(Clone, Debug)] +#[derive(Clone)] pub struct PrimaryOplogService { indexed_storage: Arc, blob_storage: Arc, @@ -300,6 +434,32 @@ pub struct PrimaryOplogService { retry_config: RetryConfig, oplogs: OpenOplogs, stream_session_index: Arc>>, + /// Told of every refusal the storage returns for an oplog this service opened, so the epochs + /// the refusals carry can reach the shard manager. `None` reports nothing. + fence_observer: Option>, +} + +impl Debug for PrimaryOplogService { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PrimaryOplogService") + .field("indexed_storage", &self.indexed_storage) + .field("blob_storage", &self.blob_storage) + .field("replicas", &self.replicas) + .field( + "max_operations_before_commit", + &self.max_operations_before_commit, + ) + .field( + "max_operations_before_commit_ephemeral", + &self.max_operations_before_commit_ephemeral, + ) + .field("max_payload_size", &self.max_payload_size) + .field("retry_config", &self.retry_config) + .field("oplogs", &self.oplogs) + .field("stream_session_index", &self.stream_session_index) + .field("fence_observer", &self.fence_observer.is_some()) + .finish() + } } impl PrimaryOplogService { @@ -326,9 +486,17 @@ impl PrimaryOplogService { retry_config, oplogs: OpenOplogs::new("primary oplog"), stream_session_index: Arc::new(std::sync::OnceLock::new()), + fence_observer: None, } } + /// Reports every refusal the storage returns for an oplog this service opens to `observer`, + /// with the epoch recorded on the oplog. + pub fn with_fence_observer(mut self, observer: Arc) -> Self { + self.fence_observer = Some(observer); + self + } + fn oplog_key(agent_id: &AgentId) -> String { agent_id.to_redis_key() } @@ -362,6 +530,7 @@ impl PrimaryOplogService { op_name: &str, api_name: &'static str, entry: &OplogEntry, + shard_epoch: Option, ) { let key = Self::oplog_key(&owned_agent_id.agent_id); let namespace = IndexedStorageNamespace::OpLog { @@ -381,8 +550,65 @@ impl PrimaryOplogService { api_name, &key, SerializedOplogAppend::Entry((1, value)), + shard_epoch, ) - .await; + .await + .unwrap_or_else(|err| { + // Only a fence reaches here - every other permanent failure already panicked inside + // `retry_oplog_append`. Nothing more is written: `open` records the epoch again and, + // being refused there too, hands back an oplog that refuses every write. + warn!( + agent_id = %owned_agent_id, + error = %err, + "Initial oplog entry fenced: the shard has a new owner" + ); + }); + } + + async fn open_with( + &self, + lifecycle: &mut OplogLifecycleGuard, + owned_agent_id: &OwnedAgentId, + agent_mode: AgentMode, + last_oplog_index: Option, + initial_worker_metadata: AgentMetadata, + shard_epoch: Option, + epoch_record: EpochRecord, + reconcile_last_index: bool, + ) -> Arc { + record_oplog_call("open"); + + let key = Self::oplog_key(&owned_agent_id.agent_id); + let max_operations_before_commit = match agent_mode { + AgentMode::Durable => self.max_operations_before_commit, + AgentMode::Ephemeral => self.max_operations_before_commit_ephemeral, + }; + + self.oplogs + .get_or_open( + lifecycle, + &owned_agent_id.agent_id, + CreateOplogConstructor::new( + shard_epoch, + epoch_record, + self.indexed_storage.clone(), + self.blob_storage.clone(), + self.replicas, + max_operations_before_commit, + self.max_payload_size, + self.retry_config.clone(), + key, + last_oplog_index, + reconcile_last_index, + owned_agent_id.clone(), + agent_mode, + initial_worker_metadata.created_by, + initial_worker_metadata.fingerprint, + self.stream_session_index(), + self.fence_observer.clone(), + ), + ) + .await } async fn get_last_index_from_storage( @@ -513,6 +739,10 @@ impl OplogService for PrimaryOplogService { let key = Self::staged_oplog_key(&owned_agent_id.agent_id, stage_id); let namespace = Self::staged_namespace(&owned_agent_id.agent_id, agent_mode); Ok(Arc::new(PrimaryOplog::new( + // A stage is hidden and has exactly one writer, so it asserts no epoch and can never + // be fenced: it is published into the target only once it is fully committed. + None, + None, self.indexed_storage.clone(), self.blob_storage.clone(), self.replicas, @@ -527,6 +757,7 @@ impl OplogService for PrimaryOplogService { initial_worker_metadata.created_by, initial_worker_metadata.fingerprint, None, + None, Box::new(|| {}), ))) } @@ -607,50 +838,78 @@ impl OplogService for PrimaryOplogService { agent_mode: AgentMode, initial_entry: OplogEntry, initial_worker_metadata: AgentMetadata, - last_known_status: read_only_lock::arc_swap::ReadOnlyView, - execution_status: read_only_lock::std::ReadOnlyLock, + _last_known_status: read_only_lock::arc_swap::ReadOnlyView, + _execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { record_oplog_call("create"); lifecycle.assert_agent(&owned_agent_id.agent_id); let key = Self::oplog_key(&owned_agent_id.agent_id); - let already_exists: bool = { - let is = self.indexed_storage.clone(); - let agent_id = owned_agent_id.agent_id(); - let key = key.clone(); - retry_storage_op(&self.retry_config, "create_exists", &key, || { - let is = is.clone(); - let ns = IndexedStorageNamespace::OpLog { - agent_id: agent_id.clone(), + + // The record goes in before the existence probe and the first entry. A probe taken before + // the claim can miss a `Create` that an executor at an older epoch lands in between, and + // the initial append would then collide with it. If the claim is refused, this executor + // has already lost the shard: it writes nothing, so whether the owner created the oplog + // first is not its question, and `open` below hands back an oplog that refuses every write. + let fence_at_create = match shard_epoch { + Some(epoch) => { + record_owning_epoch( + &*self.indexed_storage, + &self.retry_config, + owned_agent_id, agent_mode, - }; - let key = key.clone(); - async move { is.with("oplog", "create").exists(ns, &key).await } - }) - .await + &key, + epoch, + self.fence_observer.as_deref(), + ) + .await + } + None => None, }; - if already_exists { - panic!("oplog for worker {owned_agent_id} already exists in indexed storage") - } + if fence_at_create.is_none() { + let already_exists: bool = { + let is = self.indexed_storage.clone(); + let agent_id = owned_agent_id.agent_id(); + let key = key.clone(); + retry_storage_op(&self.retry_config, "create_exists", &key, || { + let is = is.clone(); + let ns = IndexedStorageNamespace::OpLog { + agent_id: agent_id.clone(), + agent_mode, + }; + let key = key.clone(); + async move { is.with("oplog", "create").exists(ns, &key).await } + }) + .await + }; - self.append_initial_entry( - owned_agent_id, - agent_mode, - "create_append", - "create", - &initial_entry, - ) - .await; + if already_exists { + panic!("oplog for worker {owned_agent_id} already exists in indexed storage") + } - self.open( + self.append_initial_entry( + owned_agent_id, + agent_mode, + "create_append", + "create", + &initial_entry, + shard_epoch, + ) + .await; + } + + // The claim came before the initial entry, so `INITIAL` is exact and needs no re-read. + self.open_with( lifecycle, owned_agent_id, agent_mode, Some(OplogIndex::INITIAL), initial_worker_metadata, - last_known_status, - execution_status, + shard_epoch, + EpochRecord::Recorded(fence_at_create), + false, ) .await } @@ -662,32 +921,57 @@ impl OplogService for PrimaryOplogService { agent_mode: AgentMode, initial_entry: OplogEntry, initial_worker_metadata: AgentMetadata, - last_known_status: read_only_lock::arc_swap::ReadOnlyView, - execution_status: read_only_lock::std::ReadOnlyLock, + _last_known_status: read_only_lock::arc_swap::ReadOnlyView, + _execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { record_oplog_call("create_fresh"); lifecycle.assert_agent(&owned_agent_id.agent_id); // The caller guarantees the agent id is freshly derived and unused, so // the existence probe performed by `create` is skipped: the initial - // entry is appended directly without any prior read. - self.append_initial_entry( - owned_agent_id, - agent_mode, - "create_fresh_append", - "create_fresh", - &initial_entry, - ) - .await; + // entry is appended directly without any prior read. The epoch record still goes in + // first - a fresh agent id does not mean a fresh shard. + let key = Self::oplog_key(&owned_agent_id.agent_id); + let fence_at_create = match shard_epoch { + Some(epoch) => { + record_owning_epoch( + &*self.indexed_storage, + &self.retry_config, + owned_agent_id, + agent_mode, + &key, + epoch, + self.fence_observer.as_deref(), + ) + .await + } + None => None, + }; + + if fence_at_create.is_none() { + self.append_initial_entry( + owned_agent_id, + agent_mode, + "create_fresh_append", + "create_fresh", + &initial_entry, + shard_epoch, + ) + .await; + } - self.open( + // Claimed before the initial entry, so `INITIAL` is exact; not re-reading it keeps a fresh + // create free of storage reads. + self.open_with( lifecycle, owned_agent_id, agent_mode, Some(OplogIndex::INITIAL), initial_worker_metadata, - last_known_status, - execution_status, + shard_epoch, + EpochRecord::Recorded(fence_at_create), + false, ) .await } @@ -701,36 +985,21 @@ impl OplogService for PrimaryOplogService { initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { - record_oplog_call("open"); - - let key = Self::oplog_key(&owned_agent_id.agent_id); - let max_operations_before_commit = match agent_mode { - AgentMode::Durable => self.max_operations_before_commit, - AgentMode::Ephemeral => self.max_operations_before_commit_ephemeral, - }; - - self.oplogs - .get_or_open( - lifecycle, - &owned_agent_id.agent_id, - CreateOplogConstructor::new( - self.indexed_storage.clone(), - self.blob_storage.clone(), - self.replicas, - max_operations_before_commit, - self.max_payload_size, - self.retry_config.clone(), - key, - last_oplog_index, - owned_agent_id.clone(), - agent_mode, - initial_worker_metadata.created_by, - initial_worker_metadata.fingerprint, - self.stream_session_index(), - ), - ) - .await + // An index handed in by a caller was read before this open claims the epoch. + let reconcile_last_index = last_oplog_index.is_some(); + self.open_with( + lifecycle, + owned_agent_id, + agent_mode, + last_oplog_index, + initial_worker_metadata, + shard_epoch, + EpochRecord::Pending, + reconcile_last_index, + ) + .await } async fn get_last_index( @@ -753,24 +1022,58 @@ impl OplogService for PrimaryOplogService { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, - ) { + expected_epoch: Option, + ) -> Result<(), OplogError> { record_oplog_call("delete"); lifecycle.assert_agent(&owned_agent_id.agent_id); - { - let is = self.indexed_storage.clone(); - let agent_id = owned_agent_id.agent_id(); - let key = Self::oplog_key(&owned_agent_id.agent_id); - retry_storage_op(&self.retry_config, "delete", &key, || { - let is = is.clone(); - let ns = IndexedStorageNamespace::OpLog { - agent_id: agent_id.clone(), - agent_mode, + let is = self.indexed_storage.clone(); + let agent_id = owned_agent_id.agent_id(); + let key = Self::oplog_key(&owned_agent_id.agent_id); + // The entries and the epoch record go in one step, and only while the record is still + // this executor's: a delete that outlived the agent's shard is refused like a write, and + // leaves the new owner's oplog alone. + let outcome = retry_storage_op_fenceable(&self.retry_config, "delete", &key, || { + let is = is.clone(); + let ns = IndexedStorageNamespace::OpLog { + agent_id: agent_id.clone(), + agent_mode, + }; + let key = key.clone(); + async move { + is.delete_with_epoch("oplog", "delete", ns, &key, expected_epoch) + .await + } + }) + .await; + match outcome { + Ok(()) => Ok(()), + Err(IndexedStorageError::Fenced { + expected, + actual, + writer_conflict, + .. + }) => { + warn!( + agent_id = %owned_agent_id, + expected_epoch = expected.0, + actual_epoch = ?actual.map(|epoch| epoch.0), + writer_conflict, + "Oplog delete refused: the shard has a new owner" + ); + let fence = OplogFence { + agent_id, + expected_epoch: expected, + actual_epoch: actual, + writer_conflict, }; - let key = key.clone(); - async move { is.with("oplog", "delete").delete(ns, &key).await } - }) - .await; + if let Some(observer) = &self.fence_observer { + observer.fenced(&fence); + } + Err(OplogError::Fenced(fence)) + } + // `retry_storage_op_fenceable` panics on every other permanent failure. + Err(other) => unreachable!("unexpected storage error: {other}"), } } @@ -940,16 +1243,24 @@ struct CreateOplogConstructor { retry_config: RetryConfig, key: String, last_oplog_idx: Option, + /// `last_oplog_idx` was read before this constructor claims the epoch, so it may be behind + /// entries an executor at an older epoch committed in between. + reconcile_last_index: bool, owned_agent_id: OwnedAgentId, agent_mode: AgentMode, account_id: AccountId, fingerprint: AgentFingerprint, stream_session_index: Option>, + shard_epoch: Option, + epoch_record: EpochRecord, + fence_observer: Option>, } impl CreateOplogConstructor { #[allow(clippy::too_many_arguments)] fn new( + shard_epoch: Option, + epoch_record: EpochRecord, indexed_storage: Arc, blob_storage: Arc, replicas: u8, @@ -958,13 +1269,17 @@ impl CreateOplogConstructor { retry_config: RetryConfig, key: String, last_oplog_idx: Option, + reconcile_last_index: bool, owned_agent_id: OwnedAgentId, agent_mode: AgentMode, account_id: AccountId, fingerprint: AgentFingerprint, stream_session_index: Option>, + fence_observer: Option>, ) -> Self { Self { + shard_epoch, + epoch_record, indexed_storage, blob_storage, replicas, @@ -973,35 +1288,74 @@ impl CreateOplogConstructor { retry_config, key, last_oplog_idx, + reconcile_last_index, owned_agent_id, agent_mode, account_id, fingerprint, stream_session_index, + fence_observer, } } } #[async_trait] impl OplogConstructor for CreateOplogConstructor { + fn shard_epoch(&self) -> Option { + self.shard_epoch + } + async fn create_oplog( self, _lifecycle: &mut OplogLifecycleGuard, close: Box, ) -> Arc { - let last_oplog_idx = match self.last_oplog_idx { - Some(idx) => idx, - None => { - PrimaryOplogService::get_last_index_from_storage( + // Recorded before the oplog is usable, so an executor whose shard has moved is refused + // at its very first write rather than after replaying the new owner's entries. + let fence = match (self.epoch_record, self.shard_epoch) { + (EpochRecord::Recorded(fence), _) => fence, + (EpochRecord::Pending, Some(shard_epoch)) => { + record_owning_epoch( &*self.indexed_storage, + &self.retry_config, &self.owned_agent_id, self.agent_mode, - &self.retry_config, + &self.key, + shard_epoch, + self.fence_observer.as_deref(), ) .await } + (EpochRecord::Pending, None) => None, + }; + + // Read after the claim: once it returns, every writer at an older epoch is refused, so the + // read sees every entry that will ever precede this handle's first write. An index a + // caller read before the claim can be behind by whatever a losing executor committed in + // between, and this handle's first append would collide with it. That index is merged + // rather than replaced, because it may count entries this layer no longer holds, such as + // ones already moved to an archive. + let stored_last_index = || { + PrimaryOplogService::get_last_index_from_storage( + &*self.indexed_storage, + &self.owned_agent_id, + self.agent_mode, + &self.retry_config, + ) + }; + let last_oplog_idx = match self.last_oplog_idx { + None => stored_last_index().await, + Some(idx) + if self.reconcile_last_index && self.shard_epoch.is_some() && fence.is_none() => + { + OplogIndex::from_u64(idx.as_u64().max(stored_last_index().await.as_u64())) + } + Some(idx) => idx, }; + Arc::new(PrimaryOplog::new( + self.shard_epoch, + fence, self.indexed_storage, self.blob_storage, self.replicas, @@ -1016,6 +1370,7 @@ impl OplogConstructor for CreateOplogConstructor { self.account_id, self.fingerprint, self.stream_session_index, + self.fence_observer, close, )) } @@ -1061,6 +1416,11 @@ struct PrimaryOplog { owned_agent_id: OwnedAgentId, agent_mode: AgentMode, fingerprint: AgentFingerprint, + /// The epoch the actor's state asserts on every append, copied here so that reading it does + /// not have to go through the actor. Fixed for the oplog's lifetime. + shard_epoch: Option, + /// The refusal the actor's state has latched, shared so the handle can report it. + fence: Arc>, stream_session_index: Option>, close: Mutex>>, } @@ -1073,29 +1433,29 @@ enum OplogJob { Close, Add { entry: OplogEntry, - done: tokio::sync::oneshot::Sender, + done: tokio::sync::oneshot::Sender>, }, AddDurableStreamBatch { make_batch: DurableStreamBatchBuilder, - done: tokio::sync::oneshot::Sender, String>>, + done: tokio::sync::oneshot::Sender, OplogError>>, }, AddPair { start: OplogEntry, make_second: Box OplogEntry + Send>, - done: tokio::sync::oneshot::Sender<(OplogIndex, OplogIndex)>, + done: tokio::sync::oneshot::Sender>, }, AddStart { serialized_request: Vec, build_start: ReservedRawStartBuilder, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender>, }, AddIndexedStart { build_request: IndexedReservedStartBuilder, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender>, }, Commit { level: CommitLevel, - done: tokio::sync::oneshot::Sender>, + done: tokio::sync::oneshot::Sender, OplogError>>, }, Flush { done: tokio::sync::oneshot::Sender<()>, @@ -1160,6 +1520,8 @@ impl Drop for PrimaryOplog { impl PrimaryOplog { #[allow(clippy::too_many_arguments)] fn new( + shard_epoch: Option, + fence: Option, indexed_storage: Arc, blob_storage: Arc, replicas: u8, @@ -1174,11 +1536,19 @@ impl PrimaryOplog { account_id: AccountId, fingerprint: AgentFingerprint, stream_session_index: Option>, + fence_observer: Option>, close: Box, ) -> Self { let account_id_label = account_id.to_string(); let environment_id_label = owned_agent_id.environment_id().to_string(); + let fence = Arc::new(match fence { + Some(fence) => std::sync::OnceLock::from(fence), + None => std::sync::OnceLock::new(), + }); let mut state = PrimaryOplogState { + shard_epoch, + fence: fence.clone(), + fence_observer, indexed_storage, blob_storage, replicas, @@ -1212,14 +1582,26 @@ impl PrimaryOplog { OplogJob::Close => break, OplogJob::Add { entry, done } => { record_oplog_call("add"); - let idx = state.push(entry); - if state.over_commit_threshold() { - state.commit(CommitLevel::Always).await; + if let Err(error) = state.refuse_if_fenced() { + let _ = done.send(Err(error)); + continue; } - let _ = done.send(idx); + let idx = state.push(entry); + // A threshold commit failing must fail the `add` that triggered it: the + // caller would otherwise be told its entry landed when the batch it was + // folded into was refused. + let result = match state.maybe_commit().await { + Ok(()) => Ok(idx), + Err(error) => Err(error), + }; + let _ = done.send(result); } OplogJob::AddDurableStreamBatch { make_batch, done } => { record_oplog_call("add_durable_stream_batch"); + if let Err(error) = state.refuse_if_fenced() { + let _ = done.send(Err(error)); + continue; + } let first_index = state.last_oplog_idx.next(); let records = make_batch(first_index); let serialized = records @@ -1244,9 +1626,11 @@ impl PrimaryOplog { } Ok(result) }); - if result.is_ok() && state.over_commit_threshold() { - state.commit(CommitLevel::Always).await; - } + let result = match (result, state.maybe_commit().await) { + (Ok(value), Ok(())) => Ok(value), + (Err(error), _) => Err(error.into()), + (Ok(_), Err(error)) => Err(error), + }; let _ = done.send(result); } OplogJob::AddPair { @@ -1255,13 +1639,18 @@ impl PrimaryOplog { done, } => { record_oplog_call("add_pair"); + if let Err(error) = state.refuse_if_fenced() { + let _ = done.send(Err(error)); + continue; + } let first_idx = state.push(start); let second = make_second(first_idx); let second_idx = state.push(second); - if state.over_commit_threshold() { - state.commit(CommitLevel::Always).await; - } - let _ = done.send((first_idx, second_idx)); + // Both halves of the pair share the threshold commit, so a refused + // commit fails the pair rather than reporting a write that was rolled + // back. + let result = state.maybe_commit().await.map(|()| (first_idx, second_idx)); + let _ = done.send(result); } OplogJob::AddStart { serialized_request, @@ -1282,6 +1671,13 @@ impl PrimaryOplog { // `guard`: this actor future must stay `Send` for `tokio::spawn`, so a // refactor holding the guard across an `.await` is rejected rather than // silently breaking ordering. Do not move `drop(guard)` before `push`. + // + // A fenced oplog refuses before reserving, so no upload is started for a + // `Start` that can never be written. + if let Err(error) = state.refuse_if_fenced() { + let _ = done.send(Err(error)); + continue; + } let result = { let ReservedPayload { raw, @@ -1301,9 +1697,11 @@ impl PrimaryOplog { Err(err) => Err(err), } }; - if result.is_ok() && state.over_commit_threshold() { - state.commit(CommitLevel::Always).await; - } + let result = match (result, state.maybe_commit().await) { + (Ok(value), Ok(())) => Ok(value), + (Err(error), _) => Err(error.into()), + (Ok(_), Err(error)) => Err(error), + }; let _ = done.send(result); } OplogJob::AddIndexedStart { @@ -1311,6 +1709,10 @@ impl PrimaryOplog { done, } => { record_oplog_call("add_start_with_indexed_reserved_raw_payload"); + if let Err(error) = state.refuse_if_fenced() { + let _ = done.send(Err(error)); + continue; + } let result = build_request(state.last_oplog_idx.next()).and_then( |(serialized_request, build_start)| { let ReservedPayload { @@ -1328,21 +1730,36 @@ impl PrimaryOplog { }) }, ); - if result.is_ok() && state.over_commit_threshold() { - state.commit(CommitLevel::Always).await; - } + let result = match (result, state.maybe_commit().await) { + (Ok(value), Ok(())) => Ok(value), + (Err(error), _) => Err(error.into()), + (Ok(_), Err(error)) => Err(error), + }; let _ = done.send(result); } OplogJob::Commit { level, done } => { let previously_committed_through = state.last_committed_idx; - let committed = state.commit(level).await; - let result = state - .committed_since_last_report(previously_committed_through, committed) - .await; + let result = match state.commit(level).await { + Ok(committed) => Ok(state + .committed_since_last_report( + previously_committed_through, + committed, + ) + .await), + Err(error) => Err(error), + }; let _ = done.send(result); } OplogJob::Flush { done } => { - state.commit(CommitLevel::Always).await; + // The job has no error to reply with. A fence is latched on the state, + // where `wait_for_replicas` reads it after this job, so a fenced flush is + // reported as not durable rather than lost; every subsequent write fails on + // it without asking the storage again. A transient storage failure is fatal + // here as everywhere else. + match state.commit(CommitLevel::Always).await { + Ok(_) | Err(OplogError::Fenced(_)) => {} + Err(error) => panic!("oplog write: {error}"), + } let _ = done.send(()); } OplogJob::DropPrefix { @@ -1443,6 +1860,8 @@ impl PrimaryOplog { owned_agent_id, agent_mode, fingerprint, + shard_epoch, + fence, stream_session_index, close: Mutex::new(Some(close)), } @@ -1625,6 +2044,21 @@ struct PrimaryOplogState { /// any buffered entries, so no committed entry can reference a not-yet-written blob. pending_uploads: Vec, durable_stream_sessions: super::raw_session::RawSessionCache, + /// The shard epoch this executor held for the agent's shard when the oplog was opened, and + /// the one every append asserts. + /// + /// Cached at open rather than read per write: one live oplog is one ownership generation, and + /// this is the value its metadata row was written with. A renewal never changes it - an epoch + /// only moves when the shard changes owner, and then this oplog is the losing side. + shard_epoch: Option, + /// Set once a write has been refused, or at open when the epoch record already belonged to a + /// newer owner. Every later write fails on it immediately: the oplog is another executor's + /// now, so there is nothing to be gained by asking the storage again. Shared with the handle, + /// which answers [`Oplog::fence`] from it without a round trip through the actor. + fence: Arc>, + /// Told of the refusal that sets [`Self::fence`]; the latched fast-fail asks the storage + /// nothing and reports nothing. + fence_observer: Option>, } impl PrimaryOplogState { @@ -1698,9 +2132,20 @@ impl PrimaryOplogState { } } - async fn append(&mut self, entries: Vec) -> BTreeMap { + async fn append( + &mut self, + entries: Vec, + ) -> Result, OplogError> { record_oplog_call("append"); + // Already refused once: fail fast rather than re-asking the storage. Only entries buffered + // before the fence latched can reach here, and they go back where they were. + if let Some(fence) = self.fence.get() { + let fence = fence.clone(); + self.retain_refused(entries); + return Err(OplogError::Fenced(fence)); + } + // Commit barrier: every deferred external payload reserved during this session must be // durably written to blob storage before the entries (which may reference it) are persisted // to indexed storage. `append` flushes the whole buffer, so waiting on all outstanding @@ -1721,7 +2166,7 @@ impl PrimaryOplogState { } if entries.is_empty() { - return BTreeMap::new(); + return Ok(BTreeMap::new()); } let entry_count = entries.len() as u64; @@ -1742,7 +2187,7 @@ impl PrimaryOplogState { } let serialized_pairs: Arc<[(u64, Bytes)]> = serialized_pairs.into(); let namespace = self.namespace.clone(); - retry_oplog_append( + let appended = retry_oplog_append( &self.retry_config, self.indexed_storage.as_ref(), &namespace, @@ -1750,8 +2195,35 @@ impl PrimaryOplogState { "append", &self.key, SerializedOplogAppend::Batch(serialized_pairs), + self.shard_epoch, ) - .await; + .await + .map_err(|err| Self::as_oplog_error(&self.owned_agent_id, err)); + if self.shard_epoch.is_some() { + record_oplog_epoch_fence("append", matches!(appended, Err(OplogError::Fenced(_)))); + } + if let Err(error) = appended { + if let OplogError::Fenced(fence) = &error { + // The commit barrier above already awaited every payload the batch referenced, so + // those blobs are durable and stay behind with no stored entry pointing at them. + // The epoch the storage holds is reported, so a shard manager whose state lost + // history can mint above it. Warned only when it latches: each write after that + // fails fast on the latch without reaching the storage. + if self.fence.set(fence.clone()).is_ok() { + warn!( + agent_id = %self.owned_agent_id, + expected_epoch = fence.expected_epoch.0, + actual_epoch = ?fence.actual_epoch.map(|epoch| epoch.0), + "Oplog append fenced: the shard has a new owner, refusing further writes" + ); + } + if let Some(observer) = &self.fence_observer { + observer.fenced(fence); + } + self.retain_refused(pairs.into_iter().map(|(_, entry)| entry)); + } + return Err(error); + } record_storage_bytes_written( STORAGE_TYPE_OPLOG, @@ -1767,11 +2239,63 @@ impl PrimaryOplogState { ); self.last_committed_idx = last_idx; - BTreeMap::from_iter( + Ok(BTreeMap::from_iter( pairs .into_iter() .map(|(idx, entry)| (OplogIndex::from_u64(idx), entry)), - ) + )) + } + + /// Refuses a new write once the fence has latched, before anything is buffered or reserved. + /// + /// Without it an add below the commit threshold would only buffer, answer with an index, and + /// report a write that can never reach the storage. + fn refuse_if_fenced(&self) -> Result<(), OplogError> { + match self.fence.get() { + Some(fence) => Err(OplogError::Fenced(fence.clone())), + None => Ok(()), + } + } + + /// Puts entries a fenced append turned away back at the head of the buffer, where `commit` + /// drained them from. + /// + /// Every index this oplog has handed out stays readable from it: `last_oplog_idx` is not + /// rolled back, and the reader maps the buffer from `last_committed_idx`. A reader that took + /// `current_oplog_index` before the refusal and reads after it would otherwise find a gap and + /// fail-stop the executor. The entries are never sent again, because every later append fails + /// on the latch, and the buffer cannot grow, because every later add is refused. + fn retain_refused(&mut self, entries: impl IntoIterator) { + let mut restored: VecDeque = entries.into_iter().collect(); + restored.append(&mut self.buffer); + self.buffer = restored; + } + + /// Commits if the buffer is over the threshold. Separated out so the actor arms can fold a + /// threshold-commit failure into the job that triggered it. + async fn maybe_commit(&mut self) -> Result<(), OplogError> { + if self.over_commit_threshold() { + self.commit(CommitLevel::Always).await?; + } + Ok(()) + } + + /// Names the agent on a storage error, so the worker that hit it can be given up by id. + fn as_oplog_error(owned_agent_id: &OwnedAgentId, err: IndexedStorageError) -> OplogError { + match err { + IndexedStorageError::Fenced { + expected, + actual, + writer_conflict, + .. + } => OplogError::Fenced(OplogFence { + agent_id: owned_agent_id.agent_id(), + expected_epoch: expected, + actual_epoch: actual, + writer_conflict, + }), + other => OplogError::Storage(other.to_string()), + } } /// Pushes an entry into the in-memory buffer and advances the oplog index, @@ -1811,7 +2335,10 @@ impl PrimaryOplogState { self.buffer.len() > self.max_operations_before_commit as usize } - async fn commit(&mut self, _level: CommitLevel) -> BTreeMap { + async fn commit( + &mut self, + _level: CommitLevel, + ) -> Result, OplogError> { record_oplog_call("commit"); let entries = self.buffer.drain(..).collect::>(); @@ -1908,7 +2435,7 @@ impl Oplog for PrimaryOplog { async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { + ) -> Result, OplogError> { self.run_job(|done| OplogJob::AddDurableStreamBatch { make_batch, done }) .await } @@ -1917,7 +2444,7 @@ impl Oplog for PrimaryOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { self.run_job(|done| OplogJob::AddPair { start, make_second, @@ -1934,7 +2461,10 @@ impl Oplog for PrimaryOplog { .await } - async fn commit(&self, level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError> { self.run_job(|done| OplogJob::Commit { level, done }).await } @@ -1993,6 +2523,11 @@ impl Oplog for PrimaryOplog { record_oplog_call("wait_for_replicas"); self.run_job(|done| OplogJob::Flush { done }).await; + // A refused flush reached no replica. The storage would still answer with its replica + // count, and passing that on would tell the caller that entries it turned away are durable. + if self.fence.get().is_some() { + return false; + } let reader = self.run_job(|done| OplogJob::Reader { done }).await; let replicas = replicas.min(reader.replicas); match reader @@ -2086,7 +2621,7 @@ impl Oplog for PrimaryOplog { &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { // ORDERING (Start determinism): the job is enqueued synchronously here — there is no // `.await` between a subtask initiating its durable operation and this send — and the // actor assigns `Start` indices strictly in job order, so initiation order becomes @@ -2103,11 +2638,19 @@ impl Oplog for PrimaryOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { self.run_job(|done| OplogJob::AddIndexedStart { build_request, done, }) .await } + + fn shard_epoch(&self) -> Option { + self.shard_epoch + } + + fn fence(&self) -> Option { + self.fence.get().cloned() + } } diff --git a/golem-worker-executor/src/services/oplog/rate_limited.rs b/golem-worker-executor/src/services/oplog/rate_limited.rs index c7782365f2..3857d56b6f 100644 --- a/golem-worker-executor/src/services/oplog/rate_limited.rs +++ b/golem-worker-executor/src/services/oplog/rate_limited.rs @@ -16,12 +16,13 @@ use crate::metrics::oplog::record_oplog_rate_limited; use crate::model::ExecutionStatus; use crate::services::oplog::{ CommitLevel, DurableStreamBatchBuilder, IndexedReservedStartBuilder, Oplog, OplogAddReceipt, - OplogCloseCompletion, OplogLifecycleGuard, OplogService, OrderedOplogStart, + OplogCloseCompletion, OplogError, OplogLifecycleGuard, OplogService, OrderedOplogStart, ReservedRawStartBuilder, }; use crate::services::resource_limits::{AtomicResourceEntry, ResourceLimits}; use arc_swap::ArcSwap; use async_trait::async_trait; +use golem_common::model::ShardEpoch; use golem_common::model::account::AccountId; use golem_common::model::agent::AgentMode; use golem_common::model::component::ComponentId; @@ -200,26 +201,29 @@ impl Oplog for RateLimitedOplog { let account_id = self.account_id; let environment_id = self.environment_id; Box::pin(async move { - let idx = pending.await; + let idx = pending.await?; Self::apply_rate_limit_for(&resource_entry, &state, &account_id, &environment_id).await; - idx + Ok(idx) }) } async fn add_durable_stream_batch( &self, make_batch: DurableStreamBatchBuilder, - ) -> Result, String> { - let result = self.inner.add_durable_stream_batch(make_batch).await; + ) -> Result, OplogError> { + let result = self.inner.add_durable_stream_batch(make_batch).await?; self.apply_rate_limit().await; - result + Ok(result) } async fn drop_prefix(&self, last_dropped_id: OplogIndex) -> u64 { self.inner.drop_prefix(last_dropped_id).await } - async fn commit(&self, level: CommitLevel) -> BTreeMap { + async fn commit( + &self, + level: CommitLevel, + ) -> Result, OplogError> { self.inner.commit(level).await } @@ -284,18 +288,18 @@ impl Oplog for RateLimitedOplog { &self, start: OplogEntry, make_second: Box OplogEntry + Send>, - ) -> (OplogIndex, OplogIndex) { + ) -> Result<(OplogIndex, OplogIndex), OplogError> { // Assign the indices first, then throttle once for the pair: see `apply_rate_limit`. - let indices = self.inner.add_pair(start, make_second).await; + let indices = self.inner.add_pair(start, make_second).await?; self.apply_rate_limit().await; - indices + Ok(indices) } async fn add_start_with_reserved_raw_payload( &self, serialized_request: Vec, build_start: ReservedRawStartBuilder, - ) -> Result { + ) -> Result { // Order the `Start` first (the inner oplog assigns its index), then throttle. Applying // back-pressure before delegating would reorder concurrent calls' `Start` entries relative // to initiation order; see `apply_rate_limit`. @@ -310,7 +314,7 @@ impl Oplog for RateLimitedOplog { async fn add_start_with_indexed_reserved_raw_payload( &self, build_request: IndexedReservedStartBuilder, - ) -> Result { + ) -> Result { let ordered = self .inner .add_start_with_indexed_reserved_raw_payload(build_request) @@ -450,6 +454,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { let account_id = initial_worker_metadata.created_by; let environment_id = owned_agent_id.environment_id; @@ -464,6 +469,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ) .await; Arc::new(RateLimitedOplog::new( @@ -483,6 +489,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { let account_id = initial_worker_metadata.created_by; let environment_id = owned_agent_id.environment_id; @@ -497,6 +504,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ) .await; Arc::new(RateLimitedOplog::new( @@ -516,6 +524,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata: AgentMetadata, last_known_status: read_only_lock::arc_swap::ReadOnlyView, execution_status: read_only_lock::std::ReadOnlyLock, + shard_epoch: Option, ) -> Arc { let account_id = initial_worker_metadata.created_by; let environment_id = owned_agent_id.environment_id; @@ -530,6 +539,7 @@ impl OplogService for RateLimitedOplogService { initial_worker_metadata, last_known_status, execution_status, + shard_epoch, ) .await; Arc::new(RateLimitedOplog::new( @@ -553,9 +563,10 @@ impl OplogService for RateLimitedOplogService { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, - ) { + expected_epoch: Option, + ) -> Result<(), OplogError> { self.inner - .delete(lifecycle, owned_agent_id, agent_mode) + .delete(lifecycle, owned_agent_id, agent_mode, expected_epoch) .await } @@ -759,6 +770,7 @@ mod tests { make_agent_metadata(agent_id, account_id, env_id), last_known_status, execution_status, + None, ) .await } @@ -783,7 +795,7 @@ mod tests { let start = Instant::now(); for _ in 0..15 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let elapsed = start.elapsed(); @@ -804,7 +816,7 @@ mod tests { let start = Instant::now(); for _ in 0..100 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let elapsed = start.elapsed(); @@ -822,7 +834,7 @@ mod tests { let start = Instant::now(); for _ in 0..100 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let elapsed = start.elapsed(); @@ -842,7 +854,7 @@ mod tests { // Unlimited — should be fast. let start = Instant::now(); for _ in 0..20 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let fast_elapsed = start.elapsed(); assert!( @@ -856,7 +868,7 @@ mod tests { // 15 writes at 5/sec (burst=5) must take >= 1.5 s. let start = Instant::now(); for _ in 0..15 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let slow_elapsed = start.elapsed(); assert!( @@ -875,7 +887,7 @@ mod tests { // 15 writes at 5/sec — must be slow. let start = Instant::now(); for _ in 0..15 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let slow_elapsed = start.elapsed(); assert!( @@ -889,7 +901,7 @@ mod tests { // 100 writes at unlimited — should be fast. let start = Instant::now(); for _ in 0..100 { - oplog.add(dummy_entry()).await; + oplog.add(dummy_entry()).await.unwrap(); } let fast_elapsed = start.elapsed(); assert!( @@ -955,7 +967,7 @@ mod tests { // An inline payload is already durable. small_pending.wait().await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); // Read back from storage (no in-memory cache) and confirm the large request is external and // its deferred blob upload became durable via the commit barrier. diff --git a/golem-worker-executor/src/services/oplog/tests.rs b/golem-worker-executor/src/services/oplog/tests.rs index ec143ae4f5..831b1b384d 100644 --- a/golem-worker-executor/src/services/oplog/tests.rs +++ b/golem-worker-executor/src/services/oplog/tests.rs @@ -23,12 +23,14 @@ use crate::storage::indexed::redis::RedisIndexedStorage; use crate::storage::indexed::sqlite::SqliteIndexedStorage; use crate::storage::indexed::{ IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, + WriterId, }; use assert2::check; use bytes::Bytes; use futures::FutureExt; use futures::stream::BoxStream; use golem_common::config::RedisConfig; +use golem_common::model::ShardEpoch; use golem_common::model::account::{AccountEmail, AccountId}; use golem_common::model::agent::{AgentMode, OwnerKind, Principal}; use golem_common::model::card::{InvocationWalletPin, WalletVersionToken}; @@ -522,10 +524,18 @@ enum InjectedAppendFailure { CommitThenIndeterminate, CommitDifferentThenIndeterminate, CommitPrefixThenIndeterminate, + /// Refuses the write as a stale epoch would, naming the epoch one above whatever was + /// asserted. Simulates the storage fencing the reconciliation probe `retry_oplog_append` + /// repeats after a genuine indeterminate-write mismatch. + Fenced, } impl InjectedAppendFailure { - fn before_write_error(self) -> Option { + fn before_write_error( + self, + key: &str, + shard_epoch: Option, + ) -> Option { match self { Self::IndeterminateBeforeWrite => Some(IndexedStorageError::Indeterminate( "injected connection loss".to_string(), @@ -536,6 +546,12 @@ impl InjectedAppendFailure { Self::PermanentBeforeWrite => Some(IndexedStorageError::Other( "injected permanent failure".to_string(), )), + Self::Fenced => Some(IndexedStorageError::Fenced { + key: key.to_string(), + expected: shard_epoch.unwrap_or_default(), + actual: shard_epoch.map(|epoch| ShardEpoch(epoch.0 + 1)), + writer_conflict: false, + }), _ => None, } } @@ -550,7 +566,8 @@ impl InjectedAppendFailure { )), Self::IndeterminateBeforeWrite | Self::TransientBeforeWrite - | Self::PermanentBeforeWrite => unreachable!(), + | Self::PermanentBeforeWrite + | Self::Fenced => unreachable!(), } } } @@ -572,6 +589,8 @@ pub(crate) struct ReadCountingIndexedStorage { append_many_attempts: AtomicUsize, append_many_batch_ptr: AtomicUsize, append_many_batch_changed: AtomicBool, + drop_prefix_started: StdMutex>>, + release_drop_prefix: Option>, } impl ReadCountingIndexedStorage { @@ -579,6 +598,15 @@ impl ReadCountingIndexedStorage { Self::default() } + /// Every `drop_prefix` waits for `release`; the first one signals `started` when it arrives. + fn blocking_drop_prefix(started: oneshot::Sender<()>, release: Arc) -> Self { + Self { + drop_prefix_started: StdMutex::new(Some(started)), + release_drop_prefix: Some(release), + ..Self::default() + } + } + fn discarding_compressed_appends() -> Self { Self { discard_compressed_appends: true, @@ -643,6 +671,32 @@ impl ReadCountingIndexedStorage { #[async_trait] impl IndexedStorage for ReadCountingIndexedStorage { + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + shard_epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + self.inner + .set_key_epoch(svc_name, api_name, namespace, key, shard_epoch) + .await + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + self.inner + .delete_with_epoch(svc_name, api_name, namespace, key, expected_epoch) + .await + } + async fn number_of_replicas( &self, svc_name: &'static str, @@ -699,6 +753,7 @@ impl IndexedStorage for ReadCountingIndexedStorage { key: &str, id: u64, mut value: Vec, + shard_epoch: Option, ) -> Result<(), IndexedStorageError> { self.append_attempts.fetch_add(1, Ordering::Relaxed); if self.discard_compressed_appends @@ -712,7 +767,7 @@ impl IndexedStorage for ReadCountingIndexedStorage { .unwrap() .pop_front() .unwrap_or(InjectedAppendFailure::None); - if let Some(error) = failure.before_write_error() { + if let Some(error) = failure.before_write_error(key, shard_epoch) { return Err(error); } if matches!( @@ -722,7 +777,16 @@ impl IndexedStorage for ReadCountingIndexedStorage { value.push(0); } self.inner - .append(svc_name, api_name, entity_name, namespace, key, id, value) + .append( + svc_name, + api_name, + entity_name, + namespace, + key, + id, + value, + shard_epoch, + ) .await?; failure.after_write_result() } @@ -735,6 +799,7 @@ impl IndexedStorage for ReadCountingIndexedStorage { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + shard_epoch: Option, ) -> Result<(), IndexedStorageError> { self.append_many_attempts.fetch_add(1, Ordering::Relaxed); if self.discard_compressed_appends @@ -758,7 +823,7 @@ impl IndexedStorage for ReadCountingIndexedStorage { .unwrap() .pop_front() .unwrap_or(InjectedAppendFailure::None); - if let Some(error) = failure.before_write_error() { + if let Some(error) = failure.before_write_error(key, shard_epoch) { return Err(error); } let pairs = if matches!( @@ -780,7 +845,15 @@ impl IndexedStorage for ReadCountingIndexedStorage { pairs }; self.inner - .append_many(svc_name, api_name, entity_name, namespace, key, pairs) + .append_many( + svc_name, + api_name, + entity_name, + namespace, + key, + pairs, + shard_epoch, + ) .await?; failure.after_write_result() } @@ -908,6 +981,13 @@ impl IndexedStorage for ReadCountingIndexedStorage { key: &str, last_dropped_id: u64, ) -> Result<(), IndexedStorageError> { + if let Some(release) = &self.release_drop_prefix { + let started = self.drop_prefix_started.lock().unwrap().take(); + if let Some(started) = started { + let _ = started.send(()); + } + release.notified().await; + } self.inner .drop_prefix(svc_name, api_name, namespace, key, last_dropped_id) .await @@ -1217,6 +1297,7 @@ async fn ephemeral_create_baseline_uses_lower_storage_and_checked_reads_find_it( metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -1310,6 +1391,7 @@ async fn fresh_ephemeral_create_does_not_probe_lower_storage(_tracing: &Tracing) metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -1420,6 +1502,7 @@ async fn fresh_ephemeral_create_with_compressed_layers_does_not_read_storage(_tr metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -1480,6 +1563,7 @@ async fn primary_fresh_ephemeral_create_does_not_read_storage(_tracing: &Tracing metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -1566,9 +1650,9 @@ async fn staged_oplog_is_hidden_through_flush_and_published_without_cache_or_blo .create_staged(&owned, AgentMode::Durable, orphan_id, metadata.clone()) .await .unwrap(); - orphan.add(create.clone()).await; - orphan.add(OplogEntry::no_op(None).rounded()).await; - orphan.commit(CommitLevel::Always).await; + orphan.add(create.clone()).await.unwrap(); + orphan.add(OplogEntry::no_op(None).rounded()).await.unwrap(); + orphan.commit(CommitLevel::Always).await.unwrap(); drop(orphan); let entries = [ create.clone(), @@ -1576,8 +1660,8 @@ async fn staged_oplog_is_hidden_through_flush_and_published_without_cache_or_blo OplogEntry::no_op(None).rounded(), ]; for entry in &entries { - stage.add(entry.clone()).await; - stage.commit(CommitLevel::Always).await; + stage.add(entry.clone()).await.unwrap(); + stage.commit(CommitLevel::Always).await.unwrap(); assert!( service .staged_exists(&owned, AgentMode::Durable, stage_id) @@ -1659,6 +1743,7 @@ async fn staged_oplog_is_hidden_through_flush_and_published_without_cache_or_blo metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_eq!( @@ -1678,8 +1763,8 @@ async fn staged_oplog_is_hidden_through_flush_and_published_without_cache_or_blo .create_staged(&owned, AgentMode::Durable, losing_id, metadata) .await .unwrap(); - losing.add(create).await; - losing.commit(CommitLevel::Always).await; + losing.add(create).await.unwrap(); + losing.commit(CommitLevel::Always).await.unwrap(); assert_eq!(losing.current_oplog_index().await, OplogIndex::INITIAL); assert_eq!( reopened.current_oplog_index().await, @@ -1758,6 +1843,7 @@ async fn primary_uses_agent_mode_commit_threshold(_tracing: &Tracing) { metadata, default_last_known_status(), default_execution_status(agent_mode), + None, ) .await } @@ -1766,8 +1852,8 @@ async fn primary_uses_agent_mode_commit_threshold(_tracing: &Tracing) { let durable = open(AgentMode::Durable, "durable-threshold").await; let ephemeral = open(AgentMode::Ephemeral, "ephemeral-threshold").await; for oplog in [&durable, &ephemeral] { - oplog.add(OplogEntry::suspend().rounded()).await; - oplog.add(OplogEntry::exited().rounded()).await; + oplog.add(OplogEntry::suspend().rounded()).await.unwrap(); + oplog.add(OplogEntry::exited().rounded()).await.unwrap(); } assert_eq!(durable.length().await, 0); @@ -1827,6 +1913,7 @@ async fn fresh_ephemeral_create_with_blob_layers_does_not_read_storage(_tracing: metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -1876,6 +1963,14 @@ async fn append_reconciliation_service( async fn create_append_reconciliation_oplog( service: &PrimaryOplogService, name: &str, +) -> Arc { + create_append_reconciliation_oplog_with_epoch(service, name, None).await +} + +async fn create_append_reconciliation_oplog_with_epoch( + service: &PrimaryOplogService, + name: &str, + shard_epoch: Option, ) -> Arc { let account_id = AccountId::new(); let environment_id = EnvironmentId::new(); @@ -1899,6 +1994,7 @@ async fn create_append_reconciliation_oplog( make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + shard_epoch, ) .await } @@ -1937,6 +2033,7 @@ async fn lifecycle_reader_blocks_delete_and_late_drop_cannot_remove_replacement( metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let stale = old.clone(); @@ -1951,7 +2048,10 @@ async fn lifecycle_reader_blocks_delete_and_late_drop_cannot_remove_replacement( entered_tx.send(()).unwrap(); let mut guard = lock.await; old.stop_and_wait().await.unwrap(); - service.delete(&mut guard, &id, AgentMode::Durable).await; + service + .delete(&mut guard, &id, AgentMode::Durable, None) + .await + .unwrap(); let next_lifecycle = service.lock_lifecycle(&id.agent_id); tokio::pin!(next_lifecycle); assert!(futures::poll!(&mut next_lifecycle).is_pending()); @@ -1981,6 +2081,7 @@ async fn lifecycle_reader_blocks_delete_and_late_drop_cannot_remove_replacement( metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; drop(stale); @@ -1993,15 +2094,16 @@ async fn lifecycle_reader_blocks_delete_and_late_drop_cannot_remove_replacement( metadata, default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert!(Arc::ptr_eq(&replacement, &reopened)); assert_eq!(reopened.read(OplogIndex::INITIAL).await, replacement_entry); assert_eq!( - reopened.add(OplogEntry::no_op(None)).await, + reopened.add(OplogEntry::no_op(None)).await.unwrap(), OplogIndex::from_u64(2) ); - reopened.commit(CommitLevel::Always).await; + reopened.commit(CommitLevel::Always).await.unwrap(); reopened.stop_and_wait().await.unwrap(); } @@ -2034,6 +2136,7 @@ async fn stopped_actor_failure_does_not_poison_reopen(_tracing: &Tracing) { metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_panics(old.add_pair( @@ -2051,15 +2154,16 @@ async fn stopped_actor_failure_does_not_poison_reopen(_tracing: &Tracing) { metadata, default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert!(!Arc::ptr_eq(&old, &reopened)); assert_eq!(reopened.read(OplogIndex::INITIAL).await, initial); assert_eq!( - reopened.add(OplogEntry::no_op(None)).await, + reopened.add(OplogEntry::no_op(None)).await.unwrap(), OplogIndex::from_u64(2) ); - reopened.commit(CommitLevel::Always).await; + reopened.commit(CommitLevel::Always).await.unwrap(); drop(old); reopened.stop_and_wait().await.unwrap(); } @@ -2106,6 +2210,7 @@ async fn reopened_oplog_joins_old_root_children_before_stopping_its_writer(_trac metadata.clone(), default_last_known_status(), default_execution_status(mode), + None, ) .await; let reopened = service @@ -2117,6 +2222,7 @@ async fn reopened_oplog_joins_old_root_children_before_stopping_its_writer(_trac metadata.clone(), default_last_known_status(), default_execution_status(mode), + None, ) .await; let (started, started_rx) = oneshot::channel(); @@ -2134,10 +2240,10 @@ async fn reopened_oplog_joins_old_root_children_before_stopping_its_writer(_trac started.send(()).unwrap(); release_rx.await.unwrap(); assert_eq!( - writer.add(OplogEntry::no_op(None)).await, + writer.add(OplogEntry::no_op(None)).await.unwrap(), OplogIndex::from_u64(2) ); - writer.commit(CommitLevel::Always).await; + writer.commit(CommitLevel::Always).await.unwrap(); }); owner.finish_on_drop(job).await.unwrap(); }) @@ -2170,6 +2276,7 @@ async fn reopened_oplog_joins_old_root_children_before_stopping_its_writer(_trac metadata, default_last_known_status(), default_execution_status(mode), + None, ) .await; assert!( @@ -2219,6 +2326,7 @@ async fn explicit_commit_reports_threshold_commits_once_and_preserves_add_receip make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -2234,11 +2342,11 @@ async fn explicit_commit_reports_threshold_commits_once_and_preserves_add_receip .collect::>(); let mut expected = BTreeMap::new(); for (receipt, entry) in receipts.into_iter().zip(entries) { - expected.insert(receipt.await, entry); + expected.insert(receipt.await.unwrap(), entry); } - assert_eq!(oplog.commit(CommitLevel::Always).await, expected); - assert!(oplog.commit(CommitLevel::Always).await.is_empty()); + assert_eq!(oplog.commit(CommitLevel::Always).await.unwrap(), expected); + assert!(oplog.commit(CommitLevel::Always).await.unwrap().is_empty()); } #[test] @@ -2285,6 +2393,7 @@ async fn archiving_auto_committed_entries_does_not_consume_explicit_commit_repor make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -2295,21 +2404,21 @@ async fn archiving_auto_committed_entries_does_not_consume_explicit_commit_repor ]; let mut expected = BTreeMap::new(); for entry in entries { - let index = oplog.add(entry.clone()).await; + let index = oplog.add(entry.clone()).await.unwrap(); expected.insert(index, entry); } MultiLayerOplog::try_archive_blocking(&oplog).await; - assert_eq!(oplog.commit(CommitLevel::Always).await, expected); - assert!(oplog.commit(CommitLevel::Always).await.is_empty()); + assert_eq!(oplog.commit(CommitLevel::Always).await.unwrap(), expected); + assert!(oplog.commit(CommitLevel::Always).await.unwrap().is_empty()); let mut before_commit = BTreeMap::new(); for entry in [ OplogEntry::interrupted().rounded(), OplogEntry::resumed().rounded(), ] { - before_commit.insert(oplog.add(entry.clone()).await, entry); + before_commit.insert(oplog.add(entry.clone()).await.unwrap(), entry); } let mut commit = std::pin::pin!(oplog.commit(CommitLevel::Always)); assert!(futures::poll!(commit.as_mut()).is_pending()); @@ -2321,12 +2430,15 @@ async fn archiving_auto_committed_entries_does_not_consume_explicit_commit_repor OplogEntry::suspend().rounded(), OplogEntry::restart().rounded(), ] { - after_commit.insert(oplog.add(entry.clone()).await, entry); + after_commit.insert(oplog.add(entry.clone()).await.unwrap(), entry); } - assert_eq!(commit.await, before_commit); + assert_eq!(commit.await.unwrap(), before_commit); MultiLayerOplog::try_archive_blocking(&oplog).await; - assert_eq!(oplog.commit(CommitLevel::Always).await, after_commit); - assert!(oplog.commit(CommitLevel::Always).await.is_empty()); + assert_eq!( + oplog.commit(CommitLevel::Always).await.unwrap(), + after_commit + ); + assert!(oplog.commit(CommitLevel::Always).await.unwrap().is_empty()); } #[test] @@ -2349,14 +2461,14 @@ async fn wait_for_replicas_does_not_consume_explicit_commit_report(_tracing: &Tr ]; let mut expected = BTreeMap::new(); for entry in entries { - let index = oplog.add(entry.clone()).await; + let index = oplog.add(entry.clone()).await.unwrap(); expected.insert(index, entry); } assert!(oplog.wait_for_replicas(1, Duration::from_secs(1)).await); - assert_eq!(oplog.commit(CommitLevel::Always).await, expected); - assert!(oplog.commit(CommitLevel::Always).await.is_empty()); + assert_eq!(oplog.commit(CommitLevel::Always).await.unwrap(), expected); + assert!(oplog.commit(CommitLevel::Always).await.unwrap().is_empty()); } #[test] @@ -2381,9 +2493,12 @@ async fn retried_append_many_accepts_only_the_same_serialized_batch(_tracing: &T indexed_storage.reset_append_observations(); indexed_storage.inject_append_many_failure(InjectedAppendFailure::CommitThenIndeterminate); - oplog.add(OplogEntry::suspend()).await; - oplog.add(OplogEntry::exited()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog.add(OplogEntry::exited()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(oplog.current_oplog_index().await, OplogIndex::from_u64(3)); assert_eq!(indexed_storage.append_many_attempts(), 1); @@ -2401,8 +2516,11 @@ async fn indeterminate_append_before_write_retries_after_empty_read_back(_tracin indexed_storage.inject_append_many_failure(InjectedAppendFailure::IndeterminateBeforeWrite); let entry = OplogEntry::suspend().rounded(); - oplog.add(entry.clone()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(entry.clone()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(indexed_storage.append_many_attempts(), 2); assert_eq!(indexed_storage.reads(), 1); @@ -2424,8 +2542,11 @@ async fn exhausted_retries_after_committed_indeterminate_append_reconcile(_traci InjectedAppendFailure::TransientBeforeWrite, ]); - oplog.add(OplogEntry::suspend()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(indexed_storage.append_many_attempts(), 3); assert_eq!(indexed_storage.reads(), 3); @@ -2446,8 +2567,11 @@ async fn permanent_retry_failure_after_committed_indeterminate_append_reconciles InjectedAppendFailure::PermanentBeforeWrite, ]); - oplog.add(OplogEntry::suspend()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(indexed_storage.append_many_attempts(), 2); assert_eq!(indexed_storage.reads(), 2); @@ -2470,8 +2594,11 @@ async fn reconciliation_retries_read_failures_without_resubmitting_append(_traci "connection lost during read-back".to_string(), )); - oplog.add(OplogEntry::suspend()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(indexed_storage.append_many_attempts(), 1); assert_eq!(indexed_storage.reads(), 2); @@ -2487,8 +2614,11 @@ async fn conflict_after_initially_empty_reconciliation_accepts_exact_batch(_trac indexed_storage.inject_append_many_failure(InjectedAppendFailure::CommitThenIndeterminate); indexed_storage.hidden_reads.store(1, Ordering::Relaxed); - oplog.add(OplogEntry::suspend()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!(indexed_storage.append_many_attempts(), 2); assert_eq!(indexed_storage.reads(), 2); @@ -2522,6 +2652,7 @@ async fn direct_identical_append_conflict_from_second_writer_remains_fatal(_trac make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let second_oplog = second_service @@ -2535,12 +2666,16 @@ async fn direct_identical_append_conflict_from_second_writer_remains_fatal(_trac make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let entry = OplogEntry::suspend(); - first_oplog.add(entry.clone()).await; - second_oplog.add(entry).await; - first_oplog.commit(CommitLevel::Always).await; + first_oplog.add(entry.clone()).await.expect("oplog write"); + second_oplog.add(entry).await.expect("oplog write"); + first_oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); indexed_storage.reset(); indexed_storage.reset_append_observations(); @@ -2560,8 +2695,8 @@ async fn incomplete_read_back_after_indeterminate_append_remains_fatal(_tracing: indexed_storage .inject_append_many_failure(InjectedAppendFailure::CommitPrefixThenIndeterminate); - oplog.add(OplogEntry::suspend()).await; - oplog.add(OplogEntry::exited()).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + oplog.add(OplogEntry::exited()).await.expect("oplog write"); assert_panics(oplog.commit(CommitLevel::Always)).await; assert_eq!(indexed_storage.append_many_attempts(), 1); @@ -2578,13 +2713,135 @@ async fn differing_read_back_after_indeterminate_append_remains_fatal(_tracing: indexed_storage .inject_append_many_failure(InjectedAppendFailure::CommitDifferentThenIndeterminate); - oplog.add(OplogEntry::suspend()).await; + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); assert_panics(oplog.commit(CommitLevel::Always)).await; assert_eq!(indexed_storage.append_many_attempts(), 1); assert_eq!(indexed_storage.reads(), 1); } +/// Same mismatch as `differing_read_back_after_indeterminate_append_remains_fatal`, except this +/// writer asserts a shard epoch and the mismatch is explained: a new owner already wrote those +/// indices. The reconciliation probe this fences must return `Fenced` and let the caller +/// give up the agent, rather than panicking and aborting the whole - otherwise still live - +/// executor process (`panic = "abort"`). +#[test] +async fn differing_read_back_on_a_moved_shard_is_fenced_instead_of_panicking(_tracing: &Tracing) { + let indexed_storage = Arc::new(ReadCountingIndexedStorage::new()); + let service = append_reconciliation_service(indexed_storage.clone()).await; + let oplog = create_append_reconciliation_oplog_with_epoch( + &service, + "different-append-read-back-fenced", + Some(ShardEpoch(5)), + ) + .await; + indexed_storage.reset(); + indexed_storage.reset_append_observations(); + indexed_storage.inject_append_many_failures([ + InjectedAppendFailure::CommitDifferentThenIndeterminate, + InjectedAppendFailure::Fenced, + ]); + + oplog.add(OplogEntry::suspend()).await.expect("oplog write"); + let result = oplog.commit(CommitLevel::Always).await; + + assert!( + matches!(result, Err(OplogError::Fenced(_))), + "expected the reconciliation probe to surface a fence instead of panicking, got {result:?}" + ); + // The original attempt, then the reconciliation probe once the read-back mismatched. + assert_eq!(indexed_storage.append_many_attempts(), 2); +} + +#[test] +async fn a_delete_that_outlived_the_shard_leaves_the_oplog_to_its_new_owner(_tracing: &Tracing) { + let indexed_storage = Arc::new(InMemoryIndexedStorage::new()); + let blob_storage = Arc::new(InMemoryBlobStorage::new()); + let primary = Arc::new( + PrimaryOplogService::new( + indexed_storage.clone(), + blob_storage, + 1, + 1, + 100, + RetryConfig::default(), + ) + .await, + ); + let archive = Arc::new(CompressedOplogArchiveService::new( + indexed_storage.clone(), + 1, + RetryConfig::default(), + )); + let service = MultiLayerOplogService::new(primary.clone(), nev![archive], 100, 1); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "delete-after-the-shard-moved".to_string(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let oplog = service + .create( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + OplogEntry::no_op(None), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + oplog.add_and_commit(OplogEntry::no_op(None)).await.unwrap(); + let last = oplog.current_oplog_index().await; + drop(oplog); + + // The shard moves on and another executor takes the oplog over at the next epoch, while this + // one is still working through a deletion it accepted at epoch 5. + indexed_storage + .for_writer(WriterId(Uuid::new_v4())) + .set_key_epoch( + "oplog", + "set_key_epoch", + IndexedStorageNamespace::OpLog { + agent_id: agent_id.clone(), + agent_mode: AgentMode::Durable, + }, + &agent_id.to_redis_key(), + ShardEpoch(6), + ) + .await + .unwrap(); + + let result = service + .delete( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + Some(ShardEpoch(5)), + ) + .await; + + match result { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.expected_epoch, ShardEpoch(5)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(6))); + } + other => panic!("expected the delete to be fenced, got {other:?}"), + } + assert!( + service.exists(&owned_agent_id, AgentMode::Durable).await, + "a refused delete removes nothing" + ); + assert_eq!( + primary + .get_last_index(&owned_agent_id, AgentMode::Durable) + .await, + last + ); +} + #[test] async fn open_add_and_read_back(_tracing: &Tracing) { let indexed_storage = Arc::new(InMemoryIndexedStorage::new()); @@ -2614,6 +2871,7 @@ async fn open_add_and_read_back(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -2629,10 +2887,10 @@ async fn open_add_and_read_back(_tracing: &Tracing) { let entry3 = OplogEntry::exited().rounded(); let last_oplog_idx = oplog.current_oplog_index().await; - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; - oplog.add(entry3.clone()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); + oplog.add(entry3.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let r1 = oplog.read(last_oplog_idx.next()).await; let r2 = oplog.read(last_oplog_idx.next().next()).await; @@ -2689,6 +2947,7 @@ async fn primary_read_range_overflow_panics_without_storage_io(_tracing: &Tracin make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_panics(oplog.read_exact(start, 2)).await; @@ -2734,6 +2993,7 @@ async fn primary_storage_read_failures_panic_from_all_read_paths(_tracing: &Trac make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_panics(oplog.read_exact(OplogIndex::INITIAL, 1)).await; @@ -2748,6 +3008,7 @@ async fn primary_storage_read_failures_panic_from_all_read_paths(_tracing: &Trac make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_panics(oplog.read(OplogIndex::INITIAL)).await; @@ -2830,6 +3091,7 @@ async fn durable_stream_batch_uses_payload_threshold_for_each_record(_tracing: & make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let producer_fingerprint = AgentFingerprint(Uuid::new_v4()); @@ -2916,7 +3178,7 @@ async fn durable_stream_batch_uses_payload_threshold_for_each_record(_tracing: & })) .await .unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!(added.len(), 5); for (_, entry) in added { @@ -3021,6 +3283,7 @@ async fn ephemeral_durable_stream_batch_keeps_terminals_inline_atomically(_traci metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; let session_key = StreamInvocationId { @@ -3063,7 +3326,7 @@ async fn ephemeral_durable_stream_batch_keeps_terminals_inline_atomically(_traci let resident = oplog.read_exact(added[0].0, 2).await; assert_eq!(resident, added.iter().cloned().collect()); // Threshold handoff is asynchronous; protocol publication uses an explicit barrier. - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let persisted = service .read_exact( &owned_agent_id, @@ -3150,6 +3413,7 @@ async fn blocked_durable_stream_batch_prepares_before_atomic_commit_and_append(_ make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let generated = Arc::new(AtomicUsize::new(0)); @@ -3221,8 +3485,8 @@ async fn blocked_durable_stream_batch_prepares_before_atomic_commit_and_append(_ release_put.send(()).unwrap(); let added = batch.await.unwrap().unwrap(); - competing_commit.await; - let competing_index = competing_append.await; + competing_commit.await.unwrap(); + let competing_index = competing_append.await.unwrap(); assert_eq!(generated.load(Ordering::SeqCst), 3); assert_eq!(added.len(), 3); @@ -3321,6 +3585,7 @@ async fn durable_stream_producer_recovers_from_sqlite_storage_restart(_tracing: make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let producer = DurableStreamStore::load( @@ -3375,6 +3640,7 @@ async fn durable_stream_producer_recovers_from_sqlite_storage_restart(_tracing: make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let restarted = DurableStreamStore::load( @@ -3458,6 +3724,7 @@ async fn open_add_and_read_back_many(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -3474,12 +3741,12 @@ async fn open_add_and_read_back_many(_tracing: &Tracing) { let entry4 = OplogEntry::interrupted().rounded(); let entry5 = OplogEntry::no_op(None).rounded(); - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; - oplog.add(entry3.clone()).await; - oplog.commit(CommitLevel::Always).await; - oplog.add(entry4.clone()).await; - oplog.add(entry5.clone()).await; // uncommitted entries + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); + oplog.add(entry3.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); + oplog.add(entry4.clone()).await.unwrap(); + oplog.add(entry5.clone()).await.unwrap(); // uncommitted entries let read_count = indexed_storage.read_count(); let buffered_entries = oplog @@ -3569,6 +3836,7 @@ async fn open_add_and_read_back_ephemeral(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -3584,10 +3852,10 @@ async fn open_add_and_read_back_ephemeral(_tracing: &Tracing) { let entry3 = OplogEntry::exited().rounded(); let last_oplog_idx = oplog.current_oplog_index().await; - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; - oplog.add(entry3.clone()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); + oplog.add(entry3.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let r1 = oplog.read(last_oplog_idx.next()).await; let r2 = oplog.read(last_oplog_idx.next().next()).await; @@ -3661,6 +3929,7 @@ async fn open_add_and_read_back_many_ephemeral(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -3676,11 +3945,11 @@ async fn open_add_and_read_back_many_ephemeral(_tracing: &Tracing) { let entry3 = OplogEntry::exited().rounded(); let entry4 = OplogEntry::interrupted().rounded(); - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; - oplog.add(entry3.clone()).await; - oplog.commit(CommitLevel::Always).await; - oplog.add(entry4.clone()).await; // uncommitted + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); + oplog.add(entry3.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); + oplog.add(entry4.clone()).await.unwrap(); // uncommitted let entries = oplog .read_exact(OplogIndex::INITIAL, 4) @@ -3734,6 +4003,7 @@ async fn ephemeral_read_exact_committed_only(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -3741,10 +4011,10 @@ async fn ephemeral_read_exact_committed_only(_tracing: &Tracing) { let entry2 = OplogEntry::exited().rounded(); let entry3 = OplogEntry::interrupted().rounded(); - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; - oplog.add(entry3.clone()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); + oplog.add(entry3.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); // All committed, no buffer entries let entries = oplog @@ -3799,14 +4069,15 @@ async fn ephemeral_read_exact_uncommitted_only(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; let entry1 = OplogEntry::suspend().rounded(); let entry2 = OplogEntry::exited().rounded(); - oplog.add(entry1.clone()).await; - oplog.add(entry2.clone()).await; + oplog.add(entry1.clone()).await.unwrap(); + oplog.add(entry2.clone()).await.unwrap(); // No commit — entries only in the buffer let entries = oplog @@ -3861,6 +4132,7 @@ async fn ephemeral_read_exact_partial_range(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -3877,16 +4149,16 @@ async fn ephemeral_read_exact_partial_range(_tracing: &Tracing) { retry_policy_state: None, } .rounded(); - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); entries.push(entry); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); // Add 2 more uncommitted let uncommitted1 = OplogEntry::interrupted().rounded(); let uncommitted2 = OplogEntry::suspend().rounded(); - oplog.add(uncommitted1.clone()).await; - oplog.add(uncommitted2.clone()).await; + oplog.add(uncommitted1.clone()).await.unwrap(); + oplog.add(uncommitted2.clone()).await.unwrap(); entries.push(uncommitted1); entries.push(uncommitted2); @@ -3966,6 +4238,7 @@ async fn ephemeral_read_exact_across_archive_layers(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -3988,15 +4261,15 @@ async fn ephemeral_read_exact_across_archive_layers(_tracing: &Tracing) { let initial_oplog_idx = oplog.current_oplog_index().await; for entry in &entries { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); // Add 2 uncommitted entries let uncommitted1 = OplogEntry::interrupted().rounded(); let uncommitted2 = OplogEntry::suspend().rounded(); - oplog.add(uncommitted1.clone()).await; - oplog.add(uncommitted2.clone()).await; + oplog.add(uncommitted1.clone()).await.unwrap(); + oplog.add(uncommitted2.clone()).await.unwrap(); entries.push(uncommitted1); entries.push(uncommitted2); @@ -4092,10 +4365,11 @@ async fn ephemeral_read_exact_zero_returns_empty(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; - oplog.add(OplogEntry::suspend().rounded()).await; + oplog.add(OplogEntry::suspend().rounded()).await.unwrap(); let entries = oplog.read_exact(OplogIndex::INITIAL, 0).await; assert!(entries.is_empty()); @@ -4131,6 +4405,7 @@ async fn entries_with_small_payload(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -4192,9 +4467,9 @@ async fn entries_with_small_payload(_tracing: &Tracing) { description: desc.clone(), } .rounded(); - oplog.add(entry4.clone()).await; + oplog.add(entry4.clone()).await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let r_start = oplog.read(last_oplog_idx.next()).await.rounded(); let r_end = oplog.read(last_oplog_idx.next().next()).await.rounded(); @@ -4335,6 +4610,7 @@ async fn completed_host_call_response_upload_failure_writes_no_start(_tracing: & make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let before = oplog.current_oplog_index().await; @@ -4415,6 +4691,7 @@ async fn owned_invocation_payload_upload_failure_writes_no_entry(_tracing: &Trac make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let before = oplog.current_oplog_index().await; @@ -4469,6 +4746,7 @@ async fn entries_with_large_payload(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -4541,9 +4819,9 @@ async fn entries_with_large_payload(_tracing: &Tracing) { description: desc.clone(), } .rounded(); - oplog.add(entry4.clone()).await; + oplog.add(entry4.clone()).await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let r_start = oplog.read(last_oplog_idx.next()).await.rounded(); let r_end = oplog.read(last_oplog_idx.next().next()).await.rounded(); @@ -4756,6 +5034,7 @@ async fn multilayer_transfers_entries_after_limit_reached( make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let mut entries = Vec::new(); @@ -4777,8 +5056,8 @@ async fn multilayer_transfers_entries_after_limit_reached( durable_function_type: DurableFunctionType::ReadLocal, } .rounded(); - oplog.add(entry.clone()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(entry.clone()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); entries.push(entry); } @@ -4795,6 +5074,7 @@ async fn multilayer_transfers_entries_after_limit_reached( make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -4829,6 +5109,7 @@ async fn multilayer_transfers_entries_after_limit_reached( make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -4924,6 +5205,7 @@ async fn read_from_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -4946,13 +5228,13 @@ async fn read_from_archive_impl(use_blob: bool) { let initial_oplog_idx = oplog.current_oplog_index().await; for entry in &entries { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let uncommitted1 = OplogEntry::interrupted().rounded(); let uncommitted2 = OplogEntry::suspend().rounded(); - oplog.add(uncommitted1.clone()).await; - oplog.add(uncommitted2.clone()).await; + oplog.add(uncommitted1.clone()).await.unwrap(); + oplog.add(uncommitted2.clone()).await.unwrap(); entries.push(uncommitted1); entries.push(uncommitted2); @@ -4970,6 +5252,7 @@ async fn read_from_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -5121,6 +5404,7 @@ async fn read_initial_from_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -5267,9 +5551,10 @@ async fn ephemeral_read_initial_from_archive_impl(use_blob: bool) { }, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let read_before_archive = oplog_service .read_exact( @@ -5578,9 +5863,10 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; - oplog.add_and_commit(OplogEntry::no_op(None)).await; + oplog.add_and_commit(OplogEntry::no_op(None)).await.unwrap(); let current = oplog.current_oplog_index().await; service @@ -5588,8 +5874,10 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, &owned_agent_id, AgentMode::Durable, + None, ) - .await; + .await + .unwrap(); assert_eq!(oplog.current_oplog_index().await, current); assert!(!service.exists(&owned_agent_id, AgentMode::Durable).await); @@ -5623,6 +5911,7 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert!(!Arc::ptr_eq(&oplog, &replacement)); @@ -5641,6 +5930,7 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; drop(oplog); @@ -5653,6 +5943,7 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci metadata.clone(), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert!(Arc::ptr_eq(&replacement, &reopened)); @@ -5665,6 +5956,7 @@ async fn open_multilayer_oplog_retains_stale_index_after_service_deletion(_traci metadata, default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert!(Arc::ptr_eq(&replacement_primary, &reopened_primary)); @@ -5744,11 +6036,12 @@ async fn deleting_worker_fences_in_flight_archive_transfers_impl(agent_mode: Age }, default_last_known_status(), default_execution_status(agent_mode), + None, ) .await; - oplog.add(OplogEntry::no_op(None)).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::no_op(None)).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); if agent_mode == AgentMode::Ephemeral { EphemeralOplog::try_archive(&oplog) .await @@ -5764,8 +6057,10 @@ async fn deleting_worker_fences_in_flight_archive_transfers_impl(agent_mode: Age &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, &owned_agent_id, agent_mode, + None, ) - .await; + .await + .unwrap(); let append_completed = append_finished.notified(); release_append.notify_one(); @@ -5845,6 +6140,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; info!("FIRST OPEN DONE"); @@ -5868,9 +6164,9 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { let initial_oplog_idx = oplog.current_oplog_index().await; for entry in &entries { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); tokio::time::sleep(Duration::from_secs(2)).await; let primary_length = primary_oplog_service @@ -5884,6 +6180,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -5915,6 +6212,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await } else if reopen == Reopen::Full { @@ -5945,6 +6243,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await } else { @@ -5967,12 +6266,12 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { .collect(); for (n, entry) in entries.iter().enumerate() { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); if n % 100 == 0 { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); } } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); tokio::time::sleep(Duration::from_secs(2)).await; let primary_length = primary_oplog_service @@ -5986,6 +6285,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -6017,6 +6317,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await } else if reopen == Reopen::Full { @@ -6047,6 +6348,7 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await } else { @@ -6066,8 +6368,9 @@ async fn write_after_archive_impl(use_blob: bool, reopen: Reopen) { } .rounded(), ) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); drop(oplog); let entry1 = oplog_service @@ -6222,6 +6525,7 @@ async fn empty_layer_gets_deleted_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -6247,9 +6551,9 @@ async fn empty_layer_gets_deleted_impl(use_blob: bool) { .collect(); for entry in &entries { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); tokio::time::sleep(Duration::from_millis(100)).await; } @@ -6276,6 +6580,7 @@ async fn empty_layer_gets_deleted_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -6388,12 +6693,13 @@ async fn scheduled_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; for entry in &entries { - oplog.add(entry.clone()).await; + oplog.add(entry.clone()).await.unwrap(); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let result = MultiLayerOplog::try_archive(&oplog).await; drop(oplog); @@ -6417,6 +6723,7 @@ async fn scheduled_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -6458,6 +6765,7 @@ async fn scheduled_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let result = MultiLayerOplog::try_archive(&oplog).await; @@ -6478,6 +6786,7 @@ async fn scheduled_archive_impl(use_blob: bool) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await .length() @@ -6568,6 +6877,7 @@ async fn multilayer_scan_for_component(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -6588,7 +6898,8 @@ async fn multilayer_scan_for_component(_tracing: &Tracing) { "test".to_string(), "test".to_string(), )) - .await; + .await + .unwrap(); } } 2 => { @@ -6607,7 +6918,8 @@ async fn multilayer_scan_for_component(_tracing: &Tracing) { "test".to_string(), "test".to_string(), )) - .await; + .await + .unwrap(); } debug!("[{r:?}] => archiving {agent_id} to tertiary layer"); @@ -6622,7 +6934,8 @@ async fn multilayer_scan_for_component(_tracing: &Tracing) { "test".to_string(), "test".to_string(), )) - .await; + .await + .unwrap(); } } _ => unreachable!(), @@ -6720,9 +7033,10 @@ async fn multilayer_scan_for_component_ephemeral(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(mode), + None, ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); owned_agent_id }; @@ -6831,9 +7145,10 @@ async fn concurrent_get_or_open_does_not_cause_unique_key_violation(_tracing: &T make_agent_metadata(worker_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; - initial_oplog.commit(CommitLevel::Always).await; + initial_oplog.commit(CommitLevel::Always).await.unwrap(); drop(initial_oplog); // Wait for the weak reference to become invalid so the cache entry is evicted @@ -6871,6 +7186,7 @@ async fn concurrent_get_or_open_does_not_cause_unique_key_violation(_tracing: &T make_agent_metadata(worker_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -6878,10 +7194,10 @@ async fn concurrent_get_or_open_does_not_cause_unique_key_violation(_tracing: &T // different oplog instances (due to the get_or_open race), they'll // have independent last_committed_idx and produce duplicate ids, // causing a unique key violation on commit. - oplog.add(OplogEntry::suspend()).await; - // Use fallible_add pattern: commit can panic on unique key violation; + oplog.add(OplogEntry::suspend()).await.unwrap(); + // `add` is fallible now: commit can panic on unique key violation; // we use the Oplog trait method directly and let it propagate. - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); tokio::task::yield_now().await; } @@ -7201,6 +7517,7 @@ async fn durable_and_ephemeral_oplogs_are_isolated_for_same_agent_id(_tracing: & make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; let ephemeral_oplog = oplog_service @@ -7212,10 +7529,11 @@ async fn durable_and_ephemeral_oplogs_are_isolated_for_same_agent_id(_tracing: & make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; - durable_oplog.commit(CommitLevel::Always).await; - ephemeral_oplog.commit(CommitLevel::Always).await; + durable_oplog.commit(CommitLevel::Always).await.unwrap(); + ephemeral_oplog.commit(CommitLevel::Always).await.unwrap(); // Both namespaces report the oplog exists, independently. assert!( @@ -7257,8 +7575,10 @@ async fn durable_and_ephemeral_oplogs_are_isolated_for_same_agent_id(_tracing: & &mut oplog_service.lock_lifecycle(&owned_agent_id.agent_id).await, &owned_agent_id, AgentMode::Durable, + None, ) - .await; + .await + .unwrap(); assert!( !oplog_service .exists(&owned_agent_id, AgentMode::Durable) @@ -7308,9 +7628,10 @@ async fn make_workers( make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(mode), + None, ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); out.push(owned_agent_id); } out @@ -7565,6 +7886,7 @@ async fn owned_payload_upload_preserves_allocation_at_inline_threshold_and_round make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -7609,6 +7931,7 @@ async fn owned_payload_upload_preserves_allocation_at_inline_threshold_and_round make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; assert_eq!( @@ -7663,6 +7986,7 @@ async fn owned_snapshot_payloads_persist_and_replay_across_inline_threshold(_tra make_agent_metadata(agent_id, account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -7712,14 +8036,16 @@ async fn owned_snapshot_payloads_persist_and_replay_across_inline_threshold(_tra timestamp: Timestamp::now_utc(), description: inline_description, }) - .await; + .await + .unwrap(); let external_index = oplog .add(OplogEntry::PendingUpdate { timestamp: Timestamp::now_utc(), description: external_description, }) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let persisted = oplog_service .read_exact(&owned_agent_id, AgentMode::Durable, inline_index, 2) @@ -7796,6 +8122,7 @@ async fn reserved_large_request_is_durable_via_commit_barrier(_tracing: &Tracing make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -7820,7 +8147,7 @@ async fn reserved_large_request_is_durable_via_commit_barrier(_tracing: &Tracing .unwrap(); assert_eq!(start_idx, last_oplog_idx.next()); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); // Read back from the service (storage), so the payload reference carries no in-memory cache and // the download must hit blob storage. @@ -7885,6 +8212,7 @@ async fn reserved_small_request_stays_inline(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -7907,7 +8235,7 @@ async fn reserved_small_request_stays_inline(_tracing: &Tracing) { // Inline payloads are already durable: waiting is a no-op. pending.wait().await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let entries = oplog_service .read_exact(&owned_agent_id, AgentMode::Durable, start_idx, 1) @@ -8010,6 +8338,7 @@ async fn multilayer_reserved_start_delegates_to_primary_and_tracks_last_index(_t make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -8038,7 +8367,7 @@ async fn multilayer_reserved_start_delegates_to_primary_and_tracks_last_index(_t first_pending.wait().await.unwrap(); second_pending.wait().await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let entries = oplog_service .read_exact(&owned_agent_id, AgentMode::Durable, first_idx, 2) @@ -8108,6 +8437,7 @@ async fn ephemeral_reserved_start_uploads_payload_eagerly(_tracing: &Tracing) { metadata, default_last_known_status(), default_execution_status(AgentMode::Ephemeral), + None, ) .await; @@ -8326,6 +8656,7 @@ async fn reserved_start_through_production_stack_smoke(_tracing: &Tracing) { make_agent_metadata(agent_id.clone(), account_id, environment_id), default_last_known_status(), default_execution_status(AgentMode::Durable), + None, ) .await; @@ -8356,7 +8687,7 @@ async fn reserved_start_through_production_stack_smoke(_tracing: &Tracing) { assert_eq!(small_idx, large_idx.next()); small_pending.wait().await.unwrap(); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); // Read back through the service stack (no in-memory cache). let entries = oplog_service @@ -8418,3 +8749,1481 @@ async fn reserved_start_through_production_stack_smoke(_tracing: &Tracing) { other => panic!("unexpected request: {other:?}"), } } + +/// The fence, end to end through the real oplog service, on SQLite. +async fn fencing_oplog_service(tempdir: &tempfile::TempDir, name: &str) -> PrimaryOplogService { + let config = golem_common::config::DbSqliteConfig { + database: tempdir + .path() + .join(format!("{name}.db")) + .to_string_lossy() + .into_owned(), + max_connections: 4, + foreign_keys: false, + }; + let indexed_storage: Arc = + Arc::new(SqliteIndexedStorage::configured(&config).await.unwrap()); + PrimaryOplogService::new( + indexed_storage, + Arc::new(InMemoryBlobStorage::new()), + 100, + 1, + 128, + RetryConfig::default(), + ) + .await +} + +#[test] +async fn an_oplog_opened_at_the_owning_epoch_can_be_written(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let service = fencing_oplog_service(&tempdir, "owning").await; + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "owned".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + + let oplog = service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(7)), + ) + .await; + + // Opening records the epoch, so the writes that follow are accepted. + oplog.add(OplogEntry::suspend().rounded()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(oplog.length().await, 1); +} + +#[test] +async fn an_oplog_opened_at_a_stale_epoch_refuses_every_write(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "moved".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + + // Two services over one database, because that is what two executors are. A single service + // would not do: `OpenOplogs` caches by agent id, so a second `open` on it hands back the + // first oplog - epoch and all - instead of constructing a new one. + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared").await; + + // The shard's new owner takes it over at a higher epoch and writes. + let owner = owning_executor + .open( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(9)), + ) + .await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + + // The executor that lost the shard still believes it holds epoch 8. It is refused at its + // very first write, and told which epoch owns the oplog now. + let loser = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(8)), + ) + .await; + // `add` only buffers; the storage write happens at the commit, so the refusal is asserted + // over the pair rather than over `add` alone. + let write = async { + loser.add(OplogEntry::exited().rounded()).await?; + loser.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.agent_id, agent_id); + assert_eq!(fence.expected_epoch, golem_common::model::ShardEpoch(8)); + assert_eq!( + fence.actual_epoch, + Some(golem_common::model::ShardEpoch(9)), + "the fence must name the epoch that owns the oplog now" + ); + } + other => panic!("expected the write to be fenced, got {other:?}"), + } + + // ... and stays refused: the oplog is poisoned, so it does not even ask the storage again. + assert!(matches!( + loser.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + assert_eq!( + owner.length().await, + 1, + "the losing executor must not have appended to the owner's oplog" + ); +} + +#[test] +async fn an_oplog_opened_without_an_epoch_asserts_nothing(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let service = fencing_oplog_service(&tempdir, "unfenced").await; + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "unfenced".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + + // `None` is what the debugging service and a fork of a remote target pass: no ownership + // claim, so the record is neither written nor checked. + let oplog = service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + None, + ) + .await; + oplog.add(OplogEntry::suspend().rounded()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(oplog.length().await, 1); +} + +#[test] +async fn deleting_an_oplog_fences_a_writer_that_still_holds_it(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let service = fencing_oplog_service(&tempdir, "deleted").await; + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "deleted".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + + let oplog = service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(7)), + ) + .await; + oplog.add(OplogEntry::suspend().rounded()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); + + service + .delete( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + ) + .await + .unwrap(); + + // The handle outlives the delete, as a zombie executor's would. Its epoch is still the one + // the record held, so only the record's absence can refuse it - an entry landing here would + // bring back an oplog that was deleted. + let write = async { + oplog.add(OplogEntry::exited().rounded()).await?; + oplog.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.agent_id, agent_id); + assert_eq!(fence.expected_epoch, golem_common::model::ShardEpoch(7)); + assert_eq!( + fence.actual_epoch, None, + "the record must be gone, not moved to another epoch" + ); + } + other => panic!("expected the write to be fenced, got {other:?}"), + } + assert!( + !service.exists(&owned_agent_id, AgentMode::Durable).await, + "the refused write must not have brought the deleted oplog back" + ); +} + +#[test] +async fn a_fenced_oplog_is_not_handed_out_again_while_it_is_still_held(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "regained".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let executor = fencing_oplog_service(&tempdir, "shared").await; + let other_executor = fencing_oplog_service(&tempdir, "shared").await; + let open = |service: &PrimaryOplogService, epoch: u64| { + let service = service.clone(); + let agent_id = agent_id.clone(); + let owned_agent_id = owned_agent_id.clone(); + async move { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(epoch)), + ) + .await + } + }; + + // This executor holds the shard at epoch 8, loses it to the other executor at 9, and is + // refused - the handle is fenced and stays open, as a worker still stopping keeps it. + let fenced = open(&executor, 8).await; + let owner = open(&other_executor, 9).await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + fenced.add(OplogEntry::exited().rounded()).await.unwrap(); + assert!(matches!( + fenced.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + drop(owner); + + // Re-granted the shard at 10 while the fenced handle is still alive, the executor opens the + // agent again - recovering it - and must not be handed the finished handle: that one refuses + // every write, and its view of the oplog stops where its refused entries began. + let regained = open(&executor, 10).await; + assert!( + !Arc::ptr_eq(®ained, &fenced), + "the fenced handle was handed out again" + ); + regained.add(OplogEntry::exited().rounded()).await.unwrap(); + regained.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(regained.length().await, 2); + + // Dropping the fenced handle runs its remover; it must not evict the fresh handle, or the + // next open would construct a third, and two live writers would share one oplog. + drop(fenced); + let again = open(&executor, 10).await; + assert!( + Arc::ptr_eq(&again, ®ained), + "the fresh handle was evicted by the fenced handle's removal" + ); +} + +/// A below-threshold add on a moved shard only buffers, so nothing refuses it until the commit; +/// the refused commit latches the fence, and from then on the add itself is refused rather than +/// buffered under an index that could never reach the storage. +#[test] +async fn a_below_threshold_add_on_a_moved_shard_is_refused_once_the_commit_latches_the_fence( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "moved".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let executor = fencing_oplog_service(&tempdir, "shared").await; + let other_executor = fencing_oplog_service(&tempdir, "shared").await; + let open = |service: &PrimaryOplogService, epoch: u64| { + let service = service.clone(); + let agent_id = agent_id.clone(); + let owned_agent_id = owned_agent_id.clone(); + async move { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(epoch)), + ) + .await + } + }; + + let stale = open(&executor, 8).await; + let owner = open(&other_executor, 9).await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(stale.fence(), None, "nothing has been refused yet"); + + stale + .add(OplogEntry::exited().rounded()) + .await + .expect("a below-threshold add only buffers, so the moved shard does not refuse it"); + assert!(matches!( + stale.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + let fence = stale + .fence() + .expect("the refused commit must latch the fence before it returns"); + assert_eq!(fence.expected_epoch, ShardEpoch(8)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(9))); + + assert!( + matches!( + stale.add(OplogEntry::exited().rounded()).await, + Err(OplogError::Fenced(_)) + ), + "a latched fence refuses a below-threshold add" + ); + assert!(matches!( + stale.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + assert_eq!( + stale.fence(), + Some(fence), + "the latch keeps the first refusal" + ); +} + +#[test] +async fn an_executor_that_loses_the_shard_mid_flight_is_refused_at_its_next_write( + _tracing: &Tracing, +) { + // The realistic sequence, and the one only the per-write assertion can catch: this executor + // opened the oplog while it still owned the shard, so its epoch record went in cleanly and + // nothing was poisoned at open. The shard moves underneath it afterwards. + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "mid-flight".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + + let losing_executor = fencing_oplog_service(&tempdir, "mid-flight").await; + let gaining_executor = fencing_oplog_service(&tempdir, "mid-flight").await; + + let loser = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(4)), + ) + .await; + loser.add(OplogEntry::suspend().rounded()).await.unwrap(); + loser.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(loser.length().await, 1, "it owned the shard at this point"); + + // The shard is re-granted to another executor, which opens the oplog at the new epoch. + let _gainer = gaining_executor + .open( + &mut gaining_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(5)), + ) + .await; + + // The loser's already-open oplog is not poisoned - it had no reason to be - so this is the + // per-write epoch assertion doing the work, and nothing of its is written. + let write = async { + loser.add(OplogEntry::exited().rounded()).await?; + loser.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.expected_epoch, golem_common::model::ShardEpoch(4)); + assert_eq!(fence.actual_epoch, Some(golem_common::model::ShardEpoch(5))); + } + other => panic!("expected the in-flight write to be fenced, got {other:?}"), + } + assert_eq!( + loser.length().await, + 1, + "the refused entry must not be there" + ); +} + +#[test] +async fn wait_for_replicas_does_not_report_a_fenced_flush_as_durable(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "flushed".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let losing_executor = fencing_oplog_service(&tempdir, "flushed").await; + let owning_executor = fencing_oplog_service(&tempdir, "flushed").await; + + let loser = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(8)), + ) + .await; + let owner = owning_executor + .open( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(9)), + ) + .await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + + // The guest's `oplog-commit` path: the entry is only buffered, and the flush inside + // `wait_for_replicas` is what the storage refuses. The fencing backends have no replicas to + // wait for, so a count taken after the refusal would read as a successful commit. + loser.add(OplogEntry::exited().rounded()).await.unwrap(); + assert!( + !loser.wait_for_replicas(1, Duration::from_secs(1)).await, + "a flush the storage refused must not be reported as durable" + ); + match loser.fence() { + Some(fence) => assert_eq!( + fence.actual_epoch, + Some(golem_common::model::ShardEpoch(9)), + "the fence must name the epoch that owns the oplog now" + ), + None => panic!("the refused flush must latch the fence"), + } + assert_eq!( + owner.length().await, + 1, + "the losing executor must not have appended to the owner's oplog" + ); +} + +#[test] +async fn a_fenced_oplog_refuses_new_adds_and_keeps_the_indices_it_handed_out_readable( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "half-alive".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let losing_executor = fencing_oplog_service(&tempdir, "half-alive").await; + let owning_executor = fencing_oplog_service(&tempdir, "half-alive").await; + + let loser = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(8)), + ) + .await; + let owner = owning_executor + .open( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(golem_common::model::ShardEpoch(9)), + ) + .await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + + // Below the commit threshold both adds only buffer, and each is answered with an index. + let first = loser.add(OplogEntry::suspend().rounded()).await.unwrap(); + loser.add(OplogEntry::exited().rounded()).await.unwrap(); + // A reader takes the horizon before the storage refuses the batch and reads after it, as a + // durable session or a fork running beside the invocation loop does. + let horizon = loser.current_oplog_index().await; + assert!(matches!( + loser.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + let entries = loser + .read_exact(first, horizon.as_u64() - first.as_u64() + 1) + .await; + assert_eq!( + entries.len(), + 2, + "an index handed out before the refusal must still be readable after it" + ); + + // Once latched, an add below the threshold is refused instead of buffered under an index + // that could never reach the storage. + assert!(matches!( + loser.add(OplogEntry::suspend().rounded()).await, + Err(OplogError::Fenced(_)) + )); + assert!(matches!( + loser + .add_pair( + OplogEntry::suspend().rounded(), + Box::new(|_| OplogEntry::exited().rounded()) + ) + .await, + Err(OplogError::Fenced(_)) + )); + assert_eq!( + loser.current_oplog_index().await, + horizon, + "a refused add must not take an index" + ); + assert!(matches!( + loser.commit(CommitLevel::Always).await, + Err(OplogError::Fenced(_)) + )); + assert_eq!( + owner.length().await, + 1, + "the losing executor must not have appended to the owner's oplog" + ); +} + +#[test] +async fn an_opener_at_a_newer_epoch_is_not_handed_the_older_generations_handle(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "came-back".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let executor = fencing_oplog_service(&tempdir, "came-back").await; + let open = |epoch: u64| { + let service = executor.clone(); + let agent_id = agent_id.clone(); + let owned_agent_id = owned_agent_id.clone(); + async move { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(epoch)), + ) + .await + } + }; + + // The shard left this executor at epoch 5 and came back at 7 while the epoch-5 handle is still + // held. Nothing has fenced that handle - nobody has written since - so only the epoch it was + // opened with tells the cache it belongs to the older generation. + let old = open(5).await; + let new = open(7).await; + assert!( + !Arc::ptr_eq(&new, &old), + "the opener at epoch 7 was handed the handle opened at 5" + ); + assert_eq!(new.shard_epoch(), Some(ShardEpoch(7))); + new.add(OplogEntry::suspend().rounded()).await.unwrap(); + new.commit(CommitLevel::Always).await.unwrap(); + + let write = async { + old.add(OplogEntry::exited().rounded()).await?; + old.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.expected_epoch, ShardEpoch(5)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(7))); + } + other => panic!("expected the older generation's write to be fenced, got {other:?}"), + } + + // An equal or older request is handed the current handle. Building another would put two + // live writers on epoch 7, and they would collide on the oplog's keys. + let again = open(7).await; + assert!( + Arc::ptr_eq(&again, &new), + "an opener at the same epoch must share the handle" + ); + let stale = open(5).await; + assert!( + Arc::ptr_eq(&stale, &new), + "an opener at an older epoch must not evict the newer handle" + ); +} + +#[test] +async fn an_ephemeral_handle_is_reused_whatever_epoch_is_requested(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let primary = Arc::new(fencing_oplog_service(&tempdir, "ephemeral").await); + let archive: Arc = Arc::new(CompressedOplogArchiveService::new( + Arc::new(InMemoryIndexedStorage::new()), + 1, + RetryConfig::default(), + )); + let service = MultiLayerOplogService::new(primary, nev![archive], 10, 10); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "ephemeral".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let mut metadata = make_agent_metadata(agent_id, account_id, environment_id); + metadata.agent_mode = AgentMode::Ephemeral; + let open = |epoch: u64| { + let service = service.clone(); + let owned_agent_id = owned_agent_id.clone(); + let metadata = metadata.clone(); + async move { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Ephemeral, + None, + metadata, + default_last_known_status(), + default_execution_status(AgentMode::Ephemeral), + Some(ShardEpoch(epoch)), + ) + .await + } + }; + + // An ephemeral handle asserts no epoch whatever it was opened with, so it belongs to no + // ownership generation and a newer request is no reason to rebuild it. + let first = open(5).await; + assert_eq!(first.shard_epoch(), None); + let second = open(7).await; + assert!( + Arc::ptr_eq(&second, &first), + "an ephemeral handle was rebuilt for a newer epoch" + ); +} + +#[test] +async fn a_fork_target_handle_is_not_reused_by_the_owners_first_open(_tracing: &Tracing) { + let tempdir = tempfile::TempDir::new().unwrap(); + let primary = Arc::new(fencing_oplog_service(&tempdir, "fork-target").await); + let archive: Arc = Arc::new(CompressedOplogArchiveService::new( + Arc::new(InMemoryIndexedStorage::new()), + 1, + RetryConfig::default(), + )); + // Through the layered service, as production opens it: each layer caches its own handle, and + // every one of them has to decline the fork's. + let service = MultiLayerOplogService::new(primary, nev![archive], 10, 10); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "fork-target".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let create_entry = OplogEntry::create(Box::new(golem_common::model::oplog::CreateParameters { + agent_id: agent_id.clone(), + owner_kind: golem_common::model::agent::OwnerKind::ComponentAgent, + agent_mode: AgentMode::Durable, + component_revision: ComponentRevision::new(1).unwrap(), + env: Vec::new(), + environment_id, + created_by: account_id, + parent: None, + component_size: 100, + initial_total_linear_memory_size: 100, + initial_active_plugins: HashSet::new(), + local_agent_config: Vec::new(), + original_phantom_id: None, + instance_id: Uuid::new_v4(), + })) + .rounded(); + + // The fork copies into the target through a handle that asserts no epoch, since the target's + // shard may belong to another executor. Here that handle is still held when the owner opens + // the target. + let forked = service + .create( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + create_entry, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + None, + ) + .await; + forked.add(OplogEntry::suspend().rounded()).await.unwrap(); + forked.commit(CommitLevel::Always).await.unwrap(); + + let owner = service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(4)), + ) + .await; + assert!( + !Arc::ptr_eq(&owner, &forked), + "the owner's first open was handed the fork's unfenced handle" + ); + assert_eq!(owner.shard_epoch(), Some(ShardEpoch(4))); + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + + // Another executor takes the shard at epoch 5. The owner's next write is refused only if its + // open recorded epoch 4; through the fork's handle it would have recorded nothing and been + // accepted. + let other_executor = fencing_oplog_service(&tempdir, "fork-target").await; + let other = other_executor + .open( + &mut other_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + other.add(OplogEntry::suspend().rounded()).await.unwrap(); + other.commit(CommitLevel::Always).await.unwrap(); + + let write = async { + owner.add(OplogEntry::exited().rounded()).await?; + owner.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.expected_epoch, ShardEpoch(4)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(5))); + } + other => panic!("expected the owner's write to be fenced, got {other:?}"), + } +} + +#[test] +async fn an_owner_opening_on_a_stale_last_index_starts_after_the_losers_last_write( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "stale-last-index".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared").await; + + let loser = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + loser.add(OplogEntry::suspend().rounded()).await.unwrap(); + loser.commit(CommitLevel::Always).await.unwrap(); + + // A layer above the primary reads the last index before the primary claims the epoch, as the + // layered service does, and the losing executor commits again in that window: the record + // still says 5, so the write is accepted. + let stale_last_index = owning_executor + .get_last_index(&owned_agent_id, AgentMode::Durable) + .await; + assert_eq!(stale_last_index, OplogIndex::from_u64(1)); + loser.add(OplogEntry::suspend().rounded()).await.unwrap(); + loser.commit(CommitLevel::Always).await.unwrap(); + + let owner = owning_executor + .open( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + Some(stale_last_index), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(6)), + ) + .await; + // Starting from the stale index, this append would reuse the loser's id and fail-stop the + // executor that owns the shard. + owner.add(OplogEntry::exited().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + assert_eq!(owner.length().await, 3); + assert_eq!(owner.current_oplog_index().await, OplogIndex::from_u64(3)); + + let write = async { + loser.add(OplogEntry::exited().rounded()).await?; + loser.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + match write { + Err(OplogError::Fenced(fence)) => { + assert_eq!(fence.expected_epoch, ShardEpoch(5)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(6))); + } + other => panic!("expected the losing executor's write to be fenced, got {other:?}"), + } +} + +#[test] +async fn a_stale_create_of_an_oplog_the_owner_already_created_is_fenced_not_fatal( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "created-twice".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared").await; + let create_entry = || { + OplogEntry::create(Box::new(golem_common::model::oplog::CreateParameters { + agent_id: agent_id.clone(), + owner_kind: golem_common::model::agent::OwnerKind::ComponentAgent, + agent_mode: AgentMode::Durable, + component_revision: ComponentRevision::new(1).unwrap(), + env: Vec::new(), + environment_id, + created_by: account_id, + parent: None, + component_size: 100, + initial_total_linear_memory_size: 100, + initial_active_plugins: HashSet::new(), + local_agent_config: Vec::new(), + original_phantom_id: None, + instance_id: Uuid::new_v4(), + })) + .rounded() + }; + + let owner = owning_executor + .create( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + create_entry(), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(6)), + ) + .await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + + // An executor that lost the shard creates the same agent. Its claim is refused, so it writes + // nothing and must not fail-stop over an oplog that belongs to the owner. + let stale = losing_executor + .create( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + create_entry(), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + match stale.fence() { + Some(fence) => { + assert_eq!(fence.expected_epoch, ShardEpoch(5)); + assert_eq!(fence.actual_epoch, Some(ShardEpoch(6))); + } + None => panic!("the refused create must hand back a fenced oplog"), + } + let write = async { + stale.add(OplogEntry::exited().rounded()).await?; + stale.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + assert!( + matches!(write, Err(OplogError::Fenced(_))), + "expected the stale create's write to be fenced, got {write:?}" + ); + assert_eq!( + owner.length().await, + 2, + "the stale create must not have appended to the owner's oplog" + ); +} + +/// Keeps every fence it is told of, in order. +#[derive(Default)] +struct RecordingFenceObserver { + fences: StdMutex>, +} + +impl RecordingFenceObserver { + fn fences(&self) -> Vec { + self.fences.lock().unwrap().clone() + } +} + +impl OplogFenceObserver for RecordingFenceObserver { + fn fenced(&self, fence: &OplogFence) { + self.fences.lock().unwrap().push(fence.clone()); + } +} + +fn initial_create_entry( + agent_id: &AgentId, + environment_id: EnvironmentId, + account_id: AccountId, +) -> OplogEntry { + OplogEntry::create(Box::new(golem_common::model::oplog::CreateParameters { + agent_id: agent_id.clone(), + owner_kind: golem_common::model::agent::OwnerKind::ComponentAgent, + agent_mode: AgentMode::Durable, + component_revision: ComponentRevision::new(1).unwrap(), + env: Vec::new(), + environment_id, + created_by: account_id, + parent: None, + component_size: 100, + initial_total_linear_memory_size: 100, + initial_active_plugins: HashSet::new(), + local_agent_config: Vec::new(), + original_phantom_id: None, + instance_id: Uuid::new_v4(), + })) + .rounded() +} + +#[test] +async fn a_refused_open_or_create_reports_the_stored_epoch_to_the_fence_observer( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let recorder = Arc::new(RecordingFenceObserver::default()); + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared") + .await + .with_fence_observer(recorder.clone()); + let opened = AgentId { + component_id: ComponentId::new(), + agent_id: "opened-by-the-loser".into(), + }; + let created = AgentId { + component_id: ComponentId::new(), + agent_id: "created-by-the-loser".into(), + }; + let expected_fence = |agent_id: &AgentId| OplogFence { + agent_id: agent_id.clone(), + expected_epoch: ShardEpoch(5), + actual_epoch: Some(ShardEpoch(6)), + writer_conflict: false, + }; + + for agent_id in [&opened, &created] { + let owner = owning_executor + .create( + &mut owning_executor + .lock_lifecycle(&OwnedAgentId::new(environment_id, agent_id).agent_id) + .await, + &OwnedAgentId::new(environment_id, agent_id), + AgentMode::Durable, + initial_create_entry(agent_id, environment_id, account_id), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(6)), + ) + .await; + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); + } + + let stale = losing_executor + .open( + &mut losing_executor + .lock_lifecycle(&OwnedAgentId::new(environment_id, &opened).agent_id) + .await, + &OwnedAgentId::new(environment_id, &opened), + AgentMode::Durable, + None, + make_agent_metadata(opened.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + assert!(stale.fence().is_some(), "the stale open was not refused"); + let reported = recorder.fences(); + assert!( + !reported.is_empty(), + "the refused open reported nothing to the observer" + ); + for fence in &reported { + assert_eq!(fence, &expected_fence(&opened)); + } + + // Born fenced, so its writes fail on the latch without asking the storage again. + let write = async { + stale.add(OplogEntry::exited().rounded()).await?; + stale.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + assert!( + matches!(write, Err(OplogError::Fenced(_))), + "expected the stale open's write to be fenced, got {write:?}" + ); + assert_eq!( + recorder.fences().len(), + reported.len(), + "a write refused by the latch reported a refusal the storage never made" + ); + + // On a cache miss a refused create is reported twice, by `create` and by the open behind it, + // and the observer merges. So what is asserted is what was learned, not how often. + let reported_before_create = recorder.fences().len(); + let stale_create = losing_executor + .create( + &mut losing_executor + .lock_lifecycle(&OwnedAgentId::new(environment_id, &created).agent_id) + .await, + &OwnedAgentId::new(environment_id, &created), + AgentMode::Durable, + initial_create_entry(&created, environment_id, account_id), + make_agent_metadata(created.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await; + assert!( + stale_create.fence().is_some(), + "the stale create was not refused" + ); + let reported = recorder.fences().split_off(reported_before_create); + assert!( + !reported.is_empty(), + "the refused create reported nothing to the observer" + ); + for fence in &reported { + assert_eq!(fence, &expected_fence(&created)); + } +} + +#[test] +async fn a_create_refused_behind_a_cached_handle_still_reports_the_stored_epoch( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "created-again-while-held".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let recorder = Arc::new(RecordingFenceObserver::default()); + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared") + .await + .with_fence_observer(recorder.clone()); + let create_at_5 = || async { + losing_executor + .create( + &mut losing_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + initial_create_entry(&agent_id, environment_id, account_id), + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(5)), + ) + .await + }; + + // Created while this executor owned the shard, and held without a write. + let held = create_at_5().await; + assert!(held.fence().is_none()); + assert!(recorder.fences().is_empty()); + + // The shard moves, and its new owner claims the oplog. + let owner = owning_executor + .open( + &mut owning_executor + .lock_lifecycle(&owned_agent_id.agent_id) + .await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id.clone(), account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(6)), + ) + .await; + assert!(owner.fence().is_none()); + + // The claim is refused, and the open behind it hands back the held handle without asking the + // storage, so the refusal `create` reports is the only one before a write. + let again = create_at_5().await; + assert!( + Arc::ptr_eq(&again, &held), + "the held handle was not handed back, so this is not the cache hit under test" + ); + assert_eq!( + recorder.fences(), + vec![OplogFence { + agent_id: agent_id.clone(), + expected_epoch: ShardEpoch(5), + actual_epoch: Some(ShardEpoch(6)), + writer_conflict: false, + }] + ); +} + +#[test] +async fn a_refused_append_reports_the_stored_epoch_and_the_latch_does_not_report_again( + _tracing: &Tracing, +) { + let tempdir = tempfile::TempDir::new().unwrap(); + let account_id = AccountId::new(); + let environment_id = EnvironmentId::new(); + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "appended-by-the-loser".into(), + }; + let owned_agent_id = OwnedAgentId::new(environment_id, &agent_id); + let recorder = Arc::new(RecordingFenceObserver::default()); + let owning_executor = fencing_oplog_service(&tempdir, "shared").await; + let losing_executor = fencing_oplog_service(&tempdir, "shared") + .await + .with_fence_observer(recorder.clone()); + let open = |service: &PrimaryOplogService, epoch: u64| { + let service = service.clone(); + let agent_id = agent_id.clone(); + let owned_agent_id = owned_agent_id.clone(); + async move { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + &owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata(agent_id, account_id, environment_id), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + Some(ShardEpoch(epoch)), + ) + .await + } + }; + + let stale = open(&losing_executor, 5).await; + assert!(stale.fence().is_none()); + assert!(recorder.fences().is_empty()); + let owner = open(&owning_executor, 6).await; + + let write = async { + stale.add(OplogEntry::exited().rounded()).await?; + stale.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + }; + let refused = write.await; + assert!( + matches!(refused, Err(OplogError::Fenced(_))), + "expected the losing executor's write to be fenced, got {refused:?}" + ); + let expected = OplogFence { + agent_id: agent_id.clone(), + expected_epoch: ShardEpoch(5), + actual_epoch: Some(ShardEpoch(6)), + writer_conflict: false, + }; + assert_eq!( + recorder.fences(), + vec![expected.clone()], + "one refused append is one report" + ); + + let again = async { + stale.add(OplogEntry::exited().rounded()).await?; + stale.commit(CommitLevel::Always).await?; + Ok::<_, OplogError>(()) + } + .await; + assert!(matches!(again, Err(OplogError::Fenced(_)))); + assert_eq!( + recorder.fences(), + vec![expected], + "the latched fast-fail asked the storage nothing, so it must report nothing" + ); + + // The owner, with no observer, writes as before. + owner.add(OplogEntry::suspend().rounded()).await.unwrap(); + owner.commit(CommitLevel::Always).await.unwrap(); +} + +async fn open_unfenced_fork_target( + service: &MultiLayerOplogService, + owned_agent_id: &OwnedAgentId, +) -> Arc { + service + .open( + &mut service.lock_lifecycle(&owned_agent_id.agent_id).await, + owned_agent_id, + AgentMode::Durable, + None, + make_agent_metadata( + owned_agent_id.agent_id.clone(), + AccountId::new(), + owned_agent_id.environment_id, + ), + default_last_known_status(), + default_execution_status(AgentMode::Durable), + None, + ) + .await +} + +#[test] +async fn aborting_a_transfer_waits_for_the_prefix_drop_it_handed_to_the_primary( + _tracing: &Tracing, +) { + let (drop_prefix_started_tx, drop_prefix_started_rx) = oneshot::channel(); + let release_drop_prefix = Arc::new(Notify::new()); + let primary_storage = Arc::new(ReadCountingIndexedStorage::blocking_drop_prefix( + drop_prefix_started_tx, + release_drop_prefix.clone(), + )); + let blob_storage = Arc::new(InMemoryBlobStorage::new()); + let primary = Arc::new( + PrimaryOplogService::new( + primary_storage.clone(), + blob_storage.clone(), + 1, + 1, + 100, + RetryConfig::default(), + ) + .await, + ); + let service = MultiLayerOplogService::new( + primary.clone(), + nev![ + Arc::new(CompressedOplogArchiveService::new( + Arc::new(InMemoryIndexedStorage::new()), + 1, + RetryConfig::default(), + )) as Arc, + Arc::new(BlobOplogArchiveService::new(blob_storage.clone(), 2)) + as Arc + ], + 2, + 1, + ); + let owned_agent_id = OwnedAgentId::new( + EnvironmentId::new(), + &AgentId { + component_id: ComponentId::new(), + agent_id: "fork-target-prefix-drop".to_string(), + }, + ); + let target = open_unfenced_fork_target(&service, &owned_agent_id).await; + + for _ in 0..3 { + target.add(OplogEntry::no_op(None).rounded()).await.unwrap(); + } + target.commit(CommitLevel::Always).await.unwrap(); + tokio::time::timeout(Duration::from_secs(1), drop_prefix_started_rx) + .await + .expect("the transfer did not reach the primary's prefix drop") + .expect("prefix drop start signal dropped"); + + // The transfer is waiting for the primary's actor, which is inside the prefix drop. + let abort = tokio::spawn({ + let target = target.clone(); + async move { MultiLayerOplog::try_abort_transfer(&target).await } + }); + tokio::time::sleep(Duration::from_millis(100)).await; + assert!( + !abort.is_finished(), + "the abort returned while the primary was still dropping the transferred prefix" + ); + + release_drop_prefix.notify_one(); + tokio::time::timeout(Duration::from_secs(1), abort) + .await + .expect("the abort did not return once the prefix drop finished") + .unwrap(); + assert!( + primary + .read_source(&owned_agent_id, AgentMode::Durable, OplogIndex::INITIAL, 3) + .await + .is_empty(), + "the abort returned before the primary finished dropping the transferred prefix" + ); +} + +/// `try_abort_transfer` can land between `append_target` and `drop_source_prefix` (see +/// `BackgroundTransfer::run`'s doc comment): the chunk this test appends models one that already +/// reached the archive when that happened. The real owner's next transfer would start from the +/// same, never-trimmed source range and derive the identical chunk id and bytes - exercised here +/// directly against the archive rather than by racing a real abort, since the archive is what +/// must tolerate the repeat. +#[test] +async fn compressed_archive_append_reconciles_a_resumed_transfers_repeat_chunk(_tracing: &Tracing) { + let indexed_storage = Arc::new(InMemoryIndexedStorage::new()); + let archive_service = + CompressedOplogArchiveService::new(indexed_storage.clone(), 1, RetryConfig::default()); + let owned_agent_id = OwnedAgentId::new( + EnvironmentId::new(), + &AgentId { + component_id: ComponentId::new(), + agent_id: "resumed-transfer".into(), + }, + ); + let archive = archive_service + .open_fresh(&owned_agent_id, AgentMode::Durable) + .await; + + let chunk = vec![ + (OplogIndex::from_u64(1), OplogEntry::suspend().rounded()), + (OplogIndex::from_u64(2), OplogEntry::exited().rounded()), + ]; + archive.append(&chunk).await; + assert_eq!(archive.length().await, 1); + + // The resumed transfer's repeat: identical id, identical bytes. + archive.append(&chunk).await; + assert_eq!( + archive.length().await, + 1, + "a resumed transfer's identical repeat chunk must not duplicate" + ); + + // A different chunk landing at the same id is not explainable as a replay and must stay + // fatal rather than being papered over. + let different_chunk = vec![ + (OplogIndex::from_u64(1), OplogEntry::suspend().rounded()), + (OplogIndex::from_u64(2), OplogEntry::suspend().rounded()), + ]; + assert_panics(archive.append(&different_chunk)).await; +} diff --git a/golem-worker-executor/src/services/oplog_sweep.rs b/golem-worker-executor/src/services/oplog_sweep.rs index 85e76e0acf..8539290e65 100644 --- a/golem-worker-executor/src/services/oplog_sweep.rs +++ b/golem-worker-executor/src/services/oplog_sweep.rs @@ -1493,9 +1493,68 @@ mod tests { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.inner - .append(svc_name, api_name, entity_name, namespace, key, id, value) + .append( + svc_name, + api_name, + entity_name, + namespace, + key, + id, + value, + expected_epoch, + ) + .await + } + + async fn append_many( + &self, + svc_name: &'static str, + api_name: &'static str, + entity_name: &'static str, + namespace: &IndexedStorageNamespace, + key: &str, + pairs: Arc<[(u64, bytes::Bytes)]>, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + self.inner + .append_many( + svc_name, + api_name, + entity_name, + namespace, + key, + pairs, + expected_epoch, + ) + .await + } + + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + epoch: golem_common::model::ShardEpoch, + ) -> Result<(), IndexedStorageError> { + self.inner + .set_key_epoch(svc_name, api_name, namespace, key, epoch) + .await + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + self.inner + .delete_with_epoch(svc_name, api_name, namespace, key, expected_epoch) .await } @@ -1862,6 +1921,7 @@ mod tests { metadata(&owned_agent_id.agent_id, owned_agent_id.environment_id), status_lock(), execution_lock(), + None, ) .await; Ok(match MultiLayerOplog::try_archive_blocking(&oplog).await { @@ -1908,7 +1968,7 @@ mod tests { 1, &HashMap::from([(ShardId::new(0), ShardEpoch(0))]), None, - ShardLeaseRevision(0), + ShardLeaseRevision::of(0), ); shard_service } @@ -1987,11 +2047,12 @@ mod tests { metadata(agent_id, environment_id), status_lock(), execution_lock(), + None, ) .await; - oplog.add(OplogEntry::suspend()).await; - oplog.add(OplogEntry::exited()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.unwrap(); + oplog.add(OplogEntry::exited()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); drop(oplog); } @@ -2834,11 +2895,12 @@ mod tests { metadata(&agent_id, environment_id), status_lock(), execution_lock(), + None, ) .await; - oplog.add(OplogEntry::suspend()).await; - oplog.add(OplogEntry::exited()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.unwrap(); + oplog.add(OplogEntry::exited()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); drop(oplog); let stranded = layers.archives[0] @@ -3117,10 +3179,11 @@ mod tests { metadata(agent_id, environment_id), status_lock(), execution_lock(), + None, ) .await; - oplog.add(OplogEntry::suspend()).await; - oplog.commit(CommitLevel::Always).await; + oplog.add(OplogEntry::suspend()).await.unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); drop(oplog); } @@ -3187,7 +3250,7 @@ mod tests { stranded_ephemeral_oplog(&layers, &agent_id, environment_id).await; let shards = Arc::new(ShardServiceDefault::new()); - shards.register(4, &HashMap::new(), None, ShardLeaseRevision(0)); + shards.register(4, &HashMap::new(), None, ShardLeaseRevision::of(0)); let sweeper = build(&layers, manual(), shards, environment_id, HashSet::new()); sweeper.sweep_once(&CancellationToken::new()).await; @@ -3218,7 +3281,7 @@ mod tests { // The shard moves to another executor before the agent ever went quiet for us. shards - .assign_shards(4, &HashMap::new(), ShardLeaseRevision(1)) + .assign_shards(4, &HashMap::new(), ShardLeaseRevision::of(1)) .expect("assignment"); sweeper.sweep_once(&CancellationToken::new()).await; @@ -3853,7 +3916,7 @@ mod tests { 0, &HashMap::from([(ShardId::new(0), ShardEpoch(0))]), None, - ShardLeaseRevision(0), + ShardLeaseRevision::of(0), ); let sweeper = build(&layers, manual(), shards, environment_id, HashSet::new()); @@ -3878,7 +3941,7 @@ mod tests { 1, &HashMap::from([(ShardId::new(0), ShardEpoch(0))]), Some(Instant::now()), - ShardLeaseRevision(0), + ShardLeaseRevision::of(0), ); let sweeper = build(&layers, manual(), shards, environment_id, HashSet::new()); diff --git a/golem-worker-executor/src/services/quota.rs b/golem-worker-executor/src/services/quota.rs index 034ebf1289..0bfc17be36 100644 --- a/golem-worker-executor/src/services/quota.rs +++ b/golem-worker-executor/src/services/quota.rs @@ -1342,6 +1342,7 @@ mod tests { _port: u16, _pod_name: Option, _executor_id: Uuid, + _previous_shard_epochs: BTreeMap, ) -> Result { unimplemented!() } @@ -1350,6 +1351,7 @@ mod tests { &self, _executor_id: Uuid, _shard_epochs: BTreeMap, + _fenced_shard_epochs: BTreeMap, ) -> Result { unimplemented!() } diff --git a/golem-worker-executor/src/services/rpc.rs b/golem-worker-executor/src/services/rpc.rs index ae795a5034..80a8136c7a 100644 --- a/golem-worker-executor/src/services/rpc.rs +++ b/golem-worker-executor/src/services/rpc.rs @@ -234,9 +234,10 @@ impl DurableStreamReadError { map: impl FnOnce(String) -> E, ) -> Self { match error { - crate::durable_host::durable_stream::StreamStoreError::RecoveryRequired => { - Self::Unavailable - } + // A fenced store is as unavailable here as one awaiting recovery: the stream lives on + // with the shard's new owner. + crate::durable_host::durable_stream::StreamStoreError::RecoveryRequired + | crate::durable_host::durable_stream::StreamStoreError::Fenced(_) => Self::Unavailable, error => Self::Other(map(error.to_string())), } } diff --git a/golem-worker-executor/src/services/scheduler.rs b/golem-worker-executor/src/services/scheduler.rs index bc5d642d58..d8d7ab8154 100644 --- a/golem-worker-executor/src/services/scheduler.rs +++ b/golem-worker-executor/src/services/scheduler.rs @@ -1594,6 +1594,7 @@ mod tests { _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, _fingerprint: golem_common::model::AgentFingerprint, + _expected_epoch: Option, ) -> Result<(), WorkerExecutorError> { Ok(()) } diff --git a/golem-worker-executor/src/services/shard.rs b/golem-worker-executor/src/services/shard.rs index d939deb385..0954a72f31 100644 --- a/golem-worker-executor/src/services/shard.rs +++ b/golem-worker-executor/src/services/shard.rs @@ -14,19 +14,23 @@ use crate::metrics::sharding::*; use crate::model::ShardAssignmentCheck; +use crate::services::oplog::{OplogFence, OplogFenceObserver}; use golem_common::model::{ AgentId, ShardAssignment, ShardDeliveryOutcome, ShardEpoch, ShardId, ShardLeaseRevision, }; use golem_service_base::error::worker_executor::WorkerExecutorError; use itertools::Itertools; -use std::collections::{HashMap, HashSet}; +use std::collections::{BTreeMap, HashMap, HashSet}; use std::convert::identity; -use std::sync::{Arc, RwLock}; +use std::sync::{Arc, Mutex, RwLock}; use std::time::Instant; use tracing::debug; -/// Service for assigning shards to worker executors -pub trait ShardService: Send + Sync { +/// Service for assigning shards to worker executors. +/// +/// Also the oplog fence's observer: the epoch a refused write found belongs to the agent's shard, +/// and this is where the shard an agent routes to is known. +pub trait ShardService: OplogFenceObserver + Send + Sync { /// True once an assignment exists **and** its lease is still live. Gates /// the scheduler's poll loop, which admits work without going through /// `check_admission`. @@ -71,7 +75,7 @@ pub trait ShardService: Send + Sync { ) -> Result; /// A granted lease renewal: the shard manager's set for this executor, at /// a new expiry anchored where the renewal was sent. Normally the set that - /// was claimed; when it is not, it is the manager correcting a push this + /// was held; when it is not, it is the manager correcting a push this /// executor never received, and `set_changed` tells the caller to sweep /// and recover agents exactly as it would for a push. The set gates on /// `revision`; the lease clock always moves, even when the set is stale. @@ -87,10 +91,25 @@ pub trait ShardService: Send + Sync { fn clear_assignment(&self); fn current_assignment(&self) -> Result; fn try_get_current_assignment(&self) -> Option; + /// The epochs refused oplog writes found on the rows, per shard, that no granted renewal has + /// reported yet. A snapshot: learning more afterwards does not change it. + fn fence_learned_epochs(&self) -> BTreeMap; + /// Retires what a granted renewal reported. An entry goes only while its epoch is still at or + /// below the reported one, so an epoch learned after the snapshot was taken is reported again. + fn retire_fence_learned_epochs(&self, reported: &BTreeMap); } pub struct ShardServiceDefault { shard_assignment: Arc>>, + /// The highest epoch a refused oplog write found on the rows, per shard, until a granted + /// renewal has reported it. + /// + /// Evidence, not ownership: somebody wrote at that epoch, and a shard manager whose state lost + /// history has to mint above it. So it is kept apart from the assignment and survives a lost + /// lease, and it is not filtered by ownership - a fence on a shard this executor no longer + /// holds is still what the manager forgot, and the manager re-mints that shard's owner. At + /// most one entry per shard. Never held across an await. + fence_learned: Mutex>, } impl Default for ShardServiceDefault { @@ -103,6 +122,7 @@ impl ShardServiceDefault { pub fn new() -> Self { Self { shard_assignment: Arc::new(RwLock::new(None)), + fence_learned: Mutex::new(BTreeMap::new()), } } @@ -284,16 +304,75 @@ impl ShardService for ShardServiceDefault { fn try_get_current_assignment(&self) -> Option { self.shard_assignment.read().unwrap().clone() } + + fn fence_learned_epochs(&self) -> BTreeMap { + self.fence_learned.lock().unwrap().clone() + } + + fn retire_fence_learned_epochs(&self, reported: &BTreeMap) { + let mut learned = self.fence_learned.lock().unwrap(); + for (shard_id, reported_epoch) in reported { + if learned + .get(shard_id) + .is_some_and(|learned_epoch| learned_epoch <= reported_epoch) + { + learned.remove(shard_id); + } + } + } +} + +impl OplogFenceObserver for ShardServiceDefault { + fn fenced(&self, fence: &OplogFence) { + // Only a record ahead of the epoch this executor asserted says anything the shard manager + // may have lost. An absent record carries no epoch, and one at or below the assertion is + // not a generation above it. + // + // The exception is a record at the assertion held by another writer: that one says the + // manager handed the same generation to two executors, which only a manager that lost its + // state does, and it has to mint past the epoch rather than leave it shared. + let Some(stored) = fence.actual_epoch.filter(|stored| { + *stored > fence.expected_epoch + || (fence.writer_conflict && *stored == fence.expected_epoch) + }) else { + return; + }; + let shard_id = { + let guard = self.shard_assignment.read().unwrap(); + match guard.as_ref() { + // `ShardId::from_agent_id` divides by the count, and the placeholder a + // registration starts from holds zero. + Some(assignment) if assignment.number_of_shards > 0 => { + ShardId::from_agent_id(&fence.agent_id, assignment.number_of_shards) + } + _ => return, + } + }; + debug!( + agent_id = %fence.agent_id, + %shard_id, + expected_epoch = %fence.expected_epoch, + stored_epoch = %stored, + "Learned a shard epoch from a fenced oplog write" + ); + self.fence_learned + .lock() + .unwrap() + .entry(shard_id) + .and_modify(|learned| *learned = (*learned).max(stored)) + .or_insert(stored); + } } /// Records what every delivery updates: the resulting shard count, and, when the delivery was -/// dropped for being older than the last applied, the staleness itself. +/// dropped - older than the last applied, or pushed by a manager this executor does not follow - +/// the drop itself. /// /// One function rather than a line per delivery, so a delivery added later cannot record the /// count and silently forget the staleness. fn record_delivery(delivery: ShardDelivery, outcome: &ShardDeliveryOutcome, assigned: usize) { record_assigned_shard_count(assigned); - if matches!(outcome, ShardDeliveryOutcome::Stale { .. }) { + if !matches!(outcome, ShardDeliveryOutcome::Applied { .. }) { record_stale_shard_delivery(delivery); } } @@ -375,6 +454,97 @@ mod tests { Instant::now() + Duration::from_secs(60) } + fn fence(agent_id: &AgentId, expected: u64, actual: Option) -> OplogFence { + OplogFence { + agent_id: agent_id.clone(), + expected_epoch: ShardEpoch(expected), + actual_epoch: actual.map(ShardEpoch), + writer_conflict: false, + } + } + + fn learned(entries: impl IntoIterator) -> BTreeMap { + entries + .into_iter() + .map(|(shard_id, epoch)| (ShardId::new(shard_id), ShardEpoch(epoch))) + .collect() + } + + /// A refused write's stored epoch is learned under the shard its agent routes to, merged by + /// maximum so a repeated or older report never lowers it. It is evidence rather than + /// ownership: learned whether or not this executor holds the shard, and kept when the lease + /// is lost, because that is exactly when a shard manager that lost history needs it. + #[test] + fn a_fence_learns_the_stored_epoch_keyed_by_the_agents_shard() { + let service = service_holding(&epochs([(3, 0)]), Some(live())); + let on_held = agent_on_shard(3); + + service.fenced(&fence(&on_held, 0, Some(4))); + assert_eq!(service.fence_learned_epochs(), learned([(3, 4)])); + + service.fenced(&fence(&on_held, 0, Some(2))); + assert_eq!( + service.fence_learned_epochs(), + learned([(3, 4)]), + "an older report lowered what was learned" + ); + service.fenced(&fence(&on_held, 0, Some(4))); + assert_eq!( + service.fence_learned_epochs(), + learned([(3, 4)]), + "a repeated report changed what was learned" + ); + + // An absent record carries no epoch, and one at or below the assertion is no generation + // above it. + let on_other = agent_on_shard(1); + service.fenced(&fence(&on_other, 0, None)); + service.fenced(&fence(&on_other, 2, Some(2))); + service.fenced(&fence(&on_other, 3, Some(2))); + assert_eq!(service.fence_learned_epochs(), learned([(3, 4)])); + + let on_unowned = agent_on_shard(5); + service.fenced(&fence(&on_unowned, 0, Some(1))); + assert_eq!(service.fence_learned_epochs(), learned([(3, 4), (5, 1)])); + + service.clear_assignment(); + assert_eq!( + service.fence_learned_epochs(), + learned([(3, 4), (5, 1)]), + "a lost lease dropped the epochs the re-registered executor has to report" + ); + + // With no shard count to route by, a fence is ignored rather than divided by zero. + let unregistered = ShardServiceDefault::new(); + unregistered.fenced(&fence(&on_held, 0, Some(4))); + assert!(unregistered.fence_learned_epochs().is_empty()); + unregistered.with_write_shard_assignment(|shard_assignment| { + *shard_assignment = Some(ShardAssignment::default()) + }); + unregistered.fenced(&fence(&on_held, 0, Some(4))); + assert!(unregistered.fence_learned_epochs().is_empty()); + } + + /// A granted renewal retires the snapshot it reported, and nothing learned since: a higher + /// epoch on a reported shard, or a new shard, still goes with the next renewal. + #[test] + fn retiring_reported_epochs_keeps_ones_learned_since() { + let service = service_holding(&epochs([(3, 0)]), Some(live())); + service.fenced(&fence(&agent_on_shard(3), 0, Some(4))); + let reported = service.fence_learned_epochs(); + + service.fenced(&fence(&agent_on_shard(3), 0, Some(6))); + service.fenced(&fence(&agent_on_shard(5), 0, Some(1))); + service.retire_fence_learned_epochs(&reported); + assert_eq!(service.fence_learned_epochs(), learned([(3, 6), (5, 1)])); + + service.retire_fence_learned_epochs(&learned([(3, 6), (5, 1)])); + assert!(service.fence_learned_epochs().is_empty()); + + service.retire_fence_learned_epochs(&learned([(3, 9)])); + assert!(service.fence_learned_epochs().is_empty()); + } + /// Nothing is installed until a registration: a push, a renewal or a /// revoke that arrives first is refused, never applied to a placeholder /// whose shard count of zero the routing hash would divide by. @@ -386,9 +556,9 @@ mod tests { // `ShardingNotReady` by refreshing its routing table and retrying, and answers an opaque // error by failing the call. for refused in [ - service.assign_shards(SHARDS, &epochs([(0, 1)]), ShardLeaseRevision(1)), - service.update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision(1)), - service.revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision(1)), + service.assign_shards(SHARDS, &epochs([(0, 1)]), ShardLeaseRevision::of(1)), + service.update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision::of(1)), + service.revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision::of(1)), ] { assert!( matches!(refused, Err(WorkerExecutorError::ShardingNotReady)), @@ -409,12 +579,12 @@ mod tests { SHARDS, &epochs([(0, 1), (1, 1)]), Some(live()), - ShardLeaseRevision(5), + ShardLeaseRevision::of(5), ); let before = stale_shard_delivery_count(ShardDelivery::Renewal); let outcome = service - .update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision(3)) + .update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision::of(3)) .expect("a registered executor can be renewed"); assert!( @@ -438,18 +608,18 @@ mod tests { SHARDS, &epochs([(0, 1), (1, 1)]), Some(live()), - ShardLeaseRevision(5), + ShardLeaseRevision::of(5), ); let outcome = service - .revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision(3)) + .revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision::of(3)) .expect("a registered executor can be revoked from"); assert_eq!( outcome, ShardDeliveryOutcome::Stale { - delivered: ShardLeaseRevision(3), - applied: ShardLeaseRevision(5), + delivered: ShardLeaseRevision::of(3), + applied: ShardLeaseRevision::of(5), } ); assert_eq!( @@ -464,7 +634,7 @@ mod tests { // ...while one at or above the applied revision does take the shard. let outcome = service - .revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision(5)) + .revoke_shards(&HashSet::from([ShardId::new(0)]), ShardLeaseRevision::of(5)) .expect("a registered executor can be revoked from"); assert_eq!(outcome, ShardDeliveryOutcome::Applied { set_changed: true }); assert_eq!( @@ -485,7 +655,7 @@ mod tests { assert!(service.check_worker(&on_kept).is_ok()); service - .assign_shards(SHARDS, &epochs([(1, 1)]), ShardLeaseRevision(1)) + .assign_shards(SHARDS, &epochs([(1, 1)]), ShardLeaseRevision::of(1)) .unwrap(); assert_eq!( @@ -510,7 +680,7 @@ mod tests { assert!(service.check_admission(&agent).is_ok()); service - .update_lease(&epochs([(0, 3)]), lapsed(), ShardLeaseRevision(1)) + .update_lease(&epochs([(0, 3)]), lapsed(), ShardLeaseRevision::of(1)) .unwrap(); assert!( @@ -584,19 +754,19 @@ mod tests { SHARDS, &epochs([(0, 1), (1, 1)]), Some(lapsed()), - ShardLeaseRevision(5), + ShardLeaseRevision::of(5), ); assert!(!service.is_ready()); let outcome = service - .update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision(3)) + .update_lease(&epochs([(0, 1)]), live(), ShardLeaseRevision::of(3)) .expect("a registered executor can be renewed"); assert_eq!( outcome, ShardDeliveryOutcome::Stale { - delivered: ShardLeaseRevision(3), - applied: ShardLeaseRevision(5), + delivered: ShardLeaseRevision::of(3), + applied: ShardLeaseRevision::of(5), } ); assert_eq!( diff --git a/golem-worker-executor/src/services/shard_manager.rs b/golem-worker-executor/src/services/shard_manager.rs index 0e13c34a31..303b727271 100644 --- a/golem-worker-executor/src/services/shard_manager.rs +++ b/golem-worker-executor/src/services/shard_manager.rs @@ -22,7 +22,7 @@ use golem_common::model::{ }; use golem_service_base::clients::shard_manager::{ShardLeaseError, ShardManagerError}; use std::collections::{BTreeMap, HashMap}; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering}; use std::sync::{Arc, RwLock, Weak}; use std::time::{Duration, Instant}; use tracing::{error, info, warn}; @@ -87,6 +87,11 @@ pub trait ShardManagerService: Send + Sync { /// Graceful release of the shard lease. Never fails a shutdown. async fn deregister(&self); + /// Runs the next renewal pass now instead of when its timer fires. Called when a push arrived + /// from a shard manager process this executor does not follow: the renewal's answer names + /// the process in charge and carries its set. No-op by default. + fn renew_now(&self) {} + /// Installs the hook fired when a re-registration or a corrected renewal /// replaces this executor's shard assignment. No-op by default: an /// implementation that never re-registers has nothing to announce. @@ -117,6 +122,106 @@ async fn sleep_or_park(delay: RenewalDelay) { } } +/// Coordinates the announcements made from the renewal loop, so a slow one cannot stall lease +/// renewal. +/// +/// Giving up a still-loading agent waits for its whole component load and replay with no +/// timeout, so a set-changing renewal that awaited [`GrpcShardManagerService::announce_assignment_changed`] +/// inline would send no further renewal RPCs until that sweep finished, and the lease lapses for +/// the whole executor - the bug this exists to fix. At most one announcement runs at a time; a +/// request that arrives while one is already running is coalesced into exactly one more run after +/// it finishes, so nothing requested is ever dropped, and a loop that keeps correcting the set +/// does not pile up concurrent sweeps racing each other over the same agents. +/// +/// Deliberately not tracked by [`Shutdown::spawn`]: a sweep can run for as long as the slowest +/// agent's replay, and the renewal loop's own deregister - which *is* tracked, so `main` waits for +/// it - has to still land inside the shutdown grace regardless. Tracking this task the same way +/// would let a stale sweep hold the process open past that grace for a recovery nobody is waiting +/// on any more; a bare `tokio::spawn` is simply cut off at its next await point when the runtime +/// is dropped, which is the right fate for best-effort work like this one. +struct AnnouncementSingleFlight { + state: AtomicU8, +} + +impl AnnouncementSingleFlight { + /// No task running. + const IDLE: u8 = 0; + /// A task is running the announcement. + const RUNNING: u8 = 1; + /// A task is running, and a request arrived while it did; it reruns once more before going + /// idle. + const RUNNING_PENDING: u8 = 2; + + fn new() -> Self { + Self { + state: AtomicU8::new(Self::IDLE), + } + } + + /// Requests one announcement, without waiting for it to run. + /// + /// Spawns the task that runs it only when none is already running; a request that arrives + /// mid-run instead marks the running task to loop once more, so a burst of requests while a + /// sweep is in flight still produces at most one extra run after it. + fn request(svc: &Arc) { + loop { + let state = &svc.announcement_single_flight.state; + match state.compare_exchange( + Self::IDLE, + Self::RUNNING, + Ordering::SeqCst, + Ordering::SeqCst, + ) { + Ok(_) => { + let svc = svc.clone(); + tokio::spawn(Self::run(svc)); + return; + } + Err(Self::RUNNING) => { + match state.compare_exchange( + Self::RUNNING, + Self::RUNNING_PENDING, + Ordering::SeqCst, + Ordering::SeqCst, + ) { + Ok(_) => return, + // The running task raced this and just went idle (or another requester + // already coalesced): re-read the state and try again from scratch. + Err(_) => continue, + } + } + // Already coalesced: a rerun is guaranteed without this request doing anything. + Err(_) => return, + } + } + } + + /// The task body [`Self::request`] spawns: runs the announcement, then either goes idle or, + /// if a request was coalesced while it ran, runs it once more. + async fn run(svc: Arc) { + loop { + svc.announce_assignment_changed().await; + let state = &svc.announcement_single_flight.state; + match state.compare_exchange( + Self::RUNNING, + Self::IDLE, + Ordering::SeqCst, + Ordering::SeqCst, + ) { + Ok(_) => return, + Err(Self::RUNNING_PENDING) => { + state.store(Self::RUNNING, Ordering::SeqCst); + continue; + } + Err(other) => unreachable!( + "AnnouncementSingleFlight in state {other}, which only `request` and `run` \ + touch and neither ever stores it" + ), + } + } + } +} + pub struct GrpcShardManagerService { client: Arc, shard_service: Arc, @@ -128,10 +233,19 @@ pub struct GrpcShardManagerService { /// The registration arguments, kept so a re-register after `LeaseNotFound` /// can repeat it without going back through `WorkerExecutorImpl`. registration: RwLock)>>, + /// The shards, with their epochs, this process held before a `LeaseNotFound`, sent with the + /// re-registration that follows so a shard manager whose state lost history mints above them. + /// It has to outlive a failed re-registration: the assignment is already cleared by then, so + /// the next attempt has nothing else to read the set from. Merged by maximum epoch, and emptied + /// once a registration or a renewal succeeds, because the manager then knows this executor + /// again and the next loss carries only what it held since. + carried_epochs: RwLock>, /// Weak self-reference, so `register` can spawn the renewal loop without /// the loop keeping this service alive. me: Weak, renewal_loop_started: AtomicBool, + /// Wakes the renewal loop ahead of its timer; see [`ShardManagerService::renew_now`]. + renew_now: Arc, /// Exponential backoff for failed renewals: doubles from /// `MIN_RENEWAL_INTERVAL` up to `retry_cap()`, reset by every grant. retry_backoff: RwLock, @@ -159,6 +273,9 @@ pub struct GrpcShardManagerService { /// Source of tickets; never reused, so an attempt that outlives a full clear-and-defer cycle /// cannot collide with a later one. recovery_tickets: AtomicU64, + /// Coordinates announcements requested from the renewal loop; see + /// [`AnnouncementSingleFlight`]. + announcement_single_flight: AnnouncementSingleFlight, } impl GrpcShardManagerService { @@ -189,14 +306,17 @@ impl GrpcShardManagerService { shutdown, executor_id: RwLock::new(Uuid::new_v4()), registration: RwLock::new(None), + carried_epochs: RwLock::new(BTreeMap::new()), me: me.clone(), renewal_loop_started: AtomicBool::new(false), + renew_now: Arc::new(tokio::sync::Notify::new()), retry_backoff: RwLock::new(MIN_RENEWAL_INTERVAL), granted_cadence: RwLock::new(None), assignment_changed_hook: RwLock::new(None), rpc_deadline_floor, recovery_outstanding: AtomicU64::new(0), recovery_tickets: AtomicU64::new(0), + announcement_single_flight: AnnouncementSingleFlight::new(), }) } @@ -301,6 +421,7 @@ impl GrpcShardManagerService { ); let svc_weak = self.me.clone(); let shutdown_token = self.shutdown.token(); + let renew_now = self.renew_now.clone(); // Through the shutdown tracker rather than `tokio::spawn`: the shutdown // arm below deregisters, and `main` waits for tracked tasks so that RPC // lands before the runtime is torn down. @@ -317,6 +438,7 @@ impl GrpcShardManagerService { // `None` parks here forever, so a never-expiring lease // issues no renewal RPCs at all. _ = sleep_or_park(renewal_delay) => {} + _ = renew_now.notified() => {} } let svc = match svc_weak.upgrade() { Some(svc) => svc, @@ -325,93 +447,108 @@ impl GrpcShardManagerService { break; } }; - // Raced against the token as well, not just the sleep before it: a renewal can - // be followed by a re-registration and a full agent recovery, and a termination - // signal arriving meanwhile has to be seen inside the grace `main` waits, which + // Raced against the token as well, not just the sleep before it: a termination + // signal arriving mid-renewal has to be seen inside the grace `main` waits, which // is the window the deregister below must be sent in. Abandoning the renewal // costs nothing: the process is stopping and handing the lease back. + // + // The renewal itself never runs the announcement inline any more - see + // `renew_shard_lease_internal` and `AnnouncementSingleFlight` - specifically so + // that an unbounded sweep following a set change or a re-registration can never + // be what the token is raced against here: this arm always returns promptly, and + // the loop keeps renewing while a sweep runs in its own task. tokio::select! { _ = shutdown_token.cancelled() => { svc.deregister().await; break; } - delay = svc.renew_shard_lease() => renewal_delay = delay, + (delay, announcement_owed) = svc.renew_shard_lease_internal() => { + renewal_delay = delay; + if announcement_owed { + AnnouncementSingleFlight::request(&svc); + } + } } } }); } - /// Applies a granted lease and returns the cadence for the next pass. + /// Applies a granted lease and returns the cadence for the next pass, together with whether + /// [`Self::announce_assignment_changed`] is owed - the caller decides how to run it, rather + /// than this awaiting it inline, so that a caller which must not block on a slow sweep (the + /// renewal loop, via `renew_shard_lease_internal`) can hand it to + /// [`AnnouncementSingleFlight`] instead. [`ShardManagerService::renew_shard_lease`] awaits it + /// right here for every other caller, so nothing outside the loop observes any change: a test + /// calling it directly still sees the hook run before it returns. /// /// The grant is the shard manager's set for this executor. Normally that - /// is exactly what was claimed, and only the lease clock moves. When it is + /// is exactly what was held, and only the lease clock moves. When it is /// not, the manager is correcting a push this executor never received, /// wider or narrower, and it is an assignment change like any other: the /// hook runs the same sweep-and-recover the push path does. A grant whose /// set is older than the last delivery applied crossed with a push on the /// network: its set is ignored, or it would put the older set back, and /// its lease is adopted, because it answers this executor's own request. - async fn adopt_lease( + fn adopt_lease( &self, shard_epochs: BTreeMap, expires_at: Instant, revision: ShardLeaseRevision, - ) -> RenewalDelay { + ) -> (RenewalDelay, bool) { let shard_epochs: HashMap = shard_epochs.into_iter().collect(); - match self - .shard_service - .update_lease(&shard_epochs, expires_at, revision) - { + let announcement_owed = match self.shard_service.update_lease( + &shard_epochs, + expires_at, + revision, + ) { Ok(ShardDeliveryOutcome::Applied { set_changed: true }) => { info!( %revision, "Shard lease renewal corrected the shard set; sweeping and recovering agents" ); - self.announce_assignment_changed().await + true } Ok(ShardDeliveryOutcome::Applied { set_changed: false }) => { if self.recovery_outstanding.load(Ordering::SeqCst) != 0 { info!(%revision, "Running the agent recovery still owed from an earlier delivery"); - self.announce_assignment_changed().await + true + } else { + false } } - Ok(ShardDeliveryOutcome::Stale { delivered, applied }) => warn!( - %delivered, - %applied, - "Ignoring the shard set of a renewal older than the last delivery applied; the lease clock moved" - ), - Err(error) => warn!(%error, "Failed to apply a renewed shard lease"), - } + Ok(ShardDeliveryOutcome::Stale { delivered, applied }) => { + warn!( + %delivered, + %applied, + "Ignoring the shard set of a renewal older than the last delivery applied; the lease clock moved" + ); + false + } + // Only a push is refused for its sender: a reply's sender is the one followed. + Ok(ShardDeliveryOutcome::FromAnotherManager { .. }) => false, + Err(error) => { + warn!(%error, "Failed to apply a renewed shard lease"); + false + } + }; let cadence = renewal_interval_for(Some(expires_at), Instant::now()); self.record_granted(cadence); - cadence + (cadence, announcement_owed) } -} -/// `(expires_at - now) / 3`, floored, so three attempts fit inside one lease. -/// -/// A lease that never expires yields `None`, which parks the -/// renewal loop instead of polling it — there is nothing to renew, and a -/// polling loop would be one wasted RPC per second per executor. -fn renewal_interval_for(expires_at: Option, now: Instant) -> RenewalDelay { - let expires_at = expires_at?; - Some(shard_lease::renewal_interval( - expires_at.saturating_duration_since(now), - )) -} - -#[async_trait] -impl ShardManagerService for GrpcShardManagerService { - async fn register( + /// [`ShardManagerService::register`], carrying `previous_epochs` to the shard manager: the set + /// this process held under an earlier `executor_id`, or empty on a first registration. + async fn register_with_previous_epochs( &self, port: u16, pod_name: Option, + previous_epochs: BTreeMap, ) -> Result { *self.registration.write().unwrap() = Some((port, pod_name.clone())); let registration = self .client - .register(port, pod_name, self.executor_id()) + .register(port, pod_name, self.executor_id(), previous_epochs) .await?; let number_of_shards: usize = registration.number_of_shards.try_into().map_err(|_| { @@ -449,44 +586,97 @@ impl ShardManagerService for GrpcShardManagerService { Ok(assignment) } - async fn renew_shard_lease(&self) -> RenewalDelay { - let claim = match self.shard_service.current_assignment() { - Ok(assignment) => assignment.claim(), + /// Adds the set held right now to the carried epochs, keeping the higher epoch where both name + /// a shard, and returns what the next re-registration sends. A shard the current set no longer + /// holds stays in the carried epochs: its epoch is still evidence the manager may have lost. + fn carry_current_epochs(&self) -> BTreeMap { + let held = self + .shard_service + .try_get_current_assignment() + .map(|assignment| assignment.held_epochs()) + .unwrap_or_default(); + let mut carried = self.carried_epochs.write().unwrap(); + for (shard_id, epoch) in held { + carried + .entry(shard_id) + .and_modify(|carried_epoch| *carried_epoch = (*carried_epoch).max(epoch)) + .or_insert(epoch); + } + carried.clone() + } + + /// [`ShardManagerService::renew_shard_lease`], minus running the announcement it can end up + /// owing: returns the delay for the next pass and whether + /// [`Self::announce_assignment_changed`] still needs to run. Every caller other than the + /// renewal loop goes through the trait method, which awaits it right there - this split exists + /// solely so the loop (`start_renewal_loop`) can hand the announcement to + /// [`AnnouncementSingleFlight`] instead of awaiting it inline, so a slow sweep cannot stall the + /// next renewal RPC. + async fn renew_shard_lease_internal(&self) -> (RenewalDelay, bool) { + let held = match self.shard_service.current_assignment() { + Ok(assignment) => assignment.held_epochs(), Err(error) => { warn!(%error, "Skipping shard lease renewal, no shard assignment yet"); - return self.next_retry_delay(); + return (self.next_retry_delay(), false); } }; let executor_id = self.executor_id(); + // Reported on every renewal and retired only by a grant, so a renewal that is refused or + // lost leaves them for the next one; a manager that already applied them moves nothing + // the second time. + let fenced = self.shard_service.fence_learned_epochs(); + if !fenced.is_empty() { + info!( + fenced_shard_epochs = ?fenced, + "Reporting shard epochs learned from fenced oplog writes with the lease renewal" + ); + } let deadline = self.rpc_deadline(); - let renewed = - match tokio::time::timeout(deadline, self.client.renew_shard_lease(executor_id, claim)) - .await - { - Ok(renewed) => renewed, - Err(_elapsed) => { - warn!( - deadline_ms = deadline.as_millis(), - "Shard lease renewal did not answer in time; backing off" - ); - return self.next_retry_delay(); - } - }; + let renewed = match tokio::time::timeout( + deadline, + self.client + .renew_shard_lease(executor_id, held, fenced.clone()), + ) + .await + { + Ok(renewed) => renewed, + Err(_elapsed) => { + warn!( + deadline_ms = deadline.as_millis(), + "Shard lease renewal did not answer in time; backing off" + ); + return (self.next_retry_delay(), false); + } + }; match renewed { Ok(lease) => { + // The manager knows this executor, so its state has the history the carried epochs + // were kept to restore; one stored by a re-registration whose reply was lost + // lands here too. + self.carried_epochs.write().unwrap().clear(); + // Stored with this renewal. Only what was sent is retired: an epoch learned while + // the renewal was in flight goes with the next one. + self.shard_service.retire_fence_learned_epochs(&fenced); self.adopt_lease(lease.shard_epochs, lease.expires_at, lease.revision) - .await } Err(ShardLeaseError::LeaseNotFound(details)) => { // The manager no longer knows this executor. Drop every shard // (an empty set fences every agent) and come back as a new // instance at the same address, which is the restarted-executor - // path the manager already handles. + // path the manager already handles. The set held until now goes + // with the registration: if the manager's state was wiped or + // replaced, it has lost the epochs this executor's oplog rows + // were written at, and would otherwise mint below them. Epochs + // learned from fenced writes do not go with it: a registration + // applies what it carries as this executor's own held epochs, and + // would stamp another writer's epoch onto its entries. They + // ride on the first renewal under the fresh id instead. warn!( details, "Shard lease not found, clearing the assignment and re-registering" ); + let previous_epochs = self.carry_current_epochs(); self.shard_service.clear_assignment(); let fresh_executor_id = Uuid::new_v4(); *self.executor_id.write().unwrap() = fresh_executor_id; @@ -495,28 +685,56 @@ impl ShardManagerService for GrpcShardManagerService { match registration { None => { error!("Cannot re-register: this executor never completed a registration"); - self.next_retry_delay() + (self.next_retry_delay(), false) } - Some((port, pod_name)) => match self.register(port, pod_name).await { - Ok(assignment) => { - self.shard_service.register( + // Bounded like a renewal: the assignment is already cleared, so a manager that + // accepts the call and stalls would otherwise hold this executor with no shards + // and no registration for as long as it stalls. + Some((port, pod_name)) => match tokio::time::timeout( + self.rpc_deadline(), + self.register_with_previous_epochs(port, pod_name, previous_epochs), + ) + .await + { + Err(_) => { + warn!( + deadline_ms = self.rpc_deadline().as_millis(), + "Re-registration after a lost lease timed out, retrying" + ); + (self.next_retry_delay(), false) + } + Ok(Ok(assignment)) => { + self.carried_epochs.write().unwrap().clear(); + let outcome = self.shard_service.register( assignment.number_of_shards, &assignment.shard_epochs, assignment.expires_at, assignment.revision, ); + if let ShardDeliveryOutcome::Stale { delivered, applied } = outcome { + // A manager that names its incarnation cannot get here: its + // revisions start over. One that does not, on a store that lost + // its history, leaves this executor on the cleared set until its + // revisions pass the one applied before. + warn!( + %delivered, + %applied, + "The re-registration's shard set is older than the last delivery applied and was ignored" + ); + } info!( executor_id = %fresh_executor_id, "Re-registered with the shard manager after a lost lease" ); // The same announcement the initial - // registration and `assign_shards_internal` make. - self.announce_assignment_changed().await; - renewal_interval_for(assignment.expires_at, Instant::now()) + // registration and `assign_shards_internal` make - owed to the + // caller, exactly like a set-changing grant. + let delay = renewal_interval_for(assignment.expires_at, Instant::now()); + (delay, true) } - Err(error) => { + Ok(Err(error)) => { warn!(%error, "Re-registration after a lost lease failed"); - self.next_retry_delay() + (self.next_retry_delay(), false) } }, } @@ -526,22 +744,62 @@ impl ShardManagerService for GrpcShardManagerService { // runs down on its own and the self-fence starts refusing // admission the moment it passes. warn!(%error, "Shard lease renewal failed, retrying"); - self.next_retry_delay() + (self.next_retry_delay(), false) } } } +} + +/// `(expires_at - now) / 3`, floored, so three attempts fit inside one lease. +/// +/// A lease that never expires yields `None`, which parks the +/// renewal loop instead of polling it — there is nothing to renew, and a +/// polling loop would be one wasted RPC per second per executor. +fn renewal_interval_for(expires_at: Option, now: Instant) -> RenewalDelay { + let expires_at = expires_at?; + Some(shard_lease::renewal_interval( + expires_at.saturating_duration_since(now), + )) +} + +#[async_trait] +impl ShardManagerService for GrpcShardManagerService { + fn renew_now(&self) { + self.renew_now.notify_one(); + } + + async fn register( + &self, + port: u16, + pod_name: Option, + ) -> Result { + self.register_with_previous_epochs(port, pod_name, BTreeMap::new()) + .await + } + + /// Drives [`Self::renew_shard_lease_internal`] and, unlike the renewal loop, awaits the + /// announcement it may come back owing right here - so every caller other than the loop + /// itself (every test in this file included) sees exactly the synchronous behaviour this had + /// before the split: the hook has run by the time this returns. + async fn renew_shard_lease(&self) -> RenewalDelay { + let (delay, announcement_owed) = self.renew_shard_lease_internal().await; + if announcement_owed { + self.announce_assignment_changed().await; + } + delay + } async fn deregister(&self) { - let claim = self + let held = self .shard_service .try_get_current_assignment() - .map(|assignment| assignment.claim()) + .map(|assignment| assignment.held_epochs()) .unwrap_or_default(); let executor_id = self.executor_id(); match tokio::time::timeout( shutdown::DEREGISTER_DEADLINE, - self.client.deregister(executor_id, claim), + self.client.deregister(executor_id, held), ) .await { @@ -607,6 +865,7 @@ impl ShardManagerService for ShardManagerServiceSingleShard { #[cfg(test)] mod tests { use super::*; + use crate::services::oplog::{OplogFence, OplogFenceObserver}; use crate::services::shard::ShardServiceDefault; use golem_common::model::component::ComponentId; use golem_common::model::environment::EnvironmentId; @@ -635,7 +894,7 @@ mod tests { .collect() } - fn claim(entries: impl IntoIterator) -> BTreeMap { + fn epoch_map(entries: impl IntoIterator) -> BTreeMap { entries .into_iter() .map(|(shard_id, epoch)| (ShardId::new(shard_id), ShardEpoch(epoch))) @@ -665,13 +924,32 @@ mod tests { ShardRegistration { number_of_shards: SHARDS as u32, lease: ShardLease { - shard_epochs: claim(shard_epochs), + shard_epochs: epoch_map(shard_epochs), expires_at, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }, } } + fn revision_of(manager: Uuid, number: u64) -> ShardLeaseRevision { + ShardLeaseRevision { + incarnation: Some(manager), + number, + } + } + + /// [`registration`], answered by the shard manager process `manager` at `revision`. + fn registration_from( + manager: Uuid, + revision: u64, + expires_at: Instant, + shard_epochs: impl IntoIterator, + ) -> ShardRegistration { + let mut registration = registration(expires_at, shard_epochs); + registration.lease.revision = revision_of(manager, revision); + registration + } + type RegisterFn = Box Result + Send + Sync>; type RenewFn = Box< @@ -690,9 +968,16 @@ mod tests { renew_gate: StdMutex>>, /// The same for a deregistration. deregister_gate: StdMutex>>, - register_calls: StdMutex>, + /// Each registration's executor id and the previous epochs it carried. + register_calls: StdMutex)>>, renew_calls: StdMutex)>>, + /// The fenced epochs each renewal reported, in the order of `renew_calls`. + renew_fenced_calls: StdMutex>>, deregister_calls: StdMutex)>>, + /// How many of the next registrations never answer. + hanging_registrations: std::sync::atomic::AtomicUsize, + /// Holds only the first renewal, until notified. + first_renewal_gate: StdMutex>>, } impl MockShardManager { @@ -704,10 +989,23 @@ mod tests { deregister_gate: StdMutex::new(None), register_calls: StdMutex::new(Vec::new()), renew_calls: StdMutex::new(Vec::new()), + renew_fenced_calls: StdMutex::new(Vec::new()), deregister_calls: StdMutex::new(Vec::new()), + hanging_registrations: std::sync::atomic::AtomicUsize::new(0), + first_renewal_gate: StdMutex::new(None), } } + fn with_first_renewal_gate(self, gate: Arc) -> Self { + *self.first_renewal_gate.lock().unwrap() = Some(gate); + self + } + + fn hang_next_registrations(&self, count: usize) { + self.hanging_registrations + .store(count, std::sync::atomic::Ordering::SeqCst); + } + fn with_register( self, f: impl Fn(Uuid) -> Result + Send + Sync + 'static, @@ -737,7 +1035,7 @@ mod tests { self } - fn register_calls(&self) -> Vec { + fn register_calls(&self) -> Vec<(Uuid, BTreeMap)> { self.register_calls.lock().unwrap().clone() } @@ -745,6 +1043,10 @@ mod tests { self.renew_calls.lock().unwrap().clone() } + fn renew_fenced_calls(&self) -> Vec> { + self.renew_fenced_calls.lock().unwrap().clone() + } + fn deregister_calls(&self) -> Vec<(Uuid, BTreeMap)> { self.deregister_calls.lock().unwrap().clone() } @@ -761,8 +1063,23 @@ mod tests { _port: u16, _pod_name: Option, executor_id: Uuid, + previous_shard_epochs: BTreeMap, ) -> Result { - self.register_calls.lock().unwrap().push(executor_id); + self.register_calls + .lock() + .unwrap() + .push((executor_id, previous_shard_epochs)); + let hangs = self + .hanging_registrations + .fetch_update( + std::sync::atomic::Ordering::SeqCst, + std::sync::atomic::Ordering::SeqCst, + |left| left.checked_sub(1), + ) + .is_ok(); + if hangs { + std::future::pending::<()>().await; + } let guard = self.register_fn.lock().unwrap(); let f = guard.as_ref().expect("register_fn not configured"); f(executor_id) @@ -772,16 +1089,26 @@ mod tests { &self, executor_id: Uuid, shard_epochs: BTreeMap, + fenced_shard_epochs: BTreeMap, ) -> Result { self.renew_calls .lock() .unwrap() .push((executor_id, shard_epochs.clone())); + self.renew_fenced_calls + .lock() + .unwrap() + .push(fenced_shard_epochs); // Cloned out before the await: the guard must not be held across it. let gate = self.renew_gate.lock().unwrap().clone(); if let Some(gate) = gate { gate.notified().await; } + let first = self.renew_calls.lock().unwrap().len() == 1; + let first_gate = self.first_renewal_gate.lock().unwrap().clone(); + if let (true, Some(gate)) = (first, first_gate) { + gate.notified().await; + } let guard = self.renew_fn.lock().unwrap(); let f = guard.as_ref().expect("renew_fn not configured"); f(executor_id, shard_epochs) @@ -917,11 +1244,11 @@ mod tests { let mock = Arc::new( MockShardManager::new() .with_register(move |_| Ok(registration(expiry, [(0, 1)]))) - .with_renew(move |_, claimed| { + .with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) }) .with_renew_gate(gate.clone()), @@ -978,7 +1305,7 @@ mod tests { lease: ShardLease { shard_epochs: BTreeMap::new(), expires_at: expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }, }) })); @@ -1008,11 +1335,11 @@ mod tests { let mock = Arc::new( MockShardManager::new() .with_register(move |_| Ok(registration(expiry, [(0, 1)]))) - .with_renew(move |_, claimed| { + .with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) }) .with_renew_gate(gate.clone()), @@ -1048,6 +1375,302 @@ mod tests { ); } + #[test] + // A reply from another shard-manager process is believed because it answers the request this + // executor has just made. That rests on the renewal loop having one request out at a time and + // giving up on it for good at its deadline: a reply from a manager the executor has since + // stopped following, delivered late, would otherwise be adopted and take it back to that + // manager's set. + async fn a_reply_that_arrives_after_its_renewal_was_given_up_on_is_never_applied() { + let expiry = Instant::now() + Duration::from_secs(3); + let (old_manager, new_manager) = (Uuid::new_v4(), Uuid::new_v4()); + let late_reply = Arc::new(tokio::sync::Notify::new()); + let renewals = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration_from(old_manager, 10, expiry, [(0, 1)]))) + .with_renew({ + let renewals = renewals.clone(); + move |_, _| { + let lease = + if renewals.fetch_add(1, std::sync::atomic::Ordering::SeqCst) == 0 { + // The old manager's answer to the first renewal, which the loop has + // given up on by the time it is released. + ShardLease { + shard_epochs: epoch_map([(1, 7)]), + expires_at: Instant::now() + Duration::from_secs(3), + revision: revision_of(old_manager, 11), + } + } else { + ShardLease { + shard_epochs: epoch_map([(0, 2)]), + expires_at: Instant::now() + Duration::from_secs(3), + revision: revision_of(new_manager, 1), + } + }; + Ok(lease) + } + }) + .with_first_renewal_gate(late_reply.clone()), + ); + let (service, shard_service) = make_service_with_rpc_deadline_floor( + mock.clone(), + Shutdown::new(), + Duration::from_millis(200), + ); + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + let follows = |manager: Uuid| { + shard_service + .current_assignment() + .is_ok_and(|assignment| assignment.revision.incarnation == Some(manager)) + }; + for _ in 0..160 { + if follows(new_manager) { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!( + follows(new_manager), + "the renewal after the abandoned one must be answered by the new manager" + ); + + // Now the old manager's answer to the abandoned renewal comes back. + late_reply.notify_waiters(); + tokio::time::sleep(Duration::from_millis(500)).await; + + let assignment = shard_service.current_assignment().unwrap(); + assert_eq!(assignment.revision.incarnation, Some(new_manager)); + assert_eq!(assignment.shard_epochs, epochs([(0, 2)])); + } + + #[test] + // Re-registering after a lost lease starts with the assignment already cleared, so a manager + // that accepts the call and never answers must not hold the executor there: the attempt is + // given up on at the per-attempt deadline and the loop tries again. + async fn a_re_registration_that_never_answers_times_out_and_is_tried_again() { + let expiry = Instant::now() + Duration::from_secs(3); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration(expiry, [(0, 1)]))) + .with_renew(|_, _| Err(ShardLeaseError::LeaseNotFound("unknown".to_string()))), + ); + let (service, shard_service) = make_service_with_rpc_deadline_floor( + mock.clone(), + Shutdown::new(), + Duration::from_millis(200), + ); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + mock.hang_next_registrations(1); + + // The first renewal is refused, the re-registration it triggers never answers, and only a + // deadline on it lets the loop reach a second one. + for _ in 0..160 { + if mock.register_calls().len() >= 3 { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!( + mock.register_calls().len() >= 3, + "a re-registration that never answers must time out so the loop can try again; \ + registrations made: {}", + mock.register_calls().len() + ); + } + + #[test] + // Giving a still-loading agent up waits for its whole component load and replay, with no + // timeout. If the renewal loop awaited the assignment-changed hook inline, that sweep would + // hold every later renewal RPC back and the lease would lapse for the whole executor while a + // single agent is loading. The hook here blocks on a flag the test controls, standing in for + // that sweep, and the assertion is that the loop keeps renewing right through it. + async fn a_slow_assignment_changed_hook_does_not_stall_the_renewal_loop() { + // Short enough that a stalled loop would be obvious within the test's own timeout, but + // the cadence this derives floors at `MIN_RENEWAL_INTERVAL` (1 s) regardless - see + // `renewal_interval` - so there is no point going below that. + let ttl = Duration::from_secs(3); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration(Instant::now() + ttl, [(0, 1)]))) + .with_renew(move |_, _held| { + Ok(ShardLease { + // Always the widened set: the first grant is a real assignment change, + // and every later one only echoes it back (it now matches what was + // held), so only the first renewal ever owes an announcement. + shard_epochs: epoch_map([(0, 1), (1, 1)]), + expires_at: Instant::now() + ttl, + revision: ShardLeaseRevision::of(1), + }) + }), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + + let hook_calls = Arc::new(AtomicUsize::new(0)); + let released = Arc::new(AtomicBool::new(false)); + let calls = hook_calls.clone(); + let hook_released = released.clone(); + let hook: ShardAssignmentChangedHook = Arc::new(move || { + let calls = calls.clone(); + let released = hook_released.clone(); + Box::pin(async move { + while !released.load(Ordering::SeqCst) { + tokio::time::sleep(Duration::from_millis(5)).await; + } + calls.fetch_add(1, Ordering::SeqCst); + Ok(RecoveryOutcome::Recovered) + }) + }); + service.set_assignment_changed_hook(&hook); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + // The first renewal widens the set, so its announcement enters the hook and blocks + // there - on its own task, not on the loop. + for _ in 0..200 { + if !mock.renew_calls().is_empty() { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + let renewals_before = mock.renew_calls().len(); + assert!( + renewals_before >= 1, + "the loop must have issued the first renewal" + ); + + // Several renewal periods pass (cadence is ~1s) while the hook stays blocked: the loop + // must keep renewing rather than waiting on the sweep. Each of these renewals also finds + // `recovery_outstanding` still set by the first + // one - the sweep has not reported back yet - and asks for the announcement again; that + // is correct (an unchanged grant re-runs a recovery still owed), and the single-flight + // coalesces every one of these requests into exactly one rerun, asserted below. + for _ in 0..100 { + if mock.renew_calls().len() >= renewals_before + 3 { + break; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + let renewals_after = mock.renew_calls().len(); + assert!( + renewals_after >= renewals_before + 3, + "further renewals must happen while the hook is blocked; before={renewals_before} \ + after={renewals_after}" + ); + assert_eq!( + hook_calls.load(Ordering::SeqCst), + 0, + "the hook must still be blocked at this point" + ); + + // Releasing it lets the sweep complete - and, deterministically, exactly one coalesced + // rerun right after: at least one of the renewals just observed above landed while the + // single-flight was already running, which is what put it in its "one more owed" state. + released.store(true, Ordering::SeqCst); + for _ in 0..200 { + if hook_calls.load(Ordering::SeqCst) >= 2 { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + assert_eq!( + hook_calls.load(Ordering::SeqCst), + 2, + "the blocked sweep and exactly one coalesced rerun, never one rerun per renewal" + ); + } + + #[test] + // The single-flight coordinator behind the fix above must not let a burst of requests that + // arrive while one announcement is already running turn into one run per request: exactly one + // more run is owed, coalescing the rest. Drives `AnnouncementSingleFlight` directly, bypassing + // the renewal loop entirely, since that is the unit actually being tested here. + async fn concurrent_announcement_requests_coalesce_into_one_more_run() { + let mock = Arc::new(MockShardManager::new()); + let (service, _shard_service) = make_service(mock, Shutdown::new()); + + let calls = Arc::new(AtomicUsize::new(0)); + let started = Arc::new(AtomicUsize::new(0)); + let released = Arc::new(AtomicBool::new(false)); + let hook_calls = calls.clone(); + let hook_started = started.clone(); + let hook_released = released.clone(); + let hook: ShardAssignmentChangedHook = Arc::new(move || { + let calls = hook_calls.clone(); + let started = hook_started.clone(); + let released = hook_released.clone(); + Box::pin(async move { + // Only the first call blocks; the coalesced rerun this test looks for must be + // observable without releasing a second time. + if started.fetch_add(1, Ordering::SeqCst) == 0 { + while !released.load(Ordering::SeqCst) { + tokio::time::sleep(Duration::from_millis(5)).await; + } + } + calls.fetch_add(1, Ordering::SeqCst); + Ok(RecoveryOutcome::Recovered) + }) + }); + service.set_assignment_changed_hook(&hook); + + AnnouncementSingleFlight::request(&service); + // Wait for the run to actually start (and block) before piling more requests onto it. + for _ in 0..200 { + if started.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert_eq!(started.load(Ordering::SeqCst), 1); + + // A burst of requests while the first run is still blocked. + for _ in 0..5 { + AnnouncementSingleFlight::request(&service); + } + + released.store(true, Ordering::SeqCst); + + for _ in 0..200 { + if calls.load(Ordering::SeqCst) >= 2 { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert_eq!( + calls.load(Ordering::SeqCst), + 2, + "a burst of requests while one run is in flight must coalesce into exactly one \ + more run" + ); + // Give a wrongly-spawned extra run a chance to show up before declaring victory. + tokio::time::sleep(Duration::from_millis(50)).await; + assert_eq!( + started.load(Ordering::SeqCst), + 2, + "exactly two runs total: the first, and the one coalesced burst" + ); + } + #[test] // The deregister has a deadline of its own, sized against the grace `main` waits rather than // against the lease, so a shard manager that accepts the call and never answers cannot hold @@ -1060,11 +1683,11 @@ mod tests { let gate = Arc::new(tokio::sync::Notify::new()); let mock = Arc::new( MockShardManager::new() - .with_renew(move |_, claimed| { + .with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) }) .with_deregister_gate(gate.clone()), @@ -1074,7 +1697,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(expiry), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let started = std::time::Instant::now(); @@ -1109,9 +1732,9 @@ mod tests { let expiry = Instant::now() + Duration::from_secs(300); let mock = Arc::new(MockShardManager::new().with_renew(move |_, _| { Ok(ShardLease { - shard_epochs: claim([(0, 1), (1, 1)]), + shard_epochs: epoch_map([(0, 1), (1, 1)]), expires_at: expiry, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) })); let (service, shard_service) = make_service(mock, Shutdown::new()); @@ -1119,7 +1742,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(expiry), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let calls = Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -1158,9 +1781,9 @@ mod tests { let expiry = Instant::now() + Duration::from_secs(300); let mock = Arc::new(MockShardManager::new().with_renew(move |_, _| { Ok(ShardLease { - shard_epochs: claim([(0, 1), (1, 1)]), + shard_epochs: epoch_map([(0, 1), (1, 1)]), expires_at: expiry, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) })); let (service, shard_service) = make_service(mock, Shutdown::new()); @@ -1168,7 +1791,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(expiry), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let calls = Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -1207,11 +1830,11 @@ mod tests { // revives the lease is what runs it - through the same latch a failed recovery uses. async fn a_deferred_recovery_runs_on_the_grant_that_revives_the_lease() { let expiry = Instant::now() + Duration::from_secs(300); - let mock = Arc::new(MockShardManager::new().with_renew(move |_, claimed| { + let mock = Arc::new(MockShardManager::new().with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: expiry, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) })); let (service, shard_service) = make_service(mock, Shutdown::new()); @@ -1220,7 +1843,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(Instant::now()), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); assert!(!shard_service.is_ready()); @@ -1267,11 +1890,11 @@ mod tests { let mock = Arc::new( MockShardManager::new() .with_register(move |_| Ok(registration(lapsed, [(0, 1), (1, 1)]))) - .with_renew(move |_, claimed| { + .with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: live, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) }), ); @@ -1321,11 +1944,11 @@ mod tests { // flag would; the ticket does not. async fn a_recovery_that_finishes_after_a_newer_deferral_leaves_that_one_owed() { let expiry = Instant::now() + Duration::from_secs(300); - let mock = Arc::new(MockShardManager::new().with_renew(move |_, claimed| { + let mock = Arc::new(MockShardManager::new().with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: expiry, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) })); let (service, shard_service) = make_service(mock, Shutdown::new()); @@ -1333,7 +1956,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(expiry), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let calls = Arc::new(AtomicUsize::new(0)); let hook_calls = calls.clone(); @@ -1369,9 +1992,9 @@ mod tests { let expiry = Instant::now() + Duration::from_secs(300); let mock = Arc::new(MockShardManager::new().with_renew(move |_, _| { Ok(ShardLease { - shard_epochs: claim([(0, 1), (1, 1)]), + shard_epochs: epoch_map([(0, 1), (1, 1)]), expires_at: expiry, - revision: ShardLeaseRevision(2), + revision: ShardLeaseRevision::of(2), }) })); let (service, shard_service) = make_service(mock, Shutdown::new()); @@ -1379,7 +2002,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(expiry), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let calls = Arc::new(AtomicUsize::new(0)); @@ -1429,13 +2052,13 @@ mod tests { /// Cross-track contract: `RenewShardLeaseRequest.shard_epochs` is exactly /// the set last received, and the granted expiry replaces the local one. #[test] - async fn a_renewal_claims_the_last_received_set_and_adopts_the_granted_expiry() { + async fn a_renewal_sends_the_last_received_set_and_adopts_the_granted_expiry() { let granted_expiry = Instant::now() + Duration::from_secs(300); - let mock = Arc::new(MockShardManager::new().with_renew(move |_, claimed| { + let mock = Arc::new(MockShardManager::new().with_renew(move |_, held| { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: granted_expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) })); let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); @@ -1443,7 +2066,7 @@ mod tests { SHARDS, &epochs([(0, 7), (3, 2)]), Some(Instant::now() + Duration::from_secs(10)), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let delay = service.renew_shard_lease().await; @@ -1452,8 +2075,8 @@ mod tests { assert_eq!(calls.len(), 1); assert_eq!( calls[0].1, - claim([(0, 7), (3, 2)]), - "the claim must be exactly the set last received, epochs included" + epoch_map([(0, 7), (3, 2)]), + "the held epochs must be exactly the set last received, epochs included" ); let assignment = shard_service.current_assignment().unwrap(); assert_eq!(assignment.expires_at, Some(granted_expiry)); @@ -1496,14 +2119,14 @@ mod tests { assignment.expires_at, assignment.revision, ); - let original_executor_id = mock.register_calls()[0]; + let original_executor_id = mock.register_calls()[0].0; service.renew_shard_lease().await; let register_calls = mock.register_calls(); assert_eq!(register_calls.len(), 2, "a lost lease must re-register"); assert_ne!( - register_calls[1], original_executor_id, + register_calls[1].0, original_executor_id, "the re-registration must come back as a new instance, under a fresh UUID" ); assert!( @@ -1515,7 +2138,405 @@ mod tests { assert_eq!(assignment.expires_at, Some(fresh_expiry)); } - /// A renewal that answers with shards this executor did not claim is the + /// A lost lease re-registers carrying the set it held, so a shard manager whose state was wiped + /// mints above the epochs this executor's oplog rows were written at. The set has to survive a + /// failed re-registration, which already cleared the assignment it was read from; a set that + /// arrives afterwards naming one of its shards lower must not lower it; and once a registration + /// succeeds the next loss carries only what that registration granted. + #[test] + async fn a_lost_lease_re_registers_carrying_the_shards_it_held() { + let expiry = Instant::now() + Duration::from_secs(120); + let attempts = Arc::new(AtomicUsize::new(0)); + let attempt = attempts.clone(); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| match attempt.fetch_add(1, Ordering::SeqCst) { + 0 => Ok(registration(expiry, [(2, 5)])), + 1 | 2 => Err(ShardManagerError::InternalServerError( + "shard manager down".to_string(), + )), + _ => Ok(registration(expiry, [(3, 1)])), + }) + .with_renew(|_, _| Err(ShardLeaseError::LeaseNotFound("unknown".to_string()))), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + service.renew_shard_lease().await; + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 2); + assert_eq!( + register_calls[0].1, + BTreeMap::new(), + "a first registration has nothing to carry" + ); + assert_eq!(register_calls[1].1, epoch_map([(2, 5)])); + + // The failed attempt left the assignment cleared, so this pass renews an empty set. + service.renew_shard_lease().await; + assert_eq!(mock.renew_calls()[1].1, BTreeMap::new()); + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 3); + assert_eq!( + register_calls[2].1, + epoch_map([(2, 5)]), + "the set held before the loss must outlive the failed re-registration" + ); + + // A delivery naming shard 2 below the carried epoch: the carried set keeps the higher one, + // because the oplog rows were written at it whatever the newer set says. + shard_service.register( + SHARDS, + &epochs([(2, 3), (4, 1)]), + Some(expiry), + ShardLeaseRevision::of(1), + ); + service.renew_shard_lease().await; + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 4); + assert_eq!(register_calls[3].1, epoch_map([(2, 5), (4, 1)])); + assert_eq!( + shard_service.current_assignment().unwrap().shard_epochs, + epochs([(3, 1)]) + ); + + service.renew_shard_lease().await; + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 5); + assert_eq!( + register_calls[4].1, + epoch_map([(3, 1)]), + "a successful registration retires the epochs it carried" + ); + } + + /// A shard manager that came back on a wiped store no longer lists this executor, so the + /// renewal is refused and the re-registration carries the epochs for it to repair. Its answer + /// is read off a store that counts from the beginning again, far below the last revision this + /// executor applied, and must be adopted all the same. + #[test] + async fn a_re_registration_with_a_manager_that_lost_its_history_adopts_its_low_revision() { + let expiry = Instant::now() + Duration::from_secs(120); + let (old_manager, new_manager) = (Uuid::new_v4(), Uuid::new_v4()); + let attempts = Arc::new(AtomicUsize::new(0)); + let attempt = attempts.clone(); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| match attempt.fetch_add(1, Ordering::SeqCst) { + 0 => Ok(registration_from(old_manager, 10_000, expiry, [(2, 5)])), + // The repaired grant: minted one past the carried epoch. + _ => Ok(registration_from(new_manager, 3, expiry, [(2, 6)])), + }) + .with_renew(|_, _| Err(ShardLeaseError::LeaseNotFound("unknown".to_string()))), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + service.renew_shard_lease().await; + + assert_eq!(mock.register_calls()[1].1, epoch_map([(2, 5)])); + let assignment = shard_service.current_assignment().unwrap(); + assert_eq!( + assignment.shard_epochs, + epochs([(2, 6)]), + "the repaired grant was dropped as older than the last delivery applied" + ); + assert_eq!(assignment.revision, revision_of(new_manager, 3)); + } + + /// A shard manager restored from a backup still lists this executor, so it grants the renewal + /// - from a store whose revisions are far below the last one this executor applied. The + /// manager it replaced must not get a delivery in afterwards on the strength of its higher + /// revisions. + #[test] + async fn a_renewal_answered_by_a_manager_restored_from_a_backup_adopts_its_low_revision() { + let expiry = Instant::now() + Duration::from_secs(120); + let (old_manager, new_manager) = (Uuid::new_v4(), Uuid::new_v4()); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| { + Ok(registration_from(old_manager, 10_000, expiry, [(2, 5)])) + }) + .with_renew(move |_, _| { + Ok(ShardLease { + shard_epochs: epoch_map([(2, 6)]), + expires_at: expiry, + revision: revision_of(new_manager, 3), + }) + }), + ); + let (service, shard_service) = make_service(mock, Shutdown::new()); + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + service.renew_shard_lease().await; + + let assignment = shard_service.current_assignment().unwrap(); + assert_eq!( + assignment.shard_epochs, + epochs([(2, 6)]), + "the repaired grant was dropped as older than the last delivery applied" + ); + assert_eq!(assignment.revision, revision_of(new_manager, 3)); + + let delayed = shard_service + .assign_shards(SHARDS, &epochs([(2, 5)]), revision_of(old_manager, 10_001)) + .unwrap(); + assert!( + matches!(delayed, ShardDeliveryOutcome::FromAnotherManager { .. }), + "got {delayed:?}" + ); + assert_eq!( + shard_service.current_assignment().unwrap().shard_epochs, + epochs([(2, 6)]) + ); + } + + /// A push from a shard manager process this executor does not follow is ignored, and what it + /// owes instead is a renewal now rather than a third of a lease from now: the answer names + /// the process in charge and carries its set. + #[test] + async fn renew_now_runs_a_renewal_ahead_of_its_timer() { + let expiry = Instant::now() + Duration::from_secs(120); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration(expiry, [(0, 1)]))) + .with_renew(move |_, held| { + Ok(ShardLease { + shard_epochs: held, + expires_at: expiry, + revision: ShardLeaseRevision::of(1), + }) + }), + ); + let shutdown = Shutdown::new(); + let (service, shard_service) = make_service(mock.clone(), shutdown.clone()); + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + assert!( + mock.renew_calls().is_empty(), + "the timer is a third of the lease away" + ); + + service.renew_now(); + + for _ in 0..100 { + if !mock.renew_calls().is_empty() { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert_eq!(mock.renew_calls().len(), 1); + shutdown.cancel(); + } + + /// A re-registration can be stored by the manager and still fail here, when its reply is lost. + /// The next renewal under the fresh id is then granted, which is the manager saying it knows + /// this executor again: the carried epochs are dropped there, so the next loss does not send + /// epochs from before the last one. + #[test] + async fn a_renewal_granted_after_a_failed_re_registration_drops_the_carried_epochs() { + let expiry = Instant::now() + Duration::from_secs(120); + let registrations = Arc::new(AtomicUsize::new(0)); + let registration_attempt = registrations.clone(); + let renewals = Arc::new(AtomicUsize::new(0)); + let renewal_attempt = renewals.clone(); + let mock = Arc::new( + MockShardManager::new() + .with_register( + move |_| match registration_attempt.fetch_add(1, Ordering::SeqCst) { + 0 => Ok(registration(expiry, [(2, 5)])), + 1 => Err(ShardManagerError::InternalServerError( + "reply lost".to_string(), + )), + _ => Ok(registration(expiry, [(3, 1)])), + }, + ) + .with_renew( + move |_, _| match renewal_attempt.fetch_add(1, Ordering::SeqCst) { + 1 => Ok(ShardLease { + shard_epochs: epoch_map([(3, 1)]), + expires_at: expiry, + revision: ShardLeaseRevision::of(1), + }), + _ => Err(ShardLeaseError::LeaseNotFound("unknown".to_string())), + }, + ), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + + service.renew_shard_lease().await; + assert_eq!(mock.register_calls()[1].1, epoch_map([(2, 5)])); + + service.renew_shard_lease().await; + assert_eq!( + shard_service.current_assignment().unwrap().shard_epochs, + epochs([(3, 1)]) + ); + + service.renew_shard_lease().await; + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 3); + assert_eq!( + register_calls[2].1, + epoch_map([(3, 1)]), + "the granted renewal retired the epochs carried from the earlier loss" + ); + } + + /// The epoch a fenced write found, on the shard `agent_on_shard(shard)` routes to. + fn fence_on_shard(shard: i64, expected: u64, stored: u64) -> OplogFence { + OplogFence { + agent_id: agent_on_shard(shard), + expected_epoch: ShardEpoch(expected), + actual_epoch: Some(ShardEpoch(stored)), + writer_conflict: false, + } + } + + /// Epochs learned from fenced oplog writes ride on the next renewal beside the held ones, so a + /// shard manager whose state lost history can mint above them. A granted renewal has stored + /// whatever they moved, so it retires them and the next renewal reports nothing. + #[test] + async fn a_renewal_reports_fence_learned_epochs_and_retires_them_once_granted() { + let expiry = Instant::now() + Duration::from_secs(120); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration(expiry, [(2, 5)]))) + .with_renew(move |_, held| { + Ok(ShardLease { + shard_epochs: held, + expires_at: expiry, + revision: ShardLeaseRevision::of(1), + }) + }), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + shard_service.fenced(&fence_on_shard(2, 5, 7)); + + service.renew_shard_lease().await; + assert_eq!(mock.renew_calls()[0].1, epoch_map([(2, 5)])); + assert_eq!(mock.renew_fenced_calls()[0], epoch_map([(2, 7)])); + assert_eq!( + shard_service.fence_learned_epochs(), + BTreeMap::new(), + "a granted renewal retires the epochs it reported" + ); + + service.renew_shard_lease().await; + assert_eq!( + mock.renew_fenced_calls()[1], + BTreeMap::new(), + "an epoch the manager already stored was reported again" + ); + } + + /// Nothing but a grant says the manager stored a report. A renewal lost in transport, and one + /// refused as a lease not found, keep the learned epochs for the next pass. The + /// re-registration after the refusal carries only the held epochs, because a registration + /// applies what it carries as this executor's own; the learned epochs go on the first renewal + /// under the fresh id. + #[test] + async fn a_refused_renewal_keeps_fence_learned_epochs_for_the_next_one() { + let expiry = Instant::now() + Duration::from_secs(120); + let renewals = Arc::new(AtomicUsize::new(0)); + let renewal_attempt = renewals.clone(); + let mock = Arc::new( + MockShardManager::new() + .with_register(move |_| Ok(registration(expiry, [(2, 5)]))) + .with_renew( + move |_, held| match renewal_attempt.fetch_add(1, Ordering::SeqCst) { + 0 => Err(ShardLeaseError::InternalServerError( + "shard manager down".to_string(), + )), + 1 => Err(ShardLeaseError::LeaseNotFound("unknown".to_string())), + _ => Ok(ShardLease { + shard_epochs: held, + expires_at: expiry, + revision: ShardLeaseRevision::of(1), + }), + }, + ), + ); + let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); + + let assignment = service.register(PORT, None).await.unwrap(); + shard_service.register( + assignment.number_of_shards, + &assignment.shard_epochs, + assignment.expires_at, + assignment.revision, + ); + shard_service.fenced(&fence_on_shard(2, 5, 7)); + + service.renew_shard_lease().await; + assert_eq!(mock.renew_fenced_calls()[0], epoch_map([(2, 7)])); + assert_eq!( + shard_service.fence_learned_epochs(), + epoch_map([(2, 7)]), + "a renewal lost in transport retired what it never delivered" + ); + + service.renew_shard_lease().await; + assert_eq!(mock.renew_fenced_calls()[1], epoch_map([(2, 7)])); + let register_calls = mock.register_calls(); + assert_eq!(register_calls.len(), 2, "a lost lease must re-register"); + assert_eq!( + register_calls[1].1, + epoch_map([(2, 5)]), + "the re-registration carries the held epochs, never a fenced one" + ); + assert_eq!( + shard_service.fence_learned_epochs(), + epoch_map([(2, 7)]), + "a refused renewal retired what the manager never stored" + ); + + service.renew_shard_lease().await; + assert_eq!(mock.renew_fenced_calls()[2], epoch_map([(2, 7)])); + assert_eq!(shard_service.fence_learned_epochs(), BTreeMap::new()); + } + + /// A renewal that answers with shards this executor did not send is the /// shard manager correcting a push that never arrived, so it has to recover /// agents for them exactly as a push would. The common path — the same set /// back, because a renewal never advances an epoch — must NOT fire the @@ -1524,16 +2545,16 @@ mod tests { #[test] async fn a_renewal_that_changes_the_set_announces_it_and_an_unchanged_one_does_not() { let granted_expiry = Instant::now() + Duration::from_secs(300); - // `None` echoes the claim; `Some` is the manager correcting it. + // `None` echoes the held set; `Some` is the manager correcting it. let correction: Arc>>> = Arc::new(StdMutex::new(None)); let correct = correction.clone(); - let mock = Arc::new(MockShardManager::new().with_renew(move |_, claimed| { - let shard_epochs = correct.lock().unwrap().clone().unwrap_or(claimed); + let mock = Arc::new(MockShardManager::new().with_renew(move |_, held| { + let shard_epochs = correct.lock().unwrap().clone().unwrap_or(held); Ok(ShardLease { shard_epochs, expires_at: granted_expiry, - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) })); let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); @@ -1541,7 +2562,7 @@ mod tests { SHARDS, &epochs([(0, 7), (3, 2)]), Some(Instant::now() + Duration::from_secs(30)), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let announced = Arc::new(AtomicUsize::new(0)); @@ -1563,7 +2584,7 @@ mod tests { ); // wider: a shard this executor never knew it owned - *correction.lock().unwrap() = Some(claim([(0, 7), (3, 2), (4, 9)])); + *correction.lock().unwrap() = Some(epoch_map([(0, 7), (3, 2), (4, 9)])); service.renew_shard_lease().await; assert_eq!( announced.load(Ordering::SeqCst), @@ -1576,7 +2597,7 @@ mod tests { ); // narrower: a shard the manager has given to someone else - *correction.lock().unwrap() = Some(claim([(0, 7), (4, 9)])); + *correction.lock().unwrap() = Some(epoch_map([(0, 7), (4, 9)])); service.renew_shard_lease().await; assert_eq!( announced.load(Ordering::SeqCst), @@ -1600,9 +2621,9 @@ mod tests { let granted_expiry = Instant::now() + Duration::from_secs(300); let mock = Arc::new(MockShardManager::new().with_renew(move |_, _| { Ok(ShardLease { - shard_epochs: claim([(0, 7)]), + shard_epochs: epoch_map([(0, 7)]), expires_at: granted_expiry, - revision: ShardLeaseRevision(4), + revision: ShardLeaseRevision::of(4), }) })); let (service, shard_service) = make_service(mock.clone(), Shutdown::new()); @@ -1611,7 +2632,7 @@ mod tests { SHARDS, &epochs([(0, 7), (4, 9)]), Some(Instant::now() + Duration::from_secs(30)), - ShardLeaseRevision(5), + ShardLeaseRevision::of(5), ); let announced = Arc::new(AtomicBool::new(false)); @@ -1638,7 +2659,7 @@ mod tests { Some(granted_expiry), "the set is stale; the lease is not, and it answers this executor's own request" ); - assert_eq!(assignment.revision, ShardLeaseRevision(5)); + assert_eq!(assignment.revision, ShardLeaseRevision::of(5)); assert!( !announced.load(Ordering::SeqCst), "an ignored delivery is not an assignment change" @@ -1667,7 +2688,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(Instant::now() + Duration::from_secs(60)), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); service.renew_shard_lease().await; @@ -1703,7 +2724,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(Instant::now() + Duration::from_secs(1)), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let first = service.renew_shard_lease().await; @@ -1742,13 +2763,13 @@ mod tests { let calls = Arc::new(AtomicUsize::new(0)); let grant_at = grant_at_call.clone(); let seen = calls.clone(); - let mock = Arc::new(MockShardManager::new().with_renew(move |_, claimed| { + let mock = Arc::new(MockShardManager::new().with_renew(move |_, held| { let call = seen.fetch_add(1, Ordering::SeqCst) + 1; if call == grant_at.load(Ordering::SeqCst) { Ok(ShardLease { - shard_epochs: claimed, + shard_epochs: held, expires_at: Instant::now() + Duration::from_secs(30), - revision: ShardLeaseRevision(1), + revision: ShardLeaseRevision::of(1), }) } else { Err(ShardLeaseError::InternalServerError("down".to_string())) @@ -1759,7 +2780,7 @@ mod tests { SHARDS, &epochs([(0, 1)]), Some(Instant::now() + Duration::from_secs(300)), - ShardLeaseRevision(1), + ShardLeaseRevision::of(1), ); let mut delays = Vec::new(); diff --git a/golem-worker-executor/src/services/worker.rs b/golem-worker-executor/src/services/worker.rs index 4b610bb8d6..dcd105c494 100644 --- a/golem-worker-executor/src/services/worker.rs +++ b/golem-worker-executor/src/services/worker.rs @@ -21,7 +21,7 @@ use crate::metrics::workers::{ record_agent_identity_resolution, record_derived_cache_publication_failed, record_stale_running_worker, record_status_cache_publication, record_worker_call, }; -use crate::services::oplog::{OplogLifecycleGuard, OplogService}; +use crate::services::oplog::{OplogError, OplogLifecycleGuard, OplogService}; use crate::services::shard::ShardService; use crate::services::stream_session_index::StreamSessionIndexService; use crate::storage::keyvalue::{ @@ -38,7 +38,7 @@ use golem_common::model::{ AgentFingerprint, AgentId, AgentMetadata, AgentStatus, AgentStatusRecord, DurableStreamPublicBinding, DurableStreamSessionStatus, FailedUpdateRecord, IdempotencyKey, InvocationResultMembership, OwnedAgentId, ReceivedCardTransferIndex, ReceivedCardTransferState, - ShardId, SuccessfulUpdateRecord, + ShardEpoch, ShardId, SuccessfulUpdateRecord, }; use golem_common::serialization::{deserialize, serialize, try_deserialize}; use golem_service_base::error::worker_executor::WorkerExecutorError; @@ -371,12 +371,18 @@ pub trait WorkerService: Send + Sync { /// /// Returns `Err` when the storage could not be reached. Delete is not retried by the caller: /// a retry would re-run the oplog delete, so the error is reported instead. + /// + /// `expected_epoch` is the epoch the caller's oplog handle asserts. The oplog is deleted only + /// while this executor still holds it at that epoch; otherwise nothing at all is removed and + /// the result is [`WorkerExecutorError::OplogFenced`], because the agent's state belongs to + /// the shard's new owner. async fn remove( &self, lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, fingerprint: AgentFingerprint, + expected_epoch: Option, ) -> Result<(), WorkerExecutorError>; /// Deletes every cached status blob for the worker (live cache, clean checkpoint, the legacy @@ -1648,6 +1654,7 @@ impl WorkerService for DefaultWorkerService { owned_agent_id: &OwnedAgentId, agent_mode: AgentMode, fingerprint: AgentFingerprint, + expected_epoch: Option, ) -> Result<(), WorkerExecutorError> { lifecycle.assert_agent(&owned_agent_id.agent_id); let lifecycle_gate = self.lifecycle_gate(owned_agent_id); @@ -1670,6 +1677,22 @@ impl WorkerService for DefaultWorkerService { None => false, }; + // The oplog first, so that a refusal leaves every other piece of the agent's state in place + // too. Only this incarnation's: a recreated agent's oplog is not the deleting one's to remove. + if delete_current_oplog { + self.oplog_service + .delete(lifecycle, owned_agent_id, agent_mode, expected_epoch) + .await + .map_err(|error| match error { + OplogError::Fenced(fence) => WorkerExecutorError::oplog_fenced( + fence.agent_id, + fence.expected_epoch.0, + fence.actual_epoch.map(|epoch| epoch.0), + ), + other => WorkerExecutorError::runtime(other.to_string()), + })?; + } + self.remove_cached_status(owned_agent_id, fingerprint) .await?; self.remove_all_fields( @@ -1691,12 +1714,6 @@ impl WorkerService for DefaultWorkerService { .await .map_err(WorkerExecutorError::runtime)?; - if delete_current_oplog { - self.oplog_service - .delete(lifecycle, owned_agent_id, agent_mode) - .await; - } - let shard_assignment = self .shard_service .current_assignment() @@ -2377,7 +2394,7 @@ mod tests { use golem_common::model::regions::{DeletedRegions, OplogRegion}; use golem_common::model::{ AgentInvocationPayload, AgentInvocationResult, AgentMetadata, PendingInvocationRef, - PendingUpdateKind, PendingUpdateRef, ScanCursor, ShardLeaseRevision, + PendingUpdateKind, PendingUpdateRef, ScanCursor, ShardEpoch, ShardLeaseRevision, }; use golem_common::read_only_lock; use golem_service_base::model::component::Component; @@ -2459,6 +2476,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2472,6 +2490,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2485,6 +2504,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2506,7 +2526,8 @@ mod tests { _lifecycle: &mut OplogLifecycleGuard, _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, - ) { + _expected_epoch: Option, + ) -> Result<(), crate::services::oplog::OplogError> { unreachable!() } @@ -4061,6 +4082,7 @@ mod tests { &owned_agent_id, AgentMode::Durable, first, + None, ) .await .unwrap(); @@ -4266,6 +4288,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -4279,6 +4302,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -4292,6 +4316,7 @@ mod tests { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -4309,7 +4334,8 @@ mod tests { _lifecycle: &mut OplogLifecycleGuard, _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, - ) { + _expected_epoch: Option, + ) -> Result<(), crate::services::oplog::OplogError> { unreachable!() } @@ -4641,6 +4667,7 @@ mod tests { &owned_agent_id, AgentMode::Durable, AgentFingerprint(Uuid::new_v4()), + None, ) .await .is_err(), diff --git a/golem-worker-executor/src/services/worker/session_index_tests.rs b/golem-worker-executor/src/services/worker/session_index_tests.rs index b5adbd51cc..b15bf6b833 100644 --- a/golem-worker-executor/src/services/worker/session_index_tests.rs +++ b/golem-worker-executor/src/services/worker/session_index_tests.rs @@ -178,7 +178,7 @@ async fn cancellation_receipts_prune_recovery_catalogue_after_cold_reopen() { StreamSessionRecord::ConsumerCancelIntent(intent.clone()), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( service .lookup_durable_stream_recovery_metadata( @@ -204,7 +204,7 @@ async fn cancellation_receipts_prune_recovery_catalogue_after_cold_reopen() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( service .lookup_durable_stream_recovery_metadata( @@ -228,7 +228,7 @@ async fn cancellation_receipts_prune_recovery_catalogue_after_cold_reopen() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert!( service .lookup_durable_stream_recovery_metadata( @@ -304,6 +304,7 @@ async fn committed_cancellation_probe_preserves_exact_authority_after_takeover() metadata, stale_status(), suspended_status(), + None, ) .await; let mapping = StreamSessionMappingRecord { @@ -373,7 +374,7 @@ async fn committed_cancellation_probe_preserves_exact_authority_after_takeover() reason: StreamCancelReason::Cancelled, details: Some("committed external cancellation".into()), }; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let probe = DbDirectStreamAttachmentConsumerProbe::new(Arc::new(service), oplog_service.clone()); assert_eq!( @@ -412,7 +413,7 @@ async fn committed_cancellation_probe_preserves_exact_authority_after_takeover() }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( probe .status_exact(&attachment, Some(&mapping)) @@ -438,6 +439,7 @@ async fn committed_cancellation_probe_preserves_exact_authority_after_takeover() foreign_metadata, stale_status(), suspended_status(), + None, ) .await; let foreign_attachment = StreamAttachmentKey { @@ -463,7 +465,7 @@ async fn committed_cancellation_probe_preserves_exact_authority_after_takeover() ] { append_session(foreign_oplog.as_ref(), record).await; } - foreign_oplog.commit(CommitLevel::Always).await; + foreign_oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( probe .status_exact(&foreign_attachment, Some(&mapping)) @@ -733,6 +735,7 @@ async fn create_oplog_with_identity( metadata, stale_status(), suspended_status(), + None, ) .await } @@ -741,6 +744,7 @@ async fn append_session(oplog: &dyn Oplog, record: StreamSessionRecord) -> Oplog oplog .add(DurableStreamOplogRecord::Session(None, Box::new(record)).into_inline_entry()) .await + .expect("oplog write") } async fn append_noop(oplog: &dyn Oplog) -> OplogIndex { @@ -750,6 +754,7 @@ async fn append_noop(oplog: &dyn Oplog) -> OplogIndex { entity_parent_start_index: None, }) .await + .expect("oplog write") } #[test] @@ -792,7 +797,7 @@ async fn quiescent_recovery_caches_only_read_new_committed_suffixes() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let mut cache = crate::worker::DurableTopologyRecoveryCache::default(); cache .refresh_through( @@ -844,7 +849,7 @@ async fn quiescent_recovery_caches_only_read_new_committed_suffixes() { cache.dirty.is_empty(), "buffered work is not published demand" ); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); cache .refresh_through( suffix, @@ -888,7 +893,7 @@ async fn recovery_cache_refolds_a_cut_committed_after_its_snapshot() { let fingerprint = session_key(&owner, &key).callee_fingerprint; let oplog = create_oplog(oplog_service.as_ref(), &owner).await; append_session(oplog.as_ref(), prepared_record(&owner, &key)).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let mut cache = crate::worker::DurableTopologyRecoveryCache::default(); cache .refresh( @@ -922,7 +927,8 @@ async fn recovery_cache_refolds_a_cut_committed_after_its_snapshot() { .into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; + .await + .unwrap(); // An uncommitted cut must fail closed, rather than repeatedly reloading an old index. assert!( cache @@ -936,7 +942,7 @@ async fn recovery_cache_refolds_a_cut_committed_after_its_snapshot() { .await .is_err() ); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); cache .refresh( oplog.as_ref(), @@ -986,13 +992,14 @@ async fn reverted_session_is_absent_from_warm_and_cold_indexes_across_empty_chun metadata, stale_status(), suspended_status(), + None, ) .await; append_session(oplog.as_ref(), prepared_record(&id, &key)).await; for _ in 0..2050 { append_noop(oplog.as_ref()).await; } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let end = oplog.current_oplog_index().await; if warm { assert!( @@ -1023,8 +1030,9 @@ async fn reverted_session_is_absent_from_warm_and_cold_indexes_across_empty_chun DurableStreamOplogRecord::Session(None, Box::new(cut)).into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker_entry)) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let marker = oplog.current_oplog_index().await; assert!( service @@ -1046,7 +1054,7 @@ async fn reverted_session_is_absent_from_warm_and_cold_indexes_across_empty_chun // The same invocation key can be accepted again after its old preparation is deleted. let prepared = append_session(oplog.as_ref(), prepared_record(&id, &key)).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let state = service .stream_session_index .lookup_persisted(&id, AgentMode::Durable, fingerprint, prepared, &key) @@ -1093,6 +1101,7 @@ async fn self_revert_retains_foreign_consumer_prefix_across_partial_page_and_rep metadata, stale_status(), suspended_status(), + None, ) .await; append_session( @@ -1154,7 +1163,7 @@ async fn self_revert_retains_foreign_consumer_prefix_across_partial_page_and_rep .await, ); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let region = OplogRegion { start: removed[0], @@ -1190,8 +1199,9 @@ async fn self_revert_retains_foreign_consumer_prefix_across_partial_page_and_rep let marker = DurableStreamOplogRecord::Session(None, Box::new(cut)).into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let expected_second_page = retained[256..].to_vec(); service @@ -1238,7 +1248,7 @@ async fn self_revert_retains_foreign_consumer_prefix_across_partial_page_and_rep }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); cold.stream_session_index .catch_up( &owner, @@ -1291,6 +1301,7 @@ async fn concurrent_index_services_refold_same_paired_revert_without_stale_rows( metadata, stale_status(), suspended_status(), + None, ) .await; append_session(oplog.as_ref(), prepared_with_reader(&owner, &key)).await; @@ -1314,7 +1325,7 @@ async fn concurrent_index_services_refold_same_paired_revert_without_stale_rows( .await, ); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let horizon = oplog.current_oplog_index().await; let (warm_first, warm_second) = tokio::join!( first.stream_session_index.lookup_persisted( @@ -1357,7 +1368,7 @@ async fn concurrent_index_services_refold_same_paired_revert_without_stale_rows( start: removed[0], end: *removed.last().unwrap(), }; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); first .stream_session_index .catch_up(&owner, AgentMode::Durable, fingerprint, region.end) @@ -1388,8 +1399,9 @@ async fn concurrent_index_services_refold_same_paired_revert_without_stale_rows( let marker = DurableStreamOplogRecord::Session(None, Box::new(cut)).into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); let expected = retained[256..].to_vec(); let (left_control, right_control) = tokio::join!( first.lookup_durable_stream_control_metadata( @@ -1444,6 +1456,7 @@ async fn append_pending_invocation(oplog: &dyn Oplog, key: &IdempotencyKey) -> O Vec::new(), )) .await + .expect("oplog write") } fn attached_record( @@ -1530,7 +1543,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); storage.reset(); let metadata = service .lookup_durable_stream_control_metadata(&id, AgentMode::Durable, test_fingerprint(), &key) @@ -1597,7 +1613,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit .finished_position() .is_none() ); - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); storage.reset(); let metadata = reopened .lookup_durable_stream_control_metadata(&id, AgentMode::Durable, test_fingerprint(), &key) @@ -1641,7 +1660,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit } } } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let status = AgentStatusRecord { oplog_idx: oplog.current_oplog_index().await, has_durable_stream_history: true, @@ -1704,7 +1726,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit active.push(prepared.attempt.session_key.clone()); append_session(oplog.as_ref(), record).await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let recovery = reopened .lookup_durable_stream_recovery_metadata(&id, AgentMode::Durable, test_fingerprint()) .await @@ -1769,7 +1794,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit "raw preparation must be visible before commit" ); assert_eq!(cache.dirty, HashSet::from([raw_key])); - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); storage.reset(); cache .refresh( @@ -1799,7 +1827,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit ) .await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let recovery = reopened .lookup_durable_stream_recovery_metadata(&id, AgentMode::Durable, test_fingerprint()) .await @@ -1846,7 +1877,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit ) .await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert!( restarted .lookup_durable_stream_recovery_metadata(&id, AgentMode::Durable, test_fingerprint()) @@ -1890,7 +1924,10 @@ async fn persisted_control_projection_reopens_without_history_and_catches_commit .await; historical.push((attempt, offset)); } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!( restarted @@ -2023,7 +2060,10 @@ async fn closed_remote_consumer_streams_leave_recovery_across_epochs() { mappings.push(mapping); attachments.push(attachment); } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let mut cache = crate::worker::DurableTopologyRecoveryCache::default(); cache .refresh( @@ -2099,7 +2139,10 @@ async fn closed_remote_consumer_streams_leave_recovery_across_epochs() { cache.sessions.is_empty(), "raw closure must retire resident recovery work" ); - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert!( service .lookup_durable_stream_recovery_metadata(&owner, AgentMode::Durable, test_fingerprint()) @@ -2137,7 +2180,10 @@ async fn closed_remote_consumer_streams_leave_recovery_across_epochs() { ) .await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); cache .refresh( oplog.as_ref(), @@ -2264,7 +2310,7 @@ async fn public_binding_projection_is_fenced_and_rebuildable() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let expected = DurableStreamPublicBinding::Live { session_key: second.clone(), @@ -2347,7 +2393,7 @@ async fn target_only_public_binding_status_survives_transitions_and_rebuild() { let index = append_session(oplog.as_ref(), record.clone()).await; indexed_records.push((index, record)); } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert!(matches!( service @@ -2384,7 +2430,7 @@ async fn target_only_public_binding_status_survives_transitions_and_rebuild() { }); let cut_index = append_session(oplog.as_ref(), cut.clone()).await; indexed_records.push((cut_index, cut)); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let inherited = service .stream_session_index @@ -2402,7 +2448,7 @@ async fn target_only_public_binding_status_survives_transitions_and_rebuild() { let replacement = initialize(40_000); let replacement_index = append_session(oplog.as_ref(), replacement.clone()).await; indexed_records.push((replacement_index, replacement)); - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let expected = DurableStreamPublicBinding::Live { session_key: key.clone(), @@ -2475,7 +2521,7 @@ async fn fork_cut_clears_public_bindings_after_partial_catch_up() { record.expiry_policy = StreamSessionExpiryPolicy::Sliding { ttl_seconds: 30 }; record.expiry_deadline_millis = Some(40_000); append_session(oplog.as_ref(), prepared).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert!(matches!( service @@ -2499,7 +2545,7 @@ async fn fork_cut_clears_public_bindings_after_partial_catch_up() { retained_through: None, }); append_session(oplog.as_ref(), cut).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( service @@ -2581,7 +2627,7 @@ async fn self_revert_preserves_retained_public_binding_in_warm_and_cold_indexes( record.expiry_deadline_millis = Some(40_000); append_session(oplog.as_ref(), prepared).await; let discarded = append_noop(oplog.as_ref()).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let expected = DurableStreamPublicBinding::Live { session_key: key.clone(), @@ -2618,8 +2664,9 @@ async fn self_revert_preserves_retained_public_binding_in_warm_and_cold_indexes( .into_inline_entry(); oplog .add_pair(OplogEntry::revert(region), Box::new(move |_| marker)) - .await; - oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + oplog.commit(CommitLevel::Always).await.unwrap(); assert_eq!( service @@ -2878,7 +2925,7 @@ async fn stale_mode_hint_accepts_a_same_mode_recreated_fingerprint() { let first_fingerprint = AgentFingerprint(uuid::Uuid::from_u128(101)); let second_fingerprint = AgentFingerprint(uuid::Uuid::from_u128(102)); let first = create_oplog_with_fingerprint(oplog_service.as_ref(), &id, first_fingerprint).await; - first.commit(CommitLevel::Always).await; + first.commit(CommitLevel::Always).await.unwrap(); drop(first); assert_eq!( service @@ -2895,11 +2942,13 @@ async fn stale_mode_hint_accepts_a_same_mode_recreated_fingerprint() { &mut oplog_service.lock_lifecycle(&id.agent_id).await, &id, AgentMode::Durable, + None, ) - .await; + .await + .unwrap(); let second = create_oplog_with_fingerprint(oplog_service.as_ref(), &id, second_fingerprint).await; - second.commit(CommitLevel::Always).await; + second.commit(CommitLevel::Always).await.unwrap(); drop(second); let identity = service.resolve_agent_identity(&id).await.unwrap().unwrap(); @@ -2998,7 +3047,7 @@ async fn warm_durable_identity_resolution_reads_only_the_hinted_create() { let id = owned_agent("warm-identity", ComponentId::new()); let fingerprint = AgentFingerprint(uuid::Uuid::from_u128(301)); let oplog = create_oplog_with_fingerprint(oplog_service.as_ref(), &id, fingerprint).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); service.resolve_agent_identity(&id).await.unwrap().unwrap(); let reads = indexed.read_count(); let writes = faults.calls("write_cached_agent_mode"); @@ -3056,9 +3105,10 @@ async fn identity_resolution_distinguishes_absent_and_malformed_oplogs() { metadata, stale_status(), suspended_status(), + None, ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); assert!(service.resolve_agent_identity(&malformed).await.is_err()); } @@ -3080,7 +3130,7 @@ async fn assert_raw_cache_owner_fingerprint( value.attempt.invocation.session_key.callee_fingerprint = callee_fingerprint; let foreign_key = value.attempt.session_key.clone(); let prepared_index = append_session(oplog.as_ref(), prepared).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let raw = oplog.raw_durable_stream_session_status(&foreign_key).await; assert_eq!( raw.status.unwrap().unwrap().first_prepared, @@ -3114,7 +3164,7 @@ async fn delayed_stream_projection_cannot_overwrite_a_rebuilt_generation() { let oplog = create_oplog(oplog_service.as_ref(), &id).await; let key = IdempotencyKey::new("stream".into()); append_session(oplog.as_ref(), prepared_record(&id, &key)).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let rebuilt_horizon = oplog.current_oplog_index().await; let faults = KeyValueStorageFaults::default(); @@ -3133,7 +3183,7 @@ async fn delayed_stream_projection_cannot_overwrite_a_rebuilt_generation() { prepared_record(&id, &IdempotencyKey::new("concurrent-field".into())), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let final_horizon = oplog.current_oplog_index().await; let gate = faults.gate_next_pass("advance"); let delayed_task = tokio::spawn({ @@ -3179,7 +3229,7 @@ async fn stream_projection_clear_retries_when_a_writer_advances_its_snapshot() { let oplog = create_oplog(oplog_service.as_ref(), &id).await; let key = IdempotencyKey::new("stream".into()); append_session(oplog.as_ref(), prepared_record(&id, &key)).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let faults = KeyValueStorageFaults::default(); let faulting: Arc = Arc::new( @@ -3194,7 +3244,7 @@ async fn stream_projection_clear_retries_when_a_writer_advances_its_snapshot() { .unwrap(); let concurrent_key = IdempotencyKey::new("concurrent-field".into()); append_session(oplog.as_ref(), prepared_record(&id, &concurrent_key)).await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let advanced_horizon = oplog.current_oplog_index().await; let gate = faults.gate_next_pass("clear"); @@ -3274,7 +3324,7 @@ async fn resume_lookup_rebuilds_when_projection_disappears_after_catch_up() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); let faults = KeyValueStorageFaults::default(); let faulting: Arc = Arc::new( @@ -3349,7 +3399,10 @@ async fn catchup_scans_multiple_chunks_and_recovers_evicted_completed_session() .await; bounded.insert(key, completed(prepared.as_u64(), finished.as_u64())); } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let horizon = oplog.current_oplog_index().await; assert!(horizon.as_u64() > 1024); assert!(bounded.iter().count() <= 128); @@ -3391,7 +3444,10 @@ async fn incremental_catchup_merges_later_fields_into_old_unfinished_session() { ) .await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let first_horizon = oplog.current_oplog_index().await; let status_at_first_horizon = AgentStatusRecord { oplog_idx: first_horizon, @@ -3440,7 +3496,10 @@ async fn incremental_catchup_merges_later_fields_into_old_unfinished_session() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let mut later_status = status_at_first_horizon; later_status.oplog_idx = finished; @@ -3579,7 +3638,10 @@ async fn raw_attachment_authority_fences_before_commit_and_survives_buffer_drain }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let producer = DurableStreamStore::load( oplog.clone(), @@ -3635,7 +3697,10 @@ async fn raw_attachment_authority_fences_before_commit_and_survives_buffer_drain // No Worker/status actor participates: draining the buffer must not reset raw authority // back to the older published status used when this oplog was constructed. - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!( oplog_service.get_last_index(&id, AgentMode::Durable).await, resumed @@ -3691,7 +3756,10 @@ async fn raw_cold_reopen_ignores_stale_supplied_status_and_recovers_committed_re ), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let resumed_attempt = AttemptId::fresh(); let resumed = append_session( @@ -3715,7 +3783,10 @@ async fn raw_cold_reopen_ignores_stale_supplied_status_and_recovers_committed_re }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); drop(oplog); let reopened = oplog_service @@ -3727,6 +3798,7 @@ async fn raw_cold_reopen_ignores_stale_supplied_status_and_recovers_committed_re agent_metadata(&id), stale_status(), suspended_status(), + None, ) .await; let raw = reopened @@ -3765,7 +3837,10 @@ async fn raw_cached_lookup_observes_takeover_committed_by_another_oplog_actor() ), ) .await; - first.commit(CommitLevel::Always).await; + first + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!( first .raw_durable_stream_session_status(&session_key) @@ -3786,6 +3861,7 @@ async fn raw_cached_lookup_observes_takeover_committed_by_another_oplog_actor() agent_metadata(&id), stale_status(), suspended_status(), + None, ) .await; let takeover_attempt = AttemptId::fresh(); @@ -3810,7 +3886,10 @@ async fn raw_cached_lookup_observes_takeover_committed_by_another_oplog_actor() }), ) .await; - second.commit(CommitLevel::Always).await; + second + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let observed = first .raw_durable_stream_session_status(&session_key) @@ -3851,7 +3930,10 @@ async fn raw_cache_eviction_recovers_finished_session_and_folds_buffered_then_co }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert!( oplog .raw_durable_stream_session_status(&session_key) @@ -3887,7 +3969,10 @@ async fn raw_cache_eviction_recovers_finished_session_and_folds_buffered_then_co ); assert_eq!(buffered.attachment_attached, Some(false)); - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let committed = oplog .raw_durable_stream_session_status(&session_key) .await @@ -3923,7 +4008,10 @@ async fn persisted_exact_horizon_rejects_newer_index_and_offsets_hide_newer_atta ), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); service .stream_session_index .catch_up(&id, AgentMode::Durable, test_fingerprint(), attached_idx) @@ -4021,7 +4109,10 @@ async fn raw_lookup_catches_up_archived_history_after_full_multilayer_reopen() { }), ) .await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert_eq!( MultiLayerOplog::try_archive_blocking(&oplog).await, Some(true) @@ -4069,6 +4160,7 @@ async fn raw_lookup_catches_up_archived_history_after_full_multilayer_reopen() { agent_metadata(&id), stale_status(), suspended_status(), + None, ) .await; let raw = reopened @@ -4124,7 +4216,10 @@ async fn indexed_raw_authority_cold_and_warm_lookups_do_not_read_oplog_history() for _ in 0..2048 { append_noop(oplog.as_ref()).await; } - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); let horizon = oplog.current_oplog_index().await; service .stream_session_index @@ -4141,6 +4236,7 @@ async fn indexed_raw_authority_cold_and_warm_lookups_do_not_read_oplog_history() agent_metadata(&id), stale_status(), suspended_status(), + None, ) .await; storage.reset(); @@ -4209,7 +4305,10 @@ async fn raw_authority_ignores_foreign_results_sharing_an_idempotency_key() { append_session(oplog.as_ref(), second.clone()).await; append_session(oplog.as_ref(), first).await; append_session(oplog.as_ref(), second).await; - oplog.commit(CommitLevel::Always).await; + oplog + .commit(CommitLevel::Always) + .await + .expect("oplog write"); assert!( oplog @@ -4247,7 +4346,7 @@ async fn raw_authority_ignores_foreign_results_sharing_an_idempotency_key() { .await; for committed in [false, true] { if committed { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); } let status = oplog .raw_durable_stream_session_status(&first_key) diff --git a/golem-worker-executor/src/services/worker_fork.rs b/golem-worker-executor/src/services/worker_fork.rs index a0bbfbdd24..f706b7eb28 100644 --- a/golem-worker-executor/src/services/worker_fork.rs +++ b/golem-worker-executor/src/services/worker_fork.rs @@ -664,6 +664,8 @@ impl DefaultWorkerFork { timestamp: Timestamp::now_utc(), }, ))), + // The source is read, never written, so this handle asserts no epoch. + None, ) .await; let source_oplog = Ctx::wrap_oplog( @@ -761,7 +763,7 @@ impl DefaultWorkerFork { ) .await .map_err(WorkerExecutorError::runtime)?; - new_oplog.add(target_initial_oplog_entry).await; + new_oplog.add(target_initial_oplog_entry).await?; let oplog_range = OplogIndexRange::new(OplogIndex::INITIAL.next(), oplog_index_cut_off); @@ -830,7 +832,7 @@ impl DefaultWorkerFork { *cached = Some(Arc::new(value)); } } - new_oplog.add(entry.clone()).await; + new_oplog.add(entry.clone()).await?; if let OplogEntry::Revert { dropped_region, .. } = &entry { deleted_regions_builder.add(dropped_region.clone()); @@ -891,7 +893,7 @@ impl DefaultWorkerFork { entity_parent_start_index: None, record, }) - .await; + .await?; for (idempotency_key, pending_index) in pending_invocation_keys { if let Some(candidate) = export { @@ -917,7 +919,7 @@ impl DefaultWorkerFork { timestamp: now, idempotency_key, }) - .await; + .await?; } for target_revision in pending_update_revisions { @@ -930,7 +932,7 @@ impl DefaultWorkerFork { target_revision, details: Some("cancelled by fork".to_string()), }) - .await; + .await?; } if let Some(candidate) = export @@ -1033,7 +1035,7 @@ impl DefaultWorkerFork { if let Some((scope, phantom)) = guest_result { publication::write_guest_result(oplog.as_ref(), scope, phantom).await?; } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await?; let last = oplog.current_oplog_index().await; drop(oplog); let target_lifecycle = self.oplog_service.lock_lifecycle(&target.agent_id).await; diff --git a/golem-worker-executor/src/services/worker_fork/export.rs b/golem-worker-executor/src/services/worker_fork/export.rs index ed86a1b38e..019146402c 100644 --- a/golem-worker-executor/src/services/worker_fork/export.rs +++ b/golem-worker-executor/src/services/worker_fork/export.rs @@ -45,6 +45,12 @@ impl From for Error { } } +impl From for Error { + fn from(error: crate::services::oplog::OplogError) -> Self { + Self::Worker(error.into()) + } +} + fn reject(reason: Reason) -> Error { Error::Rejected(ForkStreamSlotRejection { reason: reason as i32, @@ -260,7 +266,7 @@ async fn execute( ) .await?; } - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await?; let target_lifecycle = service.oplog_service.lock_lifecycle(&target.agent_id).await; let expiry_deadline_millis = admitted_publication_deadline(&candidate); append_target_initialization(oplog.as_ref(), &candidate, hash, expiry_deadline_millis) @@ -273,7 +279,7 @@ async fn execute( expiry_deadline_millis, ) .await?; - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await?; let last = oplog.current_oplog_index().await; drop(oplog); let published = service @@ -432,7 +438,7 @@ async fn append_target_initialization( entity_parent_start_index: None, record, }) - .await; + .await?; Ok(()) } @@ -474,9 +480,11 @@ async fn prepare_candidate( let worker = Worker::find_durable_stream_worker(service, source) .await? .ok_or_else(|| reject(Reason::NotFound))?; + // A refused commit means the source has a new owner: fail the fork rather than read a + // horizon this executor is no longer allowed to write past. worker .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; let slot = worker .resolve_export_fork_slot(&request.session, &request.slot, &request.expected_method) .await? @@ -582,7 +590,7 @@ async fn prepare_candidate( // durable before staging reads the source from storage. worker .commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; Ok(Candidate { export: StreamExportFork { source: source.agent_id.clone(), diff --git a/golem-worker-executor/src/services/worker_fork/lineage.rs b/golem-worker-executor/src/services/worker_fork/lineage.rs index 1d15abae31..1daad20e95 100644 --- a/golem-worker-executor/src/services/worker_fork/lineage.rs +++ b/golem-worker-executor/src/services/worker_fork/lineage.rs @@ -566,6 +566,7 @@ pub(crate) mod tests { timestamp: Timestamp::now_utc(), }, ))), + None, ) .await; Self { @@ -690,7 +691,8 @@ pub(crate) mod tests { timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); } if let Some(matching) = matching { fixture @@ -698,7 +700,8 @@ pub(crate) mod tests { .add(OplogEntry::revert(OplogRegion::from_range( 2..=if matching { 3 } else { 2 }, ))) - .await; + .await + .unwrap(); } else { fixture .oplog @@ -706,7 +709,8 @@ pub(crate) mod tests { timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); } let marker = StreamSessionRecord::ForkCut(self_revert(3)); fixture @@ -716,7 +720,8 @@ pub(crate) mod tests { entity_parent_start_index: None, record: fixture.oplog.upload_payload(&marker).await.unwrap(), }) - .await; + .await + .unwrap(); let result = StreamForkLineage::load(&*fixture.oplog, &identity, fingerprint).await; assert_eq!(result.is_ok(), matching == Some(true)); } @@ -730,11 +735,13 @@ pub(crate) mod tests { fixture .oplog .add(OplogEntry::jump(None, OplogRegion::from_range(2..=2))) - .await; + .await + .unwrap(); fixture .oplog .add(OplogEntry::revert(OplogRegion::from_range(2..=2))) - .await; + .await + .unwrap(); let lineage = StreamForkLineage::load(&*fixture.oplog, &identity, fingerprint) .await .unwrap(); @@ -854,10 +861,14 @@ pub(crate) mod tests { entity_parent_start_index: None, }, }; - fixture.oplog.add(entry).await; + fixture.oplog.add(entry).await.unwrap(); } let region = OplogRegion::from_range(7..=9); - fixture.oplog.add(OplogEntry::revert(region.clone())).await; + fixture + .oplog + .add(OplogEntry::revert(region.clone())) + .await + .unwrap(); let mut marker = fork(None, OplogIndex::from_u64(6)); marker.revert = Some(region); marker.epoch_floor = 2; @@ -872,8 +883,9 @@ pub(crate) mod tests { .await .unwrap(), }) - .await; - fixture.oplog.commit(CommitLevel::Always).await; + .await + .unwrap(); + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); fixture.blobs.reset(); let lineage = StreamForkLineage::load_from_service( @@ -939,7 +951,7 @@ pub(crate) mod tests { entity_parent_start_index: None, }, }; - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } for _ in 0..2 { assert_eq!( @@ -1015,7 +1027,7 @@ pub(crate) mod tests { entity_parent_start_index: None, } }; - oplog.add(entry).await; + oplog.add(entry).await.unwrap(); } assert_eq!( StreamForkLineage::load(&oplog, &target, fingerprint) @@ -1067,9 +1079,12 @@ pub(crate) mod tests { entity_parent_start_index: None, }, }; - assert_eq!(fixture.oplog.add(entry).await, OplogIndex::from_u64(index)); + assert_eq!( + fixture.oplog.add(entry).await.unwrap(), + OplogIndex::from_u64(index) + ); } - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); let from_oplog = StreamForkLineage::load(&*fixture.oplog, &target, fingerprint) .await .unwrap(); @@ -1113,11 +1128,12 @@ pub(crate) mod tests { timestamp: Timestamp::now_utc(), entity_parent_start_index: None, }) - .await, + .await + .unwrap(), OplogIndex::from_u64(index) ); } - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); assert!( !StreamForkLineage::suffix_contains_fork_cut( &*fixture.service, @@ -1138,10 +1154,11 @@ pub(crate) mod tests { entity_parent_start_index: None, record: fixture.oplog.upload_payload(&marker).await.unwrap(), }) - .await, + .await + .unwrap(), OplogIndex::from_u64(1025) ); - fixture.oplog.commit(CommitLevel::Always).await; + fixture.oplog.commit(CommitLevel::Always).await.unwrap(); assert!( StreamForkLineage::suffix_contains_fork_cut( &*fixture.service, diff --git a/golem-worker-executor/src/services/worker_fork/publication.rs b/golem-worker-executor/src/services/worker_fork/publication.rs index 65e1a1772f..8a9e2275df 100644 --- a/golem-worker-executor/src/services/worker_fork/publication.rs +++ b/golem-worker-executor/src/services/worker_fork/publication.rs @@ -129,7 +129,7 @@ pub(crate) async fn write_guest_result( forced_commit: false, }), ) - .await; + .await?; if let Some(start_index) = copied_scope_start { oplog .add(OplogEntry::End { @@ -138,7 +138,7 @@ pub(crate) async fn write_guest_result( response: None, forced_commit: true, }) - .await; + .await?; } Ok(()) } @@ -288,8 +288,9 @@ mod tests { instance_id: fingerprint.0, }, ))) - .await; - stage.add(OplogEntry::suspend()).await; + .await + .unwrap(); + stage.add(OplogEntry::suspend()).await.unwrap(); let record = StreamSessionRecord::ForkCut(StreamForkCutRecord { format_version: 1, request_hash: hash.to_vec(), @@ -308,13 +309,14 @@ mod tests { entity_parent_start_index: None, record, }) - .await; + .await + .unwrap(); write_guest_result(stage.as_ref(), None, phantom) .await .unwrap(); // Extra entries after creation must not affect retry recognition. - stage.add(OplogEntry::suspend()).await; - stage.commit(CommitLevel::Always).await; + stage.add(OplogEntry::suspend()).await.unwrap(); + stage.commit(CommitLevel::Always).await.unwrap(); let last = stage.current_oplog_index().await; drop(stage); assert!(!existing_fork(&service, &target, cut, hash).await.unwrap()); diff --git a/golem-worker-executor/src/storage/indexed/memory.rs b/golem-worker-executor/src/storage/indexed/memory.rs index eb846ebffa..1d11e4f161 100644 --- a/golem-worker-executor/src/storage/indexed/memory.rs +++ b/golem-worker-executor/src/storage/indexed/memory.rs @@ -14,22 +14,30 @@ use crate::storage::indexed::{ IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, - ScanResume, + ScanResume, WriterId, }; use async_trait::async_trait; use golem_common::model::AgentId; +use golem_common::model::ShardEpoch; use regex::Regex; use std::collections::{BTreeMap, BinaryHeap}; use std::ops::Bound::Included; +use std::sync::Arc; #[cfg(test)] use std::sync::atomic::{AtomicU64, Ordering}; use std::time::Duration; +/// The maps are shared, so [`Self::for_writer`] can hand out a second handle onto the same store +/// that writes as somebody else. #[derive(Debug)] pub struct InMemoryIndexedStorage { - data: scc::HashMap>>, + data: Arc>>>, + /// The writer generation recorded per key. An append that asserts an epoch holds this entry + /// while it writes `data`, which is what makes the check and the insert one step. + key_epochs: Arc>, + writer_id: WriterId, #[cfg(test)] - read_count: AtomicU64, + read_count: Arc, } impl Default for InMemoryIndexedStorage { @@ -41,12 +49,83 @@ impl Default for InMemoryIndexedStorage { impl InMemoryIndexedStorage { pub fn new() -> Self { Self { - data: scc::HashMap::new(), + data: Arc::new(scc::HashMap::new()), + key_epochs: Arc::new(scc::HashMap::new()), + writer_id: WriterId::process(), #[cfg(test)] - read_count: AtomicU64::new(0), + read_count: Arc::new(AtomicU64::new(0)), } } + /// A second handle onto this same store that writes as `writer_id`: how two processes racing + /// over one key are played out inside a single one. + pub fn for_writer(&self, writer_id: WriterId) -> Self { + Self { + data: self.data.clone(), + key_epochs: self.key_epochs.clone(), + writer_id, + #[cfg(test)] + read_count: self.read_count.clone(), + } + } + + /// Refuses unless `record` holds exactly `expected`, recorded by this writer; an absent record + /// refuses too. The same terms as the SQL backends' check. + fn check_record( + &self, + key: &str, + expected: ShardEpoch, + record: &scc::hash_map::Entry<'_, String, (ShardEpoch, WriterId)>, + ) -> Result<(), IndexedStorageError> { + let stored = match record { + scc::hash_map::Entry::Occupied(occupied) => Some(*occupied.get()), + scc::hash_map::Entry::Vacant(_) => None, + }; + match stored { + Some((epoch, writer)) if epoch == expected && writer == self.writer_id => Ok(()), + other => Err(IndexedStorageError::Fenced { + key: key.to_string(), + expected, + actual: other.map(|(epoch, _)| epoch), + writer_conflict: other + .is_some_and(|(epoch, writer)| epoch == expected && writer != self.writer_id), + }), + } + } + + /// Inserts `pairs` under `composite_key`, all or nothing, after checking `expected_epoch` + /// against the key's record. The record's entry is held until the insert is done. + async fn append_checked( + &self, + composite_key: String, + key: &str, + pairs: &[(u64, Vec)], + expected_epoch: Option, + primary_oplog_insert: bool, + ) -> Result<(), IndexedStorageError> { + let _record = match expected_epoch { + None => None, + Some(expected) => { + let record = self.key_epochs.entry_async(composite_key.clone()).await; + self.check_record(key, expected, &record)?; + Some(record) + } + }; + + let mut entry = self.data.entry_async(composite_key).await.or_default(); + if pairs.iter().any(|(id, _)| entry.contains_key(id)) { + return Err(if primary_oplog_insert { + IndexedStorageError::Conflict("Key already exists".to_string()) + } else { + IndexedStorageError::Other("Key already exists".to_string()) + }); + } + for (id, value) in pairs { + entry.get_mut().insert(*id, value.clone()); + } + Ok(()) + } + #[cfg(test)] pub(crate) fn read_count(&self) -> u64 { self.read_count.load(Ordering::Relaxed) @@ -210,29 +289,116 @@ impl IndexedStorage for InMemoryIndexedStorage { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { let primary_oplog_insert = matches!( &namespace, IndexedStorageNamespace::OpLog { .. } | IndexedStorageNamespace::StagedOpLog { .. } ); let composite_key = Self::composite_key(namespace, key); - let mut entry = self - .data - .entry_async(composite_key.clone()) - .await - .or_default(); - if let std::collections::btree_map::Entry::Vacant(e) = entry.entry(id) { - e.insert(value.to_vec()); - Ok(()) - } else if primary_oplog_insert { - Err(IndexedStorageError::Conflict( - "Key already exists".to_string(), - )) - } else { - Err(IndexedStorageError::Other("Key already exists".to_string())) + self.append_checked( + composite_key, + key, + &[(id, value)], + expected_epoch, + primary_oplog_insert, + ) + .await + } + + async fn append_many( + &self, + _svc_name: &'static str, + _api_name: &'static str, + _entity_name: &'static str, + namespace: &IndexedStorageNamespace, + key: &str, + pairs: Arc<[(u64, bytes::Bytes)]>, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + // Nothing to write is nothing to fence, as on every other backend. + if pairs.is_empty() { + return Ok(()); + } + let primary_oplog_insert = matches!( + namespace, + IndexedStorageNamespace::OpLog { .. } | IndexedStorageNamespace::StagedOpLog { .. } + ); + let composite_key = Self::composite_key(namespace.clone(), key); + let pairs: Vec<(u64, Vec)> = pairs + .iter() + .map(|(id, value)| (*id, value.to_vec())) + .collect(); + self.append_checked( + composite_key, + key, + &pairs, + expected_epoch, + primary_oplog_insert, + ) + .await + } + + async fn set_key_epoch( + &self, + _svc_name: &'static str, + _api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + let composite_key = Self::composite_key(namespace, key); + match self.key_epochs.entry_async(composite_key).await { + scc::hash_map::Entry::Vacant(vacant) => { + vacant.insert_entry((epoch, self.writer_id)); + Ok(()) + } + scc::hash_map::Entry::Occupied(mut occupied) => { + let (stored, writer) = *occupied.get(); + if epoch > stored || (epoch == stored && writer == self.writer_id) { + *occupied.get_mut() = (epoch, self.writer_id); + Ok(()) + } else { + Err(IndexedStorageError::Fenced { + key: key.to_string(), + expected: epoch, + actual: Some(stored), + writer_conflict: epoch == stored && writer != self.writer_id, + }) + } + } } } + async fn delete_with_epoch( + &self, + _svc_name: &'static str, + _api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + let composite_key = Self::composite_key(namespace, key); + // The record's guard first and held across the data removal, in the order an append takes + // them, so nobody can record a new generation between the check and the deletes. + let record = self.key_epochs.entry_async(composite_key.clone()).await; + if let Some(expected) = expected_epoch { + // Neither a record nor entries: already gone, most often by an earlier attempt of + // this same deletion (see the trait). + if matches!(record, scc::hash_map::Entry::Vacant(_)) + && !self.data.contains_async(&composite_key).await + { + return Ok(()); + } + self.check_record(key, expected, &record)?; + } + self.data.remove_async(&composite_key).await; + if let scc::hash_map::Entry::Occupied(occupied) = record { + let _ = occupied.remove(); + } + Ok(()) + } + async fn move_if_absent( &self, _svc_name: &'static str, @@ -474,6 +640,7 @@ mod tests { "stage", id, value.to_vec(), + None, ) .await .unwrap(); @@ -549,6 +716,7 @@ mod tests { key, id, value, + None, ) .await .unwrap(); @@ -597,6 +765,7 @@ mod tests { stage, 1, stage.as_bytes().to_vec(), + None, ) .await .unwrap(); @@ -697,6 +866,7 @@ mod tests { "third", 1, b"staged".to_vec(), + None, ) .await .unwrap(); @@ -732,6 +902,7 @@ mod tests { "ordinary-race", 1, b"ordinary".to_vec(), + None, ) .await .is_ok() @@ -755,6 +926,7 @@ mod tests { key, 1, &100, + None, ) .await .unwrap(); @@ -822,6 +994,7 @@ mod tests { key, 1, &100, + None, ) .await .unwrap(); @@ -833,6 +1006,7 @@ mod tests { key, 2, &200, + None, ) .await .unwrap(); @@ -844,6 +1018,7 @@ mod tests { key, 3, &300, + None, ) .await .unwrap(); @@ -855,6 +1030,7 @@ mod tests { key, 4, &400, + None, ) .await .unwrap(); @@ -888,6 +1064,7 @@ mod tests { key, 1, &100, + None, ) .await .unwrap(); @@ -899,6 +1076,7 @@ mod tests { key, 2, &200, + None, ) .await .unwrap(); @@ -910,6 +1088,7 @@ mod tests { key, 3, &300, + None, ) .await .unwrap(); @@ -921,6 +1100,7 @@ mod tests { key, 4, &400, + None, ) .await .unwrap(); @@ -954,6 +1134,7 @@ mod tests { key, 10, &100, + None, ) .await .unwrap(); @@ -965,6 +1146,7 @@ mod tests { key, 20, &200, + None, ) .await .unwrap(); @@ -976,6 +1158,7 @@ mod tests { key, 30, &300, + None, ) .await .unwrap(); @@ -987,6 +1170,7 @@ mod tests { key, 40, &400, + None, ) .await .unwrap(); @@ -1020,6 +1204,7 @@ mod tests { key, 10, &100, + None, ) .await .unwrap(); @@ -1031,6 +1216,7 @@ mod tests { key, 20, &200, + None, ) .await .unwrap(); @@ -1042,6 +1228,7 @@ mod tests { key, 30, &300, + None, ) .await .unwrap(); @@ -1053,6 +1240,7 @@ mod tests { key, 40, &400, + None, ) .await .unwrap(); @@ -1087,6 +1275,7 @@ mod tests { key, 10, &100, + None, ) .await .unwrap(); @@ -1098,6 +1287,7 @@ mod tests { key, 20, &200, + None, ) .await .unwrap(); @@ -1109,6 +1299,7 @@ mod tests { key, 30, &300, + None, ) .await .unwrap(); @@ -1120,6 +1311,7 @@ mod tests { key, 40, &400, + None, ) .await .unwrap(); @@ -1154,6 +1346,7 @@ mod tests { key, 10, &100, + None, ) .await .unwrap(); @@ -1165,6 +1358,7 @@ mod tests { key, 20, &200, + None, ) .await .unwrap(); @@ -1197,6 +1391,7 @@ mod tests { key, 10, &100, + None, ) .await .unwrap(); @@ -1208,6 +1403,7 @@ mod tests { key, 20, &200, + None, ) .await .unwrap(); @@ -1240,6 +1436,7 @@ mod tests { key, 1, &100, + None, ) .await .unwrap(); @@ -1251,6 +1448,7 @@ mod tests { key, 2, &200, + None, ) .await .unwrap(); @@ -1262,6 +1460,7 @@ mod tests { key, 3, &300, + None, ) .await .unwrap(); @@ -1273,6 +1472,7 @@ mod tests { key, 4, &400, + None, ) .await .unwrap(); diff --git a/golem-worker-executor/src/storage/indexed/mod.rs b/golem-worker-executor/src/storage/indexed/mod.rs index 592af61401..a349154dc4 100644 --- a/golem-worker-executor/src/storage/indexed/mod.rs +++ b/golem-worker-executor/src/storage/indexed/mod.rs @@ -13,16 +13,17 @@ // limitations under the License. use std::fmt::{self, Debug, Display, Formatter}; -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use std::time::Duration; use async_trait::async_trait; use bytes::Bytes; use desert_rust::{BinaryDeserializer, BinarySerializer}; -use golem_common::model::AgentId; use golem_common::model::agent::AgentMode; +use golem_common::model::{AgentId, ShardEpoch}; use golem_common::serialization::{deserialize, serialize}; -use golem_service_base::repo::is_transient_sqlx_error; +use golem_service_base::repo::{RepoError, is_transient_sqlx_error}; +use uuid::Uuid; pub mod memory; pub mod multi_sqlite; @@ -46,6 +47,37 @@ pub enum IndexedStorageError { InvalidResume(String), /// Permanent error — data issue or schema error. Caller should not retry. Other(String), + /// The write was refused because the epoch it asserted is not the one recorded for the key, + /// or the record is held by another writer at that epoch. + Fenced { + key: String, + expected: ShardEpoch, + actual: Option, + /// The stored epoch equals the asserted one but another writer recorded it, so the epoch + /// alone no longer says who may write - see [`WriterId`]. + writer_conflict: bool, + }, +} + +/// The process behind a write, recorded alongside the epoch it asserts. +/// +/// One value per process, kept for the life of the process, so that whatever issues epochs can +/// re-issue one without this process losing the keys it already holds at it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct WriterId(pub Uuid); + +impl WriterId { + /// This process's writer identity, created once on first use. + pub fn process() -> Self { + static PROCESS: OnceLock = OnceLock::new(); + *PROCESS.get_or_init(|| WriterId(Uuid::new_v4())) + } +} + +impl Display for WriterId { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } } impl IndexedStorageError { @@ -79,6 +111,28 @@ impl Display for IndexedStorageError { IndexedStorageError::Conflict(msg) => write!(f, "Storage conflict: {msg}"), IndexedStorageError::InvalidResume(msg) => write!(f, "Invalid scan resume: {msg}"), IndexedStorageError::Other(msg) => write!(f, "Storage error: {msg}"), + IndexedStorageError::Fenced { + key, + expected, + actual, + writer_conflict, + } => match actual { + Some(actual) if *writer_conflict => write!( + f, + "Write fenced for key {key}: asserted epoch {expected}, \ + which another writer holds - the stored epoch is {actual}" + ), + Some(actual) => write!( + f, + "Write fenced for key {key}: asserted epoch {expected}, \ + the stored epoch is {actual}" + ), + None => write!( + f, + "Write fenced for key {key}: asserted epoch {expected}, \ + but no epoch is stored for it" + ), + }, } } } @@ -91,6 +145,77 @@ impl From for IndexedStorageError { } } +/// Carries a fence rejection out of a transaction closure. +#[derive(Debug)] +pub(crate) enum FencedTxError { + Repo(RepoError), + Fenced { + key: String, + expected: ShardEpoch, + actual: Option, + writer_conflict: bool, + }, + /// A stored value the schema should have made impossible - a negative epoch, say. Not a fence: + /// nobody took the key over, the row itself cannot be trusted. + Corrupt(String), +} + +impl From for FencedTxError { + fn from(err: RepoError) -> Self { + FencedTxError::Repo(err) + } +} + +impl FencedTxError { + pub(crate) fn check_record( + key: &str, + expected: ShardEpoch, + stored: Option<(i64, String)>, + writer_id: &str, + negative_epoch_message: fn(i64, &str) -> String, + ) -> Result<(), FencedTxError> { + let mut actual = None; + let mut writer_matches = false; + if let Some((epoch, writer)) = stored { + let epoch = u64::try_from(epoch) + .map_err(|_| FencedTxError::Corrupt(negative_epoch_message(epoch, key)))?; + actual = Some(ShardEpoch(epoch)); + writer_matches = writer == writer_id; + } + if actual != Some(expected) || !writer_matches { + return Err(FencedTxError::Fenced { + key: key.to_string(), + expected, + actual, + writer_conflict: actual == Some(expected) && !writer_matches, + }); + } + Ok(()) + } + + /// `classify` is the backend's own `RepoError` classifier. + pub(crate) fn into_indexed_storage_error( + self, + classify: fn(RepoError) -> IndexedStorageError, + ) -> IndexedStorageError { + match self { + FencedTxError::Repo(err) => classify(err), + FencedTxError::Fenced { + key, + expected, + actual, + writer_conflict, + } => IndexedStorageError::Fenced { + key, + expected, + actual, + writer_conflict, + }, + FencedTxError::Corrupt(msg) => IndexedStorageError::Other(msg), + } + } +} + /// Where a [`IndexedStorage::scan_stable`] walk left off. Only the backend that produced it can /// read it; a caller passes it back unchanged. #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] @@ -233,7 +358,8 @@ pub trait IndexedStorage: Debug + Sync { count: u64, ) -> Result<(Option, Vec), IndexedStorageError>; - /// Appends an entry to the given key with the given id + /// Appends an entry to the given key with the given id. `expected_epoch` is checked as in + /// [`Self::append_many`]. async fn append( &self, svc_name: &'static str, @@ -243,9 +369,10 @@ pub trait IndexedStorage: Debug + Sync { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError>; - /// Appends multiple entries to the given key with the given id + /// Appends multiple entries to the given key with the given ids, all or nothing. async fn append_many( &self, svc_name: &'static str, @@ -254,21 +381,8 @@ pub trait IndexedStorage: Debug + Sync { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, - ) -> Result<(), IndexedStorageError> { - for (id, value) in pairs.iter() { - self.append( - svc_name, - api_name, - entity_name, - (*namespace).clone(), - key, - *id, - value.to_vec(), - ) - .await?; - } - Ok(()) - } + expected_epoch: Option, + ) -> Result<(), IndexedStorageError>; /// Atomically moves a stopped source index to a previously absent target index. The source /// must contain exactly ids 1..=expected_last_id. Returns false without mutation if the target @@ -373,6 +487,30 @@ pub trait IndexedStorage: Debug + Sync { key: &str, last_dropped_id: u64, ) -> Result<(), IndexedStorageError>; + + /// Records the writer generation for the given key: `epoch`, and this process as the writer + /// holding it. A monotonic compare-and-set - accepted when `epoch` is above the stored one, or + /// equal to it and recorded by this same writer, and refused with + /// [`IndexedStorageError::Fenced`] otherwise. Inserts the record if the key has none. + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError>; + + /// Deletes the index of the given key, as [`Self::delete`] does, together with the writer + /// generation recorded for it, in one step. + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError>; } pub trait IndexedStorageLabelledApi { @@ -532,6 +670,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { key: &str, id: u64, value: &V, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.storage .append( @@ -542,6 +681,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { key, id, serialize(value).map_err(IndexedStorageError::Other)?, + expected_epoch, ) .await } @@ -553,6 +693,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.storage .append( @@ -563,6 +704,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { key, id, value, + expected_epoch, ) .await } @@ -574,6 +716,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { namespace: &IndexedStorageNamespace, key: &str, pairs: &[(u64, &V)], + expected_epoch: Option, ) -> Result { let mut serialized_pairs = Vec::with_capacity(pairs.len()); let mut total_bytes = 0u64; @@ -582,7 +725,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { total_bytes += bytes.len() as u64; serialized_pairs.push((*id, Bytes::from(bytes))); } - self.append_many_raw(namespace, key, serialized_pairs.into()) + self.append_many_raw(namespace, key, serialized_pairs.into(), expected_epoch) .await?; Ok(total_bytes) } @@ -593,6 +736,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.storage .append_many( @@ -602,6 +746,7 @@ impl<'a, S: ?Sized + IndexedStorage> LabelledEntityIndexedStorage<'a, S> { namespace, key, pairs, + expected_epoch, ) .await } diff --git a/golem-worker-executor/src/storage/indexed/multi_sqlite.rs b/golem-worker-executor/src/storage/indexed/multi_sqlite.rs index a24d328f30..f336772430 100644 --- a/golem-worker-executor/src/storage/indexed/multi_sqlite.rs +++ b/golem-worker-executor/src/storage/indexed/multi_sqlite.rs @@ -14,7 +14,7 @@ use super::{ IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, - ScanResume, + ScanResume, WriterId, }; use crate::storage::indexed::sqlite::SqliteIndexedStorage; use async_trait::async_trait; @@ -22,6 +22,7 @@ use bytes::Bytes; use golem_common::cache::{BackgroundEvictionMode, Cache, FullCacheEvictionMode, SimpleCache}; use golem_common::config::DbSqliteConfig; use golem_common::model::AgentId; +use golem_common::model::ShardEpoch; use std::collections::HashMap; use std::fmt::{Debug, Formatter}; use std::path::{Path, PathBuf}; @@ -48,6 +49,9 @@ pub struct MultiSqliteIndexedStorage { root_dir: PathBuf, max_connections: u32, foreign_keys: bool, + /// Handed to every per-namespace SQLite storage this opens, so the whole fan-out writes as one + /// process. See [`WriterId`]. + writer_id: WriterId, } struct HashCache { @@ -84,20 +88,32 @@ impl MultiSqliteIndexedStorage { root_dir: root_dir.to_path_buf(), max_connections, foreign_keys, + writer_id: WriterId::process(), } } + /// Writes as `writer_id` rather than as this process's own. The fan-out backend uses it to + /// give every storage it opens one identity, and a test uses it to play two processes racing + /// over one key inside a single process. + pub fn for_writer(mut self, writer_id: WriterId) -> Self { + self.writer_id = writer_id; + self + } + async fn init_storage( max_connections: u32, foreign_keys: bool, database: String, + writer_id: WriterId, ) -> Result { let config = DbSqliteConfig { database, max_connections, foreign_keys, }; - SqliteIndexedStorage::configured(&config).await + Ok(SqliteIndexedStorage::configured(&config) + .await? + .for_writer(writer_id)) } async fn storage_by_namespace( @@ -186,6 +202,7 @@ impl MultiSqliteIndexedStorage { ) -> Result { let max_connections = self.max_connections; let foreign_keys = self.foreign_keys; + let writer_id = self.writer_id; let db_path = self.root_dir.join(db.clone()).to_string_lossy().to_string(); // Set when this call creates the file, which makes cached listings stale. Checked only on a // cache miss, since a hit means the file is already open. @@ -196,7 +213,7 @@ impl MultiSqliteIndexedStorage { .cache .get_or_insert_simple(&db, async move || { flag.store(!Path::new(&existing).exists(), Ordering::SeqCst); - Self::init_storage(max_connections, foreign_keys, db_path).await + Self::init_storage(max_connections, foreign_keys, db_path, writer_id).await }) .await?; if created.load(Ordering::SeqCst) { @@ -262,6 +279,34 @@ impl Debug for MultiSqliteIndexedStorage { #[async_trait] impl IndexedStorage for MultiSqliteIndexedStorage { + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + new_epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + self.storage_by_namespace(&namespace) + .await? + .set_key_epoch(svc_name, api_name, namespace, key, new_epoch) + .await + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + self.storage_by_namespace(&namespace) + .await? + .delete_with_epoch(svc_name, api_name, namespace, key, expected_epoch) + .await + } + async fn number_of_replicas( &self, _svc_name: &'static str, @@ -367,13 +412,27 @@ impl IndexedStorage for MultiSqliteIndexedStorage { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.storage_by_namespace(&namespace) .await? - .append(svc_name, api_name, entity_name, namespace, key, id, value) + .append( + svc_name, + api_name, + entity_name, + namespace, + key, + id, + value, + expected_epoch, + ) .await } + /// Overridden rather than inherited. The trait default loops [`Self::append`], which would + /// resolve the per-agent database and re-check the fence once per entry, in a separate + /// transaction each time - so a batch could land half-written, and the contract that the + /// fence is checked once per call would not hold. async fn append_many( &self, svc_name: &'static str, @@ -382,10 +441,19 @@ impl IndexedStorage for MultiSqliteIndexedStorage { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { self.storage_by_namespace(namespace) .await? - .append_many(svc_name, api_name, entity_name, namespace, key, pairs) + .append_many( + svc_name, + api_name, + entity_name, + namespace, + key, + pairs, + expected_epoch, + ) .await } @@ -574,6 +642,7 @@ mod tests { &first_namespace, "shared-key", vec![(1, Bytes::from_static(b"first-agent-value"))].into(), + None, ) .await .unwrap(); @@ -585,6 +654,7 @@ mod tests { &second_namespace, "shared-key", vec![(1, Bytes::from_static(b"second-agent-value"))].into(), + None, ) .await .unwrap(); diff --git a/golem-worker-executor/src/storage/indexed/postgres.rs b/golem-worker-executor/src/storage/indexed/postgres.rs index 01964e1d01..de50aace78 100644 --- a/golem-worker-executor/src/storage/indexed/postgres.rs +++ b/golem-worker-executor/src/storage/indexed/postgres.rs @@ -13,8 +13,8 @@ // limitations under the License. use super::{ - IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, - ScanResume, + FencedTxError, IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, + IndexedStorageNamespace, ScanResume, WriterId, }; use crate::services::golem_config::IndexedStoragePostgresConfig; use async_trait::async_trait; @@ -22,6 +22,7 @@ use bytes::Bytes; use futures::FutureExt; use golem_common::SafeDisplay; use golem_common::metrics::db::record_db_serialized_size; +use golem_common::model::ShardEpoch; use golem_service_base::db::postgres::PostgresPool; use golem_service_base::db::{Pool, PoolApi}; use golem_service_base::migration::{IncludedMigrationsDir, Migrations}; @@ -45,6 +46,9 @@ pub struct PostgresIndexedStorage { pool: PostgresPool, drop_prefix_delete_batch_size: u64, semaphore: Option>, + /// Recorded beside the epoch on every key this process claims, so an equal epoch from + /// another process is refused rather than shared. One per process; see [`WriterId`]. + writer_id: WriterId, } impl PostgresIndexedStorage { @@ -83,6 +87,7 @@ impl PostgresIndexedStorage { pool, drop_prefix_delete_batch_size: config.drop_prefix_delete_batch_size, semaphore, + writer_id: WriterId::process(), }) } @@ -91,9 +96,18 @@ impl PostgresIndexedStorage { pool, drop_prefix_delete_batch_size: 1024, semaphore: None, + writer_id: WriterId::process(), }) } + /// Writes as `writer_id` rather than as this process's own. The fan-out backend uses it to + /// give every storage it opens one identity, and a test uses it to play two executors racing + /// over one oplog inside a single process. + pub fn for_writer(mut self, writer_id: WriterId) -> Self { + self.writer_id = writer_id; + self + } + pub async fn run_metrics_loop(&self) -> anyhow::Result<()> { self.pool.run_metrics_loop("indexed_storage").await } @@ -146,6 +160,13 @@ impl PostgresIndexedStorage { }) } + /// A stored epoch that will not fit a `u64` is corruption, not a fence. `to_i64` refuses to + /// write one, so a negative column value came from outside this code - and reading it back as + /// `u64` would wrap it into a near-ceiling epoch that fences every writer out of the key. + fn negative_epoch_message(value: i64, key: &str) -> String { + format!("Postgres indexed storage read a negative epoch {value} for key '{key}'") + } + fn classify_repo_error(err: RepoError, primary_oplog_insert: bool) -> IndexedStorageError { if primary_oplog_insert && err.is_pool_timeout() { IndexedStorageError::Transient(err.to_string()) @@ -167,6 +188,12 @@ impl PostgresIndexedStorage { Self::classify_repo_error(err, false) } + /// The oplog-insert classifier as a plain `fn`, so it can be handed to + /// [`FencedTxError::into_indexed_storage_error`], which takes a function pointer. + fn classify_repo_error_oplog_insert(err: RepoError) -> IndexedStorageError { + Self::classify_repo_error(err, true) + } + async fn acquire_permit(&self) -> Option { match &self.semaphore { Some(sem) => Some(sem.clone().acquire_owned().await.expect("semaphore closed")), @@ -264,6 +291,10 @@ impl IndexedStorage for PostgresIndexedStorage { Ok((super::last_key_resume(&keys, count), keys)) } + /// Delegates to [`Self::append_many`] so a single entry and a batch share the id validation, + /// the permit and the epoch check. An entry that asserts an epoch is checked in the same + /// transaction as its insert, like a batch; one that asserts nothing is a single autocommit + /// `INSERT`. The permit is acquired there, not here. async fn append( &self, svc_name: &'static str, @@ -273,28 +304,18 @@ impl IndexedStorage for PostgresIndexedStorage { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { - let _permit = self.acquire_permit().await; - record_db_serialized_size(DB_TYPE, svc_name, entity_name, value.len()); - let primary_oplog_insert = matches!( + self.append_many( + svc_name, + api_name, + entity_name, &namespace, - IndexedStorageNamespace::OpLog { .. } | IndexedStorageNamespace::StagedOpLog { .. } - ); - let id = Self::to_i64(id, "id")?; - let query = sqlx::query( - "INSERT INTO index_storage (namespace, key, id, value) VALUES ($1, $2, $3, $4);", + key, + vec![(id, Bytes::from(value))].into(), + expected_epoch, ) - .bind(Self::namespace(namespace)) - .bind(key) - .bind(id) - .bind(value); - - self.pool - .with_rw(svc_name, api_name) - .execute(query) - .await - .map(|_| ()) - .map_err(|err| Self::classify_repo_error(err, primary_oplog_insert)) + .await } async fn append_many( @@ -305,24 +326,11 @@ impl IndexedStorage for PostgresIndexedStorage { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { if pairs.is_empty() { return Ok(()); } - if let [(id, value)] = pairs.as_ref() { - return self - .append( - svc_name, - api_name, - entity_name, - (*namespace).clone(), - key, - *id, - value.to_vec(), - ) - .await; - } - let _permit = self.acquire_permit().await; let primary_oplog_insert = matches!( namespace, @@ -335,9 +343,54 @@ impl IndexedStorage for PostgresIndexedStorage { Self::to_i64(*id, "id")?; } + // With no epoch asserted there is nothing to check atomically with the insert, so a lone + // entry is one autocommit `INSERT` rather than a transaction held open around it. An + // entry that asserts an epoch takes the transaction below, as a batch does. + if let (None, [(id, value)]) = (expected_epoch, pairs.as_ref()) { + return self + .pool + .with_rw(svc_name, api_name) + .execute( + sqlx::query( + "INSERT INTO index_storage (namespace, key, id, value) VALUES ($1, $2, $3, $4);", + ) + .bind(namespace) + .bind(key) + .bind(i64::try_from(*id).expect("validated oplog index")) + .bind(value.as_ref()), + ) + .await + .map(|_| ()) + .map_err(|err| Self::classify_repo_error(err, primary_oplog_insert)); + } + + let writer_id = self.writer_id.to_string(); self.pool - .with_tx(svc_name, api_name, |tx| { + .with_tx_err::<(), FencedTxError, _>(svc_name, api_name, |tx| { async move { + + // Inside the insert transaction, and holding the row, so a writer that has + // lost the shard cannot slip a batch in between the check and the insert. + // `FOR UPDATE` is what serialises two executors racing over the same oplog. + if let Some(expected) = expected_epoch { + let stored: Option<(i64, String)> = tx + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = $1 AND key = $2 FOR UPDATE;", + ) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + FencedTxError::check_record( + &key, + expected, + stored, + &writer_id, + Self::negative_epoch_message, + )?; + } + for chunk in pairs.chunks(Self::APPEND_MANY_CHUNK_SIZE) { let mut query_builder = QueryBuilder::::new( "INSERT INTO index_storage (namespace, key, id, value) ", @@ -360,7 +413,163 @@ impl IndexedStorage for PostgresIndexedStorage { .boxed() }) .await - .map_err(|err| Self::classify_repo_error(err, primary_oplog_insert)) + .map_err(|err| { + err.into_indexed_storage_error(if primary_oplog_insert { + Self::classify_repo_error_oplog_insert + } else { + Self::classify_repo_error_general + }) + }) + } + + /// Postgres's half of [`IndexedStorage::set_key_epoch`], which states the rule this + /// enforces. + /// + /// The `WHERE` on the conflict path is where it lives: `epoch < EXCLUDED.epoch` for a higher + /// generation, or `= EXCLUDED.epoch AND writer = EXCLUDED.writer` for the same process re-opening + /// at the one it holds. With no record there is no conflict and any epoch is inserted. Postgres + /// reports one row affected for an insert and for an accepted update, and zero when the `WHERE` + /// excludes it - which is what the read-back below turns into a fence. + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + new_epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + let _permit = self.acquire_permit().await; + let namespace = Self::namespace(namespace); + let epoch = Self::to_i64(new_epoch.0, "epoch")?; + + let writer_id = self.writer_id.to_string(); + + let mut api = self.pool.with_rw(svc_name, api_name); + let result = api + .execute( + sqlx::query( + r#"INSERT INTO indexed_key_epoch (namespace, key, epoch, writer) VALUES ($1, $2, $3, $4) + ON CONFLICT (namespace, key) DO UPDATE SET epoch = EXCLUDED.epoch, writer = EXCLUDED.writer + WHERE indexed_key_epoch.epoch < EXCLUDED.epoch + OR (indexed_key_epoch.epoch = EXCLUDED.epoch AND indexed_key_epoch.writer = EXCLUDED.writer);"#, + ) + .bind(namespace.clone()) + .bind(key) + .bind(epoch) + .bind(writer_id.clone()), + ) + .await + .map_err(Self::classify_repo_error_general)?; + + if result.rows_affected() == 0 { + // Rejected. Read the stored epoch back purely so the error can name it. + let stored: Option<(i64, String)> = api + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = $1 AND key = $2;", + ) + .bind(namespace) + .bind(key), + ) + .await + .map_err(Self::classify_repo_error_general)?; + let mut actual = None; + let mut writer_matches = false; + if let Some((epoch, writer)) = stored { + let epoch = u64::try_from(epoch).map_err(|_| { + IndexedStorageError::Other(Self::negative_epoch_message(epoch, key)) + })?; + actual = Some(ShardEpoch(epoch)); + writer_matches = writer == writer_id; + } + return Err(IndexedStorageError::Fenced { + key: key.to_string(), + expected: new_epoch, + actual, + writer_conflict: actual == Some(new_epoch) && !writer_matches, + }); + } + + Ok(()) + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + let _permit = self.acquire_permit().await; + let namespace = Self::namespace(namespace); + let key = key.to_string(); + let writer_id = self.writer_id.to_string(); + self.pool + .with_tx_err::<(), FencedTxError, _>(svc_name, api_name, |tx| { + async move { + // Holding the row, as an append does: a writer taking the key over waits for + // this transaction, and then finds nothing left to take over. + if let Some(expected) = expected_epoch { + let stored: Option<(i64, String)> = tx + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = $1 AND key = $2 FOR UPDATE;", + ) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + // Neither a record nor entries: the key is already gone, most often taken + // by an earlier attempt of this same deletion whose later steps failed, and + // a writer that lost the key has nothing here to destroy. Refusing would + // leave that deletion unable to finish. + if stored.is_none() { + let (has_entries,): (bool,) = tx + .fetch_one_as( + sqlx::query_as( + "SELECT EXISTS(SELECT 1 FROM index_storage WHERE namespace IN ($1, $2) AND key = $3);", + ) + .bind(format!("{namespace}-present")) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + if !has_entries { + return Ok(()); + } + } + FencedTxError::check_record( + &key, + expected, + stored, + &writer_id, + Self::negative_epoch_message, + )?; + } + tx.execute( + sqlx::query( + "DELETE FROM index_storage WHERE namespace IN ($1, $2) AND key = $3;", + ) + .bind(format!("{namespace}-present")) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + tx.execute( + sqlx::query( + "DELETE FROM indexed_key_epoch WHERE namespace = $1 AND key = $2;", + ) + .bind(namespace) + .bind(key), + ) + .await?; + Ok(()) + } + .boxed() + }) + .await + .map_err(|err| err.into_indexed_storage_error(Self::classify_repo_error_general)) } async fn move_if_absent( diff --git a/golem-worker-executor/src/storage/indexed/redis.rs b/golem-worker-executor/src/storage/indexed/redis.rs index dca7d37236..44925d291b 100644 --- a/golem-worker-executor/src/storage/indexed/redis.rs +++ b/golem-worker-executor/src/storage/indexed/redis.rs @@ -14,7 +14,7 @@ use crate::storage::indexed::{ IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, - ScanResume, + ScanResume, WriterId, }; use async_trait::async_trait; use bytes::Bytes; @@ -23,19 +23,220 @@ use fred::prelude::{Key, Value}; use fred::types::config::Options; use fred::types::streams::XCapKind; use golem_common::metrics::redis::{record_redis_deserialized_size, record_redis_serialized_size}; +use golem_common::model::ShardEpoch; use golem_common::redis::{RedisError, RedisPool}; use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; +/// Redis checks a key's epoch in a Lua script, which it runs atomically with the `XADD`s the +/// script guards. Two limits follow from Redis itself rather than from this code: an epoch is +/// only as durable as the last write the server kept, so a failover that loses the tail can lose +/// a raised epoch and let the previous writer back in; and the script names two keys, which a +/// Redis Cluster would need in one slot - this backend does not support Cluster. #[derive(Debug)] pub struct RedisIndexedStorage { redis: RedisPool, + writer_id: WriterId, } impl RedisIndexedStorage { pub fn new(redis: RedisPool) -> Self { - Self { redis } + Self { + redis, + writer_id: WriterId::process(), + } + } + + /// The same store written as `writer_id`. See [`WriterId`] for why a process uses one value. + pub fn for_writer(mut self, writer_id: WriterId) -> Self { + self.writer_id = writer_id; + self + } + + /// `ARGV`: the asserted epoch, the writer, then `id, value` pairs. + /// + /// Numbers stay decimal strings throughout, because a Lua number is a double and loses + /// precision above 2^53; with no leading zeros they order by length and then lexically. The + /// ids are checked against the stream before the first `XADD` because a script is atomic but + /// not transactional: an `XADD` failing half way would leave the ones before it behind. + const FENCED_APPEND_SCRIPT: &'static str = r#" +local stored = redis.call('HMGET', KEYS[2], 'epoch', 'writer') +if stored[1] == false then + return redis.error_reply('FENCED - 0') +end +if stored[1] ~= ARGV[1] then + return redis.error_reply('FENCED ' .. stored[1] .. ' 0') +end +if stored[2] ~= ARGV[2] then + return redis.error_reply('FENCED ' .. stored[1] .. ' 1') +end +local top = nil +if redis.call('EXISTS', KEYS[1]) == 1 then + local info = redis.call('XINFO', 'STREAM', KEYS[1]) + for i = 1, #info, 2 do + if info[i] == 'last-generated-id' then + top = string.match(info[i + 1], '^(%d+)') + end + end +end +for i = 3, #ARGV, 2 do + local id = ARGV[i] + if top and not (#id > #top or (#id == #top and id > top)) then + return redis.error_reply('ERR The ID specified in XADD is equal or smaller than the target stream top item') + end + top = id +end +for i = 3, #ARGV, 2 do + redis.call('XADD', KEYS[1], ARGV[i], 'key', ARGV[i + 1]) +end +return redis.status_reply('OK') +"#; + + /// `ARGV`: the epoch and the writer, compared the way [`Self::FENCED_APPEND_SCRIPT`] does. + const SET_KEY_EPOCH_SCRIPT: &'static str = r#" +local stored = redis.call('HMGET', KEYS[1], 'epoch', 'writer') +local epoch = stored[1] +local higher = epoch ~= false and (#ARGV[1] > #epoch or (#ARGV[1] == #epoch and ARGV[1] > epoch)) +if epoch == false or higher or (epoch == ARGV[1] and stored[2] == ARGV[2]) then + redis.call('HSET', KEYS[1], 'epoch', ARGV[1], 'writer', ARGV[2]) + return redis.status_reply('OK') +end +if epoch == ARGV[1] then + return redis.error_reply('FENCED ' .. epoch .. ' 1') +end +return redis.error_reply('FENCED ' .. epoch .. ' 0') +"#; + + /// `KEYS`: the stream, then its epoch record. `ARGV`: the epoch and the writer the delete + /// asserts, compared the way [`Self::FENCED_APPEND_SCRIPT`] does, or nothing for an + /// unconditional delete. Both keys go in the one `DEL`, so a refused delete removes neither. + const DELETE_WITH_EPOCH_SCRIPT: &'static str = r#" +if #ARGV > 0 then + local stored = redis.call('HMGET', KEYS[2], 'epoch', 'writer') + if stored[1] == false then + if redis.call('EXISTS', KEYS[1]) == 0 then + return redis.status_reply('OK') + end + return redis.error_reply('FENCED - 0') + end + if stored[1] ~= ARGV[1] then + return redis.error_reply('FENCED ' .. stored[1] .. ' 0') + end + if stored[2] ~= ARGV[2] then + return redis.error_reply('FENCED ' .. stored[1] .. ' 1') + end +end +redis.call('DEL', KEYS[1], KEYS[2]) +return redis.status_reply('OK') +"#; + + /// Where a key's epoch lives. Not under the key's own name: `scan` matches `...oplog:*`, and + /// an `...oplog::epoch` sibling would come back from it as a key of its own. + fn epoch_key(namespace: IndexedStorageNamespace, key: &str) -> String { + match namespace { + IndexedStorageNamespace::OpLog { + agent_id: _, + agent_mode, + } => { + let mode = super::agent_mode_prefix(agent_mode); + format!("worker:{mode}:oplog-epoch:{key}") + } + IndexedStorageNamespace::CompressedOpLog { + agent_id: _, + agent_mode, + level, + } => { + let mode = super::agent_mode_prefix(agent_mode); + format!("worker:{mode}:c{level}-oplog-epoch:{key}") + } + // A stage is hidden and has one writer, so it never asserts an epoch; the key exists + // only to keep this total. + IndexedStorageNamespace::StagedOpLog { + agent_id: _, + agent_mode, + } => { + let mode = super::agent_mode_prefix(agent_mode); + format!("worker:{mode}:staged-oplog-epoch:{key}") + } + } + } + + /// The `FENCED ` reply of the scripts above. + fn parse_fenced( + error: &RedisError, + key: &str, + expected: ShardEpoch, + ) -> Option { + let mut parts = error + .details() + .split_whitespace() + .skip_while(|part| *part != "FENCED"); + parts.next()?; + let actual = match parts.next()? { + "-" => None, + epoch => Some(ShardEpoch(epoch.parse::().ok()?)), + }; + let writer_conflict = parts.next()? == "1"; + Some(IndexedStorageError::Fenced { + key: key.to_string(), + expected, + actual, + writer_conflict, + }) + } + + /// The error of [`IndexedStorage::set_key_epoch`] or [`IndexedStorage::delete_with_epoch`]: + /// the scripts' fence, or else classified as a read is. A lost connection or a timeout is + /// `Transient` even if the script ran, because both repeat safely for the same writer: the + /// epoch it already holds is accepted again, and a deletion that already happened finds + /// nothing left and succeeds. + fn classify_epoch_error( + error: RedisError, + key: &str, + expected: Option, + ) -> IndexedStorageError { + expected + .and_then(|expected| Self::parse_fenced(&error, key, expected)) + .unwrap_or_else(|| Self::classify_read_error(error)) + } + + async fn append_fenced( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: &IndexedStorageNamespace, + key: &str, + pairs: impl Iterator, + expected_epoch: ShardEpoch, + options: Option<&Options>, + primary_oplog_insert: bool, + ) -> Result<(), IndexedStorageError> { + let mut args = vec![ + Value::from(expected_epoch.0.to_string()), + Value::from(self.writer_id.to_string()), + ]; + for (id, value) in pairs { + args.push(Value::from(id.to_string())); + args.push(Value::Bytes(value)); + } + self.redis + .with(svc_name, api_name) + .eval( + Self::FENCED_APPEND_SCRIPT, + &[ + Self::composite_key(namespace.clone(), key), + Self::epoch_key(namespace.clone(), key), + ], + args, + options, + ) + .await + .map(|_| ()) + .map_err(|error| { + Self::parse_fenced(&error, key, expected_epoch) + .unwrap_or_else(|| Self::classify_append_error(error, primary_oplog_insert)) + }) } fn composite_key(namespace: IndexedStorageNamespace, key: &str) -> String { @@ -259,6 +460,7 @@ impl IndexedStorage for RedisIndexedStorage { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { record_redis_serialized_size(svc_name, entity_name, value.len()); let primary_oplog_insert = matches!( @@ -270,6 +472,21 @@ impl IndexedStorage for RedisIndexedStorage { ..Default::default() }); + if let Some(expected_epoch) = expected_epoch { + return self + .append_fenced( + svc_name, + api_name, + &namespace, + key, + std::iter::once((id, Bytes::from(value))), + expected_epoch, + options.as_ref(), + primary_oplog_insert, + ) + .await; + } + let _: String = self .redis .with(svc_name, api_name) @@ -294,6 +511,7 @@ impl IndexedStorage for RedisIndexedStorage { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { if !pairs.is_empty() { let primary_oplog_insert = matches!( @@ -304,6 +522,24 @@ impl IndexedStorage for RedisIndexedStorage { max_attempts: Some(1), ..Default::default() }); + + if let Some(expected_epoch) = expected_epoch { + for (_, value) in pairs.iter() { + record_redis_serialized_size(svc_name, entity_name, value.len()); + } + return self + .append_fenced( + svc_name, + api_name, + namespace, + key, + pairs.iter().cloned(), + expected_epoch, + options.as_ref(), + primary_oplog_insert, + ) + .await; + } let mut redis_pairs = Vec::with_capacity(pairs.len()); for (id, value) in pairs.iter() { record_redis_serialized_size(svc_name, entity_name, value.len()); @@ -328,6 +564,61 @@ impl IndexedStorage for RedisIndexedStorage { Ok(()) } + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + self.redis + .with(svc_name, api_name) + .eval( + Self::SET_KEY_EPOCH_SCRIPT, + &[Self::epoch_key(namespace, key)], + vec![ + Value::from(epoch.0.to_string()), + Value::from(self.writer_id.to_string()), + ], + None, + ) + .await + .map(|_| ()) + .map_err(|error| Self::classify_epoch_error(error, key, Some(epoch))) + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + let args = match expected_epoch { + Some(expected) => vec![ + Value::from(expected.0.to_string()), + Value::from(self.writer_id.to_string()), + ], + None => vec![], + }; + self.redis + .with(svc_name, api_name) + .eval( + Self::DELETE_WITH_EPOCH_SCRIPT, + &[ + Self::composite_key(namespace.clone(), key), + Self::epoch_key(namespace, key), + ], + args, + None, + ) + .await + .map(|_| ()) + .map_err(|error| Self::classify_epoch_error(error, key, expected_epoch)) + } + async fn move_if_absent( &self, svc_name: &'static str, @@ -503,6 +794,49 @@ mod tests { use super::*; use test_r::test; + #[test] + fn a_fenced_reply_carries_the_stored_epoch_and_the_writer_conflict() { + let fenced = |details: &'static str| { + RedisIndexedStorage::parse_fenced( + &RedisError::new(ErrorKind::Unknown, details), + "k", + ShardEpoch(7), + ) + }; + + assert!(matches!( + fenced("FENCED 9 0"), + Some(IndexedStorageError::Fenced { + actual: Some(ShardEpoch(9)), + writer_conflict: false, + .. + }) + )); + assert!(matches!( + fenced("FENCED 7 1"), + Some(IndexedStorageError::Fenced { + actual: Some(ShardEpoch(7)), + writer_conflict: true, + .. + }) + )); + assert!(matches!( + fenced("FENCED - 0"), + Some(IndexedStorageError::Fenced { + actual: None, + writer_conflict: false, + .. + }) + )); + // An error raised by a command inside the script is not a fence. + assert!( + fenced( + "ERR The ID specified in XADD is equal or smaller than the target stream top item" + ) + .is_none() + ); + } + #[test] fn primary_oplog_xadd_ordering_error_is_a_conflict() { let error = RedisError::new( @@ -560,4 +894,40 @@ mod tests { )); } } + + // The oplog retries only `Transient` around these two and panics on anything else but a + // fence, which SQLite and PostgreSQL never hand it for a lost connection. + #[test] + fn epoch_record_connection_errors_are_transient() { + for expected in [Some(ShardEpoch(7)), None] { + for kind in [ErrorKind::IO, ErrorKind::Timeout, ErrorKind::Canceled] { + let error = RedisError::new(kind, "outcome is unknown"); + assert!(matches!( + RedisIndexedStorage::classify_epoch_error(error, "k", expected), + IndexedStorageError::Transient(_) + )); + } + } + } + + #[test] + fn epoch_record_fence_is_a_fence_and_other_errors_stay_permanent() { + let fenced = RedisError::new(ErrorKind::Unknown, "FENCED 9 0"); + assert!(matches!( + RedisIndexedStorage::classify_epoch_error(fenced, "k", Some(ShardEpoch(7))), + IndexedStorageError::Fenced { + actual: Some(ShardEpoch(9)), + .. + } + )); + + let wrong_type = RedisError::new( + ErrorKind::Unknown, + "WRONGTYPE Operation against a key holding the wrong kind of value", + ); + assert!(matches!( + RedisIndexedStorage::classify_epoch_error(wrong_type, "k", Some(ShardEpoch(7))), + IndexedStorageError::Other(_) + )); + } } diff --git a/golem-worker-executor/src/storage/indexed/sqlite.rs b/golem-worker-executor/src/storage/indexed/sqlite.rs index 01c5c15f38..dc6aa8638f 100644 --- a/golem-worker-executor/src/storage/indexed/sqlite.rs +++ b/golem-worker-executor/src/storage/indexed/sqlite.rs @@ -13,8 +13,8 @@ // limitations under the License. use super::{ - IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, IndexedStorageNamespace, - ScanResume, + FencedTxError, IndexedStorage, IndexedStorageError, IndexedStorageMetaNamespace, + IndexedStorageNamespace, ScanResume, WriterId, }; use async_trait::async_trait; use bytes::Bytes; @@ -22,6 +22,7 @@ use futures::FutureExt; use golem_common::SafeDisplay; use golem_common::config::DbSqliteConfig; use golem_common::metrics::db::record_db_serialized_size; +use golem_common::model::ShardEpoch; use golem_service_base::db::sqlite::SqlitePool; use golem_service_base::db::{Pool, PoolApi}; use golem_service_base::migration::{IncludedMigrationsDir, Migrations}; @@ -43,6 +44,9 @@ static DB_MIGRATIONS: include_dir::Dir = include_dir!("$CARGO_MANIFEST_DIR/db/mi #[derive(Debug, Clone)] pub struct SqliteIndexedStorage { pool: SqlitePool, + /// Recorded beside the epoch on every key this process claims, so an equal epoch from + /// another process is refused rather than shared. One per process; see [`WriterId`]. + writer_id: WriterId, } impl SqliteIndexedStorage { @@ -56,7 +60,18 @@ impl SqliteIndexedStorage { ) })?; - Ok(Self { pool }) + Ok(Self { + pool, + writer_id: WriterId::process(), + }) + } + + /// Writes as `writer_id` rather than as this process's own. The fan-out backend uses it to + /// give every storage it opens one identity, and a test uses it to play two processes racing + /// over one key inside a single process. + pub fn for_writer(mut self, writer_id: WriterId) -> Self { + self.writer_id = writer_id; + self } /// Apply the indexed storage migrations on the given sqlite config without @@ -74,7 +89,10 @@ impl SqliteIndexedStorage { } pub fn new(pool: SqlitePool) -> Self { - Self { pool } + Self { + pool, + writer_id: WriterId::process(), + } } fn namespace(namespace: IndexedStorageNamespace) -> String { @@ -117,6 +135,26 @@ impl SqliteIndexedStorage { } } + /// sqlx has no `Encode` for `u64`, so a value that must stay integer-bound (rather + /// than go through `Json`, which encodes as TEXT - see [`Self::set_key_epoch`]) has to + /// cross to `i64` first. Checked, like Postgres's own `to_i64`: an unchecked `as i64` on a + /// value above `i64::MAX` wraps to negative, and reading that back `as u64` produces a + /// spuriously huge epoch instead of failing loudly. + fn to_i64(value: u64, field_name: &'static str) -> Result { + i64::try_from(value).map_err(|_| { + IndexedStorageError::Other(format!( + "SQLite indexed storage cannot represent {field_name}={value} as i64" + )) + }) + } + + /// A stored epoch that will not fit a `u64` is corruption, not a fence: `to_i64` refuses to + /// write one, so a negative column value came from outside this code, and reading it back as + /// `u64` would wrap it into a spuriously huge epoch. + fn negative_epoch_message(value: i64, key: &str) -> String { + format!("SQLite indexed storage read a negative epoch {value} for key '{key}'") + } + fn classify_repo_error(err: RepoError) -> IndexedStorageError { if err.is_transient() { IndexedStorageError::Transient(err.to_string()) @@ -228,6 +266,8 @@ impl IndexedStorage for SqliteIndexedStorage { Ok((super::last_key_resume(&keys, count), keys)) } + /// Delegates to [`Self::append_many`] so there is exactly one fenced write path: the epoch + /// check has to happen in the same transaction as the insert. async fn append( &self, svc_name: &'static str, @@ -237,34 +277,18 @@ impl IndexedStorage for SqliteIndexedStorage { key: &str, id: u64, value: Vec, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { - record_db_serialized_size(DB_TYPE, svc_name, entity_name, value.len()); - let primary_oplog_insert = matches!( + self.append_many( + svc_name, + api_name, + entity_name, &namespace, - IndexedStorageNamespace::OpLog { .. } | IndexedStorageNamespace::StagedOpLog { .. } - ); - let query = sqlx::query( - r#" - INSERT INTO index_storage (namespace, key, id, value) VALUES (?,?,?,?); - "#, + key, + vec![(id, Bytes::from(value))].into(), + expected_epoch, ) - .bind(Self::namespace(namespace)) - .bind(key) - .bind(sqlx::types::Json(id)) - .bind(value); - - self.pool - .with_rw(svc_name, api_name) - .execute(query) - .await - .map(|_| ()) - .map_err(|err| { - if primary_oplog_insert { - Self::classify_repo_error_primary_oplog_insert(err) - } else { - Self::classify_repo_error(err) - } - }) + .await } async fn append_many( @@ -275,6 +299,7 @@ impl IndexedStorage for SqliteIndexedStorage { namespace: &IndexedStorageNamespace, key: &str, pairs: Arc<[(u64, Bytes)]>, + expected_epoch: Option, ) -> Result<(), IndexedStorageError> { if pairs.is_empty() { return Ok(()); @@ -290,9 +315,39 @@ impl IndexedStorage for SqliteIndexedStorage { record_db_serialized_size(DB_TYPE, svc_name, entity_name, value.len()); } + let writer_id = self.writer_id.to_string(); self.pool - .with_tx(svc_name, api_name, |tx| { - async move { + .with_tx_err::<(), FencedTxError, _>(svc_name, api_name, |tx| { + Box::pin(async move { + // SQLite has no `SELECT ... FOR UPDATE`, and it does not need one here: the + // write pool is capped at a single connection (golem-service-base + // db/sqlite.rs:46-50), so this transaction holds the only writer and the + // check cannot be interleaved. Raising that cap means switching this to + // `BEGIN IMMEDIATE`. + // + // That holds within one process. Two processes on one SQLite file would rely + // on SQLite's own lock upgrade, which surfaces a loser as `SQLITE_BUSY` - a + // storage error, not a fence - so a SQLite file shared between executors is + // not supported; give each its own file, or use PostgreSQL. + if let Some(expected) = expected_epoch { + let stored: Option<(i64, String)> = tx + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = ? AND key = ?;", + ) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + FencedTxError::check_record( + &key, + expected, + stored, + &writer_id, + Self::negative_epoch_message, + )?; + } + for (id, value) in pairs.iter() { tx.execute( sqlx::query( @@ -307,17 +362,161 @@ impl IndexedStorage for SqliteIndexedStorage { } Ok(()) - } - .boxed() + }) }) .await .map_err(|err| { - if primary_oplog_insert { - Self::classify_repo_error_primary_oplog_insert(err) + err.into_indexed_storage_error(if primary_oplog_insert { + Self::classify_repo_error_primary_oplog_insert } else { - Self::classify_repo_error(err) - } + Self::classify_repo_error + }) + }) + } + + /// SQLite's half of [`IndexedStorage::set_key_epoch`], which states the rule this + /// enforces. The unqualified `epoch`/`writer` in the `WHERE` are the existing row's, and + /// `excluded` is the row being written. + async fn set_key_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + new_epoch: ShardEpoch, + ) -> Result<(), IndexedStorageError> { + let namespace = Self::namespace(namespace); + // `i64`, not `u64`: sqlx has no `Encode` for `u64`, which is why ids elsewhere in + // this file go through `Json`. That encodes as TEXT, and comparison affinity is applied + // per operand, so this column stays integer-bound everywhere. Checked (see `to_i64`) + // rather than `as i64`, which would silently wrap an out-of-range epoch to negative. + let epoch = Self::to_i64(new_epoch.0, "epoch")?; + + let writer_id = self.writer_id.to_string(); + + let mut api = self.pool.with_rw(svc_name, api_name); + let result = api + .execute( + sqlx::query( + r#"INSERT INTO indexed_key_epoch (namespace, key, epoch, writer) VALUES (?, ?, ?, ?) + ON CONFLICT(namespace, key) DO UPDATE SET epoch = excluded.epoch, writer = excluded.writer + WHERE epoch < excluded.epoch + OR (epoch = excluded.epoch AND writer = excluded.writer);"#, + ) + .bind(namespace.clone()) + .bind(key) + .bind(epoch) + .bind(writer_id.clone()), + ) + .await + .map_err(Self::classify_repo_error)?; + + if result.rows_affected() == 0 { + let stored: Option<(i64, String)> = api + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = ? AND key = ?;", + ) + .bind(namespace) + .bind(key), + ) + .await + .map_err(Self::classify_repo_error)?; + let mut actual = None; + let mut writer_matches = false; + if let Some((epoch, writer)) = stored { + let epoch = u64::try_from(epoch).map_err(|_| { + IndexedStorageError::Other(Self::negative_epoch_message(epoch, key)) + })?; + actual = Some(ShardEpoch(epoch)); + writer_matches = writer == writer_id; + } + return Err(IndexedStorageError::Fenced { + key: key.to_string(), + expected: new_epoch, + actual, + writer_conflict: actual == Some(new_epoch) && !writer_matches, + }); + } + + Ok(()) + } + + async fn delete_with_epoch( + &self, + svc_name: &'static str, + api_name: &'static str, + namespace: IndexedStorageNamespace, + key: &str, + expected_epoch: Option, + ) -> Result<(), IndexedStorageError> { + let namespace = Self::namespace(namespace); + let key = key.to_string(); + let writer_id = self.writer_id.to_string(); + self.pool + .with_tx_err::<(), FencedTxError, _>(svc_name, api_name, |tx| { + Box::pin(async move { + // The single-connection write pool makes the check and the deletes one + // step, as it does for an append (see `append_many`). + if let Some(expected) = expected_epoch { + let stored: Option<(i64, String)> = tx + .fetch_optional_as( + sqlx::query_as( + "SELECT epoch, writer FROM indexed_key_epoch WHERE namespace = ? AND key = ?;", + ) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + // Neither a record nor entries: the key is already gone, most often taken + // by an earlier attempt of this same deletion whose later steps failed, and + // a writer that lost the key has nothing here to destroy. Refusing would + // leave that deletion unable to finish. + if stored.is_none() { + let (has_entries,): (bool,) = tx + .fetch_one_as( + sqlx::query_as( + "SELECT EXISTS(SELECT 1 FROM index_storage WHERE namespace IN (?, ?) AND key = ?);", + ) + .bind(format!("{namespace}-present")) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + if !has_entries { + return Ok(()); + } + } + FencedTxError::check_record( + &key, + expected, + stored, + &writer_id, + Self::negative_epoch_message, + )?; + } + tx.execute( + sqlx::query( + "DELETE FROM index_storage WHERE namespace IN (?, ?) AND key = ?;", + ) + .bind(format!("{namespace}-present")) + .bind(namespace.clone()) + .bind(key.clone()), + ) + .await?; + tx.execute( + sqlx::query( + "DELETE FROM indexed_key_epoch WHERE namespace = ? AND key = ?;", + ) + .bind(namespace) + .bind(key), + ) + .await?; + Ok(()) + }) }) + .await + .map_err(|err| err.into_indexed_storage_error(Self::classify_repo_error)) } async fn move_if_absent( @@ -655,6 +854,7 @@ mod tests { &stage, 1, vec![index as u8], + None, ) .await .unwrap(); @@ -725,6 +925,7 @@ mod tests { (2, Bytes::from_static(b"second")), ] .into(), + None, ) .await .unwrap(); @@ -762,6 +963,7 @@ mod tests { "oplog", 2, b"existing".to_vec(), + None, ) .await .unwrap(); @@ -778,6 +980,7 @@ mod tests { (2, Bytes::from_static(b"conflict")), ] .into(), + None, ) .await; @@ -791,6 +994,65 @@ mod tests { ); } + #[test] + // The column is `i64`-bound (see `to_i64`'s doc). An epoch that does not fit it must be + // rejected here rather than silently wrapped to a negative value that a later `epoch as u64` + // read turns into a spuriously huge one. + async fn set_key_epoch_rejects_an_epoch_that_does_not_fit_i64() { + let tempdir = tempfile::tempdir().unwrap(); + let storage = sqlite_storage( + tempdir + .path() + .join("indexed.db") + .to_string_lossy() + .into_owned(), + ) + .await; + let namespace = oplog_namespace("sqlite-epoch-overflow"); + + let result = storage + .set_key_epoch( + "test", + "set_key_epoch", + namespace, + "oplog", + ShardEpoch(u64::MAX), + ) + .await; + + assert!( + matches!(result, Err(IndexedStorageError::Other(_))), + "an epoch above i64::MAX must be a rejected write, not a wrapped negative one, got {result:?}" + ); + } + + #[test] + // The largest value that does fit is the boundary right below the rejected one, and must + // still succeed - a regression here would mean the checked conversion rejects valid input. + async fn set_key_epoch_accepts_the_largest_epoch_that_fits_i64() { + let tempdir = tempfile::tempdir().unwrap(); + let storage = sqlite_storage( + tempdir + .path() + .join("indexed.db") + .to_string_lossy() + .into_owned(), + ) + .await; + let namespace = oplog_namespace("sqlite-epoch-boundary"); + + storage + .set_key_epoch( + "test", + "set_key_epoch", + namespace, + "oplog", + ShardEpoch(i64::MAX as u64), + ) + .await + .unwrap(); + } + #[test] async fn bounded_scan_uses_binary_covering_index_range() { let tempdir = tempfile::tempdir().unwrap(); diff --git a/golem-worker-executor/src/worker/durable_stream_producer/tests.rs b/golem-worker-executor/src/worker/durable_stream_producer/tests.rs index b7f0a35546..7318d5b85d 100644 --- a/golem-worker-executor/src/worker/durable_stream_producer/tests.rs +++ b/golem-worker-executor/src/worker/durable_stream_producer/tests.rs @@ -185,12 +185,13 @@ async fn shutdown_fences_empty_and_idle_slots_without_flushing_buffered_entries( timestamp: golem_common::model::Timestamp::now_utc(), entity_parent_start_index: None, }) - .await; + .await + .unwrap(); let shutdown = slot.shutdown(); assert!(slot.is_retired()); shutdown.await.unwrap(); slot.retire(unused_commit()).await.unwrap(); - assert_eq!(oplog.commit(CommitLevel::Always).await.len(), 1); + assert_eq!(oplog.commit(CommitLevel::Always).await.unwrap().len(), 1); } } @@ -226,7 +227,7 @@ async fn shutdown_waits_for_admitted_commit_tail_after_cancelled_waiter() { let reached = reached.clone(); let release = release.clone(); Box::pin(async move { - oplog.commit(CommitLevel::Always).await; + oplog.commit(CommitLevel::Always).await.unwrap(); receipt.unwrap().send(()).unwrap(); reached.notify_one(); release.acquire().await.unwrap().forget(); diff --git a/golem-worker-executor/src/worker/durable_stream_slots.rs b/golem-worker-executor/src/worker/durable_stream_slots.rs index 1e87894c04..cbc55b8014 100644 --- a/golem-worker-executor/src/worker/durable_stream_slots.rs +++ b/golem-worker-executor/src/worker/durable_stream_slots.rs @@ -254,7 +254,8 @@ fn append_error(error: StreamStoreError) -> WorkerExecutorError { | StreamStoreError::UnknownStream(_) => { WorkerExecutorError::invalid_request(error.to_string()) } - _ => WorkerExecutorError::runtime(error.to_string()), + // A refused write keeps its type, so the caller is sent to the shard's new owner. + error => error.into_worker_executor_error(WorkerExecutorError::runtime), } } diff --git a/golem-worker-executor/src/worker/instance.rs b/golem-worker-executor/src/worker/instance.rs index 84d4c56096..9aa2b1d65f 100644 --- a/golem-worker-executor/src/worker/instance.rs +++ b/golem-worker-executor/src/worker/instance.rs @@ -21,7 +21,7 @@ use crate::durable_host::tool::operation::{DeferredAdmissionTable, OwnerToolOper use crate::model::ExecutionStatus; use crate::services::active_agents::WorkerComponentCharge; use crate::services::agent_filesystem::FilesystemGenerationHandle; -use crate::services::oplog::{CommitLevel, Oplog}; +use crate::services::oplog::{CommitLevel, Oplog, OplogFence}; use crate::services::resource_limits::AtomicResourceEntry; use crate::services::{HasActiveAgents, HasComponentService, HasWasmtimeEngine}; use crate::workerctx::WorkerCtx; @@ -129,6 +129,10 @@ pub struct OwnerExecution { wall_clock_now_gate: Mutex>>, #[cfg(feature = "test-utils")] skip_wall_clock_now_durability: AtomicBool, + /// Pauses an invocation between buffering its `AgentInvocationStarted` and committing it, so + /// a test can act while the entry is in the buffer and nothing has queried storage yet. + #[cfg(feature = "test-utils")] + invocation_started_gate: Mutex>>, } #[cfg(feature = "test-utils")] @@ -193,6 +197,8 @@ impl OwnerExecution { wall_clock_now_gate: Mutex::new(None), #[cfg(feature = "test-utils")] skip_wall_clock_now_durability: AtomicBool::new(false), + #[cfg(feature = "test-utils")] + invocation_started_gate: Mutex::new(None), } } @@ -356,7 +362,16 @@ impl OwnerExecution { pub(crate) async fn test_after_monotonic_clock_start(&self) -> Result<(), InterruptKind> { let gate = self.monotonic_clock_start_gate.lock().unwrap().take(); if let Some(gate) = gate { - self.oplog.commit(CommitLevel::Always).await; + // The gate is entered from inside a host call, where the fence surfaces as the + // interrupt that gives the agent up; a transient storage failure is fatal here as + // everywhere else. + match self.oplog.commit(CommitLevel::Always).await { + Ok(_) => {} + Err(crate::services::oplog::OplogError::Fenced(_)) => { + return Err(InterruptKind::ShardLost); + } + Err(error) => panic!("oplog write: {error}"), + } if let Some(entered) = gate.entered.lock().unwrap().take() { let _ = entered.send(()); } @@ -372,6 +387,34 @@ impl OwnerExecution { Ok(()) } + #[cfg(feature = "test-utils")] + #[doc(hidden)] + pub fn test_gate_next_invocation_started(&self) -> ClockNowGateHandle { + let (entered_tx, entered) = tokio::sync::oneshot::channel(); + let gate = Arc::new(ClockNowGate { + entered: Mutex::new(Some(entered_tx)), + release: tokio::sync::Semaphore::new(0), + abort_as_restart: AtomicBool::new(false), + }); + *self.invocation_started_gate.lock().unwrap() = Some(gate.clone()); + ClockNowGateHandle { entered, gate } + } + + #[cfg(feature = "test-utils")] + pub(crate) async fn test_after_invocation_started_buffered(&self) { + let gate = self.invocation_started_gate.lock().unwrap().take(); + if let Some(gate) = gate { + if let Some(entered) = gate.entered.lock().unwrap().take() { + let _ = entered.send(()); + } + gate.release + .acquire() + .await + .expect("invocation started gate was closed") + .forget(); + } + } + #[cfg(feature = "test-utils")] #[doc(hidden)] pub fn test_gate_next_wall_clock_now(&self) -> ClockNowGateHandle { @@ -413,14 +456,11 @@ impl OwnerExecution { } } - pub async fn commit(&self, level: CommitLevel) -> OplogIndex { - self.commit.commit_and_update_state(level).await.0 - } - - pub async fn add_and_commit(&self, entry: OplogEntry) -> OplogIndex { - let index = self.oplog.add(entry).await; - self.commit(CommitLevel::Always).await; - index + pub async fn commit(&self, level: CommitLevel) -> Result { + self.commit + .commit_and_update_state(level) + .await + .map(|(index, _)| index) } } @@ -727,7 +767,7 @@ impl InstanceHost { // process crash can replay and persist the same instantiation growth again. owner .add_and_commit_oplog(OplogEntry::grow_memory(live_instantiation_growth)) - .await; + .await?; owner .startup_linear_memory_bytes .store(allocated_bytes, Ordering::Release); diff --git a/golem-worker-executor/src/worker/invocation.rs b/golem-worker-executor/src/worker/invocation.rs index 1af5ffa2f5..9ee6eb42ae 100644 --- a/golem-worker-executor/src/worker/invocation.rs +++ b/golem-worker-executor/src/worker/invocation.rs @@ -120,6 +120,13 @@ pub async fn invoke_observed_and_traced( record_invocation(was_live_before, "suspended"); result } + Ok(InvokeResult::Interrupted { + interrupt_kind: InterruptKind::ShardLost, + .. + }) => { + record_invocation(was_live_before, "shard_lost"); + result + } Ok(InvokeResult::Interrupted { .. }) => { record_invocation(was_live_before, "restarted"); result diff --git a/golem-worker-executor/src/worker/invocation_loop.rs b/golem-worker-executor/src/worker/invocation_loop.rs index e861b208c4..510e56d62e 100644 --- a/golem-worker-executor/src/worker/invocation_loop.rs +++ b/golem-worker-executor/src/worker/invocation_loop.rs @@ -32,7 +32,7 @@ use crate::worker::invocation::{ }; use crate::worker::status_checkpointer; use crate::worker::{ - CreateWorkerInstanceError, FinalWorkerState, PendingLiveInvocationDisposition, + CreateWorkerInstanceError, FinalWorkerState, GiveUpReason, PendingLiveInvocationDisposition, PendingWorkerInterrupt, QueuedWorkerInvocation, RetryDecision, RunningAgent, RunningAgentRuntime, RunningWorker, UnloadReason, UnloadRequest, Worker, WorkerCommand, WorkerInterruptState, WorkerRunningAgent, WorkerTrace, @@ -261,6 +261,13 @@ impl InvocationLoop { .await .retain(|invocation| !invocation.is_abandoned()); self.release_terminal_interrupt().await; + // Never a new instance for an agent given up here, whichever path led back to this + // point: it would reopen the oplog at an epoch this executor no longer holds. + if self.parent.is_given_up() { + self.release_concurrent_agent_permit(); + self.stop_startup_given_up().await; + break; + } // ADMISSION: gates the start of a generation, so // fencing refuses new generations and never interrupts a running one. if let Err(error) = self.parent.shard_service().check_admission(&agent_id) { @@ -361,9 +368,17 @@ impl InvocationLoop { { // Core initialization has already entered the executable Store. Losing it // is terminal for an external owner, just as losing its invocation body is. - self.parent + if self + .parent .add_and_commit_oplog(OplogEntry::interrupted()) - .await; + .await + .is_err() + { + // The shard has a new owner. Give the agent up without archiving: + // the archive would move an oplog that is no longer this executor's. + self.stop_startup_given_up().await; + break; + } self.stop_unloaded( Some(super::inactive_ephemeral_agent_error()), PendingLiveInvocationDisposition::Fail, @@ -382,9 +397,15 @@ impl InvocationLoop { continue; } InterruptKind::Suspend(ts) => { - self.parent + if self + .parent .add_and_commit_oplog(OplogEntry::suspend()) - .await; + .await + .is_err() + { + self.stop_startup_given_up().await; + break; + } if ts < *self.parent.last_resume_request.lock().await { debug!( "Suspend during instantiation ignored because there was a resume request since it" @@ -405,9 +426,15 @@ impl InvocationLoop { } } InterruptKind::Interrupt(_) => { - self.parent + if self + .parent .add_and_commit_oplog(OplogEntry::interrupted()) - .await; + .await + .is_err() + { + self.stop_startup_given_up().await; + break; + } self.parent.complete_startup( self.start_attempt, Err(WorkerExecutorError::Interrupted { kind }), @@ -416,6 +443,12 @@ impl InvocationLoop { .await; break; } + InterruptKind::ShardLost => { + // Nothing is written: the oplog belongs to the shard's new owner + // now. Whoever was waiting for this start is told to look there. + self.stop_startup_given_up().await; + break; + } } } CreateInstanceResult::Failed => { @@ -577,7 +610,10 @@ impl InvocationLoop { Some(unloading.cleanup.clone()); let cleanup_failure = finish_filesystem_limit_unload(suspend, unloading, || async { - self.parent + // A refusal marks the agent given up, which the stop + // below acts on; there is nothing else to undo. + let _ = self + .parent .add_and_commit_oplog(OplogEntry::suspend()) .await; }) @@ -690,18 +726,11 @@ impl InvocationLoop { .try_get_active_agent(&self.owned_agent_id) .await { - let owner_failure = final_interrupt - .map(OwnerFailureWinner::Lifecycle) - .or_else(|| { - recovery_failure - .clone() - .map(OwnerFailureWinner::Infrastructure) - }) - .unwrap_or_else(|| { - OwnerFailureWinner::Lifecycle( - InterruptKind::Interrupt(Timestamp::now_utc()), - ) - }); + let owner_failure = exit_owner_failure( + self.parent.given_up_owner_failure(), + final_interrupt, + recovery_failure.as_ref(), + ); active_agent.fence_entity_bodies(owner_failure).await; } // Tests can shorten the deadline and pause filesystem cleanup to exercise late @@ -730,6 +759,21 @@ impl InvocationLoop { break; } + // Whatever was decided, an agent given up here is not restarted, retried later or + // parked for a resume on this executor: the shard's new owner resumes it. + if self.parent.is_given_up() { + debug!( + %agent_id, + ?final_decision, + "Invocation queue loop stopping an agent this executor has given up" + ); + self.stop_startup_given_up().await; + if cleanup_ephemeral_worker { + self.archive_ephemeral_oplog(); + } + break; + } + match final_decision { None | Some(RetryDecision::None) => { debug!( @@ -809,14 +853,31 @@ impl InvocationLoop { .await .current_idempotency_key .clone(); - match kind { + // Given up, before or by this interrupt: the oplog is + // the new owner's to write, so no lifecycle entry and no + // failure is recorded for an invocation it runs. + if matches!(kind, InterruptKind::ShardLost) + || self.parent.is_given_up() + { + self.stop_startup_given_up().await; + break 'outer; + } + let recorded = match kind { InterruptKind::Suspend(_) => { - self.parent.add_and_commit_oplog(OplogEntry::suspend()).await; + self.parent.add_and_commit_oplog(OplogEntry::suspend()).await.map(|_| ()) } InterruptKind::Interrupt(_) => { - self.parent.add_and_commit_oplog(OplogEntry::interrupted()).await; + self.parent.add_and_commit_oplog(OplogEntry::interrupted()).await.map(|_| ()) } - InterruptKind::Restart | InterruptKind::Jump => {} + InterruptKind::Restart + | InterruptKind::Jump + | InterruptKind::ShardLost => Ok(()), + }; + // Refused, the agent has been given up: nothing restarts + // in place. + if recorded.is_err() { + self.stop_startup_given_up().await; + break 'outer; } if matches!(kind, InterruptKind::Interrupt(_)) && let Some(key) = current_idempotency_key @@ -917,6 +978,21 @@ impl InvocationLoop { self.permit_state.release(); } + /// Stops a generation this executor has given up, because its shard was lost or its oplog + /// refused a lifecycle entry: the waiters are told to look for the shard's new owner. + /// + /// A fence found by a host call during instantiation arrives without `give_up()` having + /// run, so the agent is marked given up here: the stop then tears its entity bodies down as + /// `ShardLost`, fails its waiters and removes only this generation. A reason already recorded + /// is kept. + async fn stop_startup_given_up(&self) { + self.parent.mark_given_up(GiveUpReason::Fenced(None)); + self.stop_unloaded(None, PendingLiveInvocationDisposition::Fail) + .await; + } + + /// Handles an interrupt that arrived while the loop waits, unloaded, for a concurrent-agent + /// permit. Returns whether the loop exits. async fn handle_unloaded_interrupt( &self, interrupt: PendingWorkerInterrupt, @@ -928,6 +1004,13 @@ impl InvocationLoop { ?decision, "Invocation queue loop interrupted while unloaded" ); + // Given up, before or by this interrupt: the oplog is the new owner's to write, so no + // lifecycle entry and no failure is recorded for an invocation it runs, and nothing waits + // on here for a permit to restart it. + if matches!(kind, InterruptKind::ShardLost) || self.parent.is_given_up() { + self.stop_startup_given_up().await; + return true; + } if !matches!(kind, InterruptKind::Restart | InterruptKind::Jump) { let current_idempotency_key = self .parent @@ -935,18 +1018,24 @@ impl InvocationLoop { .await .current_idempotency_key .clone(); - match kind { - InterruptKind::Suspend(_) => { - self.parent - .add_and_commit_oplog(OplogEntry::suspend()) - .await; - } - InterruptKind::Interrupt(_) => { - self.parent - .add_and_commit_oplog(OplogEntry::interrupted()) - .await; - } - InterruptKind::Restart | InterruptKind::Jump => {} + let recorded = match kind { + InterruptKind::Suspend(_) => self + .parent + .add_and_commit_oplog(OplogEntry::suspend()) + .await + .map(|_| ()), + InterruptKind::Interrupt(_) => self + .parent + .add_and_commit_oplog(OplogEntry::interrupted()) + .await + .map(|_| ()), + InterruptKind::Restart | InterruptKind::Jump | InterruptKind::ShardLost => Ok(()), + }; + // Refused, the agent has been given up: no failure is cached for the invocation the + // shard's new owner resumes, and nothing restarts in place. + if recorded.is_err() { + self.stop_startup_given_up().await; + return true; } if matches!(kind, InterruptKind::Interrupt(_)) && let Some(key) = current_idempotency_key @@ -993,6 +1082,14 @@ impl InvocationLoop { startup_failure: Option, pending_live_invocations: PendingLiveInvocationDisposition, ) { + // A generation this executor has given up keeps the retry answer as its startup failure, + // like `Worker::give_up` does, whichever exit stopped it: otherwise a readiness waiter + // resolved by the stop, or a handle kept past this generation, is told it may proceed. + let startup_failure = if self.parent.is_given_up() { + Some(self.parent.give_up_error()) + } else { + startup_failure + }; self.parent.complete_startup( self.start_attempt, Err(startup_failure.clone().unwrap_or_else(|| { @@ -1005,9 +1102,10 @@ impl InvocationLoop { .try_get_active_agent(&self.owned_agent_id) .await { - let failure = startup_failure.clone().map_or_else( - || OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt(Timestamp::now_utc())), - OwnerFailureWinner::Infrastructure, + let failure = exit_owner_failure( + self.parent.given_up_owner_failure(), + None, + startup_failure.as_ref(), ); active_agent.fence_entity_bodies(failure).await; } @@ -1212,6 +1310,13 @@ impl InvocationLoop { self.parent.record_recovery_failure(&err).await; err }; + // A generation given up keeps the error that sends its callers to the shard's + // new owner, whatever failed on the way out. + let err = if self.parent.is_given_up() { + self.parent.give_up_error() + } else { + err + }; self.parent .complete_startup(self.start_attempt, Err(err.clone())); let final_state = if let Some(failure) = filesystem_cleanup_failure { @@ -1250,9 +1355,15 @@ impl InvocationLoop { // Making sure all pending commits are flushed // Make sure all pending commits are done let worker = store.lock().await.data().get_public_state().worker(); - worker + // A failed commit is a refused one, and gives the agent up. + let _ = worker .commit_oplog_and_update_state(CommitLevel::Always) .await; + if worker.is_given_up() { + // Given up, by that commit or by a revoke or reassignment that latched no fence: + // its status is not this executor's to persist any more. + return; + } // The worker is going idle; persist its cached status synchronously now instead of leaving // it for the next background sweep, so reads of an idle worker see an up-to-date blob. @@ -1633,6 +1744,13 @@ impl InnerInvocationLoop<'_, Ctx> { break self.interrupt(interrupt).await; } + // Given up by a path that queues no interrupt, such as a write refused + // outside the guest. Nothing more is taken from the queue: the shard's + // new owner runs it. + if self.parent.is_given_up() { + break CommandOutcome::BreakInnerLoop(RetryDecision::None); + } + let result = match self.select_next_work().await { SelectedWork::Resident(message) => { self.internal_invocation(message).await @@ -2449,6 +2567,11 @@ impl Invocation<'_, Ctx> { /// or a manual update request (which involves invoking the exported save-snapshot functions, so /// it is a special case of the exported function invocation). async fn external_invocation(&mut self, inner: TimestampedAgentInvocation) -> CommandOutcome { + // Rechecked here as well as where the invocation was taken: hydrating it and waiting for + // the store both leave room for the agent to be given up in between. + if self.parent.is_given_up() { + return CommandOutcome::BreakInnerLoop(RetryDecision::None); + } match inner.invocation { AgentInvocation::ManualUpdate { target_revision } => { self.manual_update(target_revision).await @@ -2465,16 +2588,26 @@ impl Invocation<'_, Ctx> { debug!( "Skipping enqueued invocation with idempotency key {idempotency_key} as it already has a result" ); - if let Err(error) = - self.parent.cancel_invocation(idempotency_key.clone()).await + match self + .parent + .cancel_invocation_from_loop(idempotency_key.clone()) + .await { - warn!( - agent_id = %self.owned_agent_id.agent_id, - "Failed to remove completed invocation from the pending queue: {error}" - ); - return CommandOutcome::BreakInnerLoop(RetryDecision::Immediate); + Ok(true) => CommandOutcome::Continue, + // A stop is waiting for this loop to exit. + Ok(false) => CommandOutcome::BreakInnerLoop(RetryDecision::None), + Err(_) if self.parent.is_given_up() => { + CommandOutcome::BreakInnerLoop(RetryDecision::None) + } + Err(error) => { + warn!( + agent_id = %self.owned_agent_id.agent_id, + %error, + "Failed to remove completed invocation from the pending queue" + ); + CommandOutcome::BreakInnerLoop(RetryDecision::Immediate) + } } - CommandOutcome::Continue } } else { self.invoke_agent(invocation).await @@ -2716,6 +2849,18 @@ impl Invocation<'_, Ctx> { .and_then(|result| result) { tracing::error!(%error, "Failed to complete durable streaming session"); + // An in-place retry would reopen the oplog at an epoch this executor no + // longer holds, so a lost shard gives the agent up instead. + if self.parent.give_up_if_shard_lost(&error) { + self.store + .data_mut() + .on_invocation_failure( + &full_function_name, + &TrapType::Interrupt(InterruptKind::ShardLost), + ) + .await; + return CommandOutcome::BreakInnerLoop(RetryDecision::None); + } return failed_agent_invocation_outcome( self.parent.agent_mode(), RetryDecision::Immediate, @@ -2734,6 +2879,9 @@ impl Invocation<'_, Ctx> { .data_mut() .on_invocation_failure(&full_function_name, &TrapType::Interrupt(kind)) .await; + if let Some(outcome) = self.given_up_outcome() { + return outcome; + } if self.uses_streams { let _ = self .parent @@ -2742,20 +2890,41 @@ impl Invocation<'_, Ctx> { } failed_agent_invocation_outcome(self.parent.agent_mode(), decision) } - Err(error) => { + // Intercepted before the arm below, which would flatten it into an + // `AgentError::InternalError` and append an `Error` entry to the very oplog that + // just refused the write. + Err(WorkerExecutorError::OplogFenced { .. }) => { self.store .data_mut() .on_invocation_failure( &full_function_name, - &TrapType::Error { - error: AgentError::InternalError(error.to_string()), - retry_from: OplogIndex::INITIAL, - in_atomic_region: false, - atomic_region_had_side_effects: false, - semantic_trap_retry_override: None, - }, + &TrapType::Interrupt(InterruptKind::ShardLost), ) .await; + CommandOutcome::BreakInnerLoop(RetryDecision::None) + } + Err(error) => { + // The success hook commits `AgentInvocationFinished`; if the storage refused that + // commit the oplog has latched the fence, and the failure is a lost shard rather + // than an internal error. + let trap_type = self + .store + .data() + .durable_ctx() + .trap_type_under_latched_fence(TrapType::Error { + error: AgentError::InternalError(error.to_string()), + retry_from: OplogIndex::INITIAL, + in_atomic_region: false, + atomic_region_had_side_effects: false, + semantic_trap_retry_override: None, + }); + self.store + .data_mut() + .on_invocation_failure(&full_function_name, &trap_type) + .await; + if let Some(outcome) = self.given_up_outcome() { + return outcome; + } if self.uses_streams { let _ = self .parent @@ -2767,6 +2936,16 @@ impl Invocation<'_, Ctx> { } } + /// The outcome of an invocation that failed on an agent this executor has given up, `None` + /// while the agent is still its own. Checked after `on_invocation_failure`, which marks a lost + /// shard. Nothing more is written: a terminal streaming-session failure, like any other + /// terminal record, would end an invocation the shard's new owner resumes. + fn given_up_outcome(&self) -> Option { + self.parent + .is_given_up() + .then_some(CommandOutcome::BreakInnerLoop(RetryDecision::None)) + } + /// The logic handling an agent invocation that did not succeed. async fn agent_invocation_failed( &mut self, @@ -2812,6 +2991,14 @@ impl Invocation<'_, Ctx> { )), }, }; + // A fence that reached the guest through a `String` boundary classifies as an ordinary + // failure; the latch still says the oplog is finished, so the agent is given up, not retried. + let trap_type = trap_type.map(|trap_type| { + self.store + .data() + .durable_ctx() + .trap_type_under_latched_fence(trap_type) + }); let decision = match trap_type { Some(trap_type) => { self.store @@ -2821,6 +3008,9 @@ impl Invocation<'_, Ctx> { } None => RetryDecision::None, }; + if let Some(outcome) = self.given_up_outcome() { + return outcome; + } if self.uses_streams && decision == RetryDecision::None { let _ = self @@ -2953,12 +3143,20 @@ impl Invocation<'_, Ctx> { .await { Ok(update_description) => { - // Enqueue the update - let _ = self.parent.enqueue_update(update_description).await; - - // Reactivate the worker - CommandOutcome::BreakInnerLoop(RetryDecision::Immediate) - // Stop processing the queue to avoid race conditions + // Refused, or the worker is stopping or being deleted: nothing restarts it + // here, and the manual update stays pending for the next generation. + match self + .parent + .enqueue_update_from_loop(update_description) + .await + { + // Reactivate the worker; stop processing the queue to avoid race + // conditions + Ok(true) => CommandOutcome::BreakInnerLoop(RetryDecision::Immediate), + Ok(false) | Err(_) => { + CommandOutcome::BreakInnerLoop(RetryDecision::None) + } + } } Err(error) => { self.fail_update( @@ -2991,6 +3189,12 @@ impl Invocation<'_, Ctx> { .await } Ok(InvokeResult::Interrupted { interrupt_kind, .. }) => { + // Marked before `fail_update` checks the mark: a `ShardLost` interrupt is a lost + // shard whether or not anything marked the agent on its way here. + self.parent + .give_up_if_shard_lost(&WorkerExecutorError::Interrupted { + kind: interrupt_kind, + }); self.fail_update( target_revision, format!("failed to get a snapshot for manual update: {interrupt_kind:?}"), @@ -3077,11 +3281,23 @@ impl Invocation<'_, Ctx> { target_revision: ComponentRevision, error: String, ) -> CommandOutcome { - self.store + // A `FailedUpdate` drops the pending manual update from the status, so written for an + // agent given up here it would keep the shard's new owner from ever applying it. Nothing + // is written; the update stays pending and runs there. + if self.parent.is_given_up() { + return CommandOutcome::BreakInnerLoop(RetryDecision::None); + } + // Refused, the agent has been given up: it stops rather than carrying on at a revision + // whose failed update was never recorded. + match self + .store .data() .on_worker_update_failed(target_revision, Some(error)) - .await; - CommandOutcome::Continue + .await + { + Ok(()) => CommandOutcome::Continue, + Err(_) => CommandOutcome::BreakInnerLoop(RetryDecision::None), + } } /// Extends the invocation context with a new span containing information about the invocation @@ -3227,14 +3443,19 @@ impl Invocation<'_, Ctx> { .agent_wallet_cards_snapshot(); let wallet_generation = self.store.data().durable_ctx().wallet_generation(); - self.parent + if self + .parent .add_and_commit_oplog(OplogEntry::snapshot( payload, snapshot.mime_type, active_cards, wallet_generation, )) - .await; + .await + .is_err() + { + return CommandOutcome::BreakInnerLoop(RetryDecision::None); + } debug!("Periodic snapshot saved successfully"); // A snapshot is committed between invocations, so no jumpable @@ -3310,6 +3531,30 @@ fn successful_agent_invocation_outcome( } } +/// The failure a loop's exit tears the agent's entity bodies down with. +/// +/// A given-up agent's shard moved, and that wins over any lifecycle interrupt still queued and +/// over a recovery failure: the bodies must not report an API interrupt or a fault for an agent +/// that simply has a new owner. A give-up first discovered by the stop's own commit, which +/// runs after this choice, cannot be reflected, because by then the bodies are already torn down; +/// the fence still holds, and that stop still fails the waiters and removes the generation. +fn exit_owner_failure( + given_up: Option, + final_interrupt: Option, + recovery_failure: Option<&WorkerExecutorError>, +) -> OwnerFailureWinner { + given_up + .or_else(|| final_interrupt.map(OwnerFailureWinner::Lifecycle)) + .or_else(|| { + recovery_failure + .cloned() + .map(OwnerFailureWinner::Infrastructure) + }) + .unwrap_or_else(|| { + OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt(Timestamp::now_utc())) + }) +} + fn failed_agent_invocation_outcome( agent_mode: AgentMode, decision: RetryDecision, @@ -3380,13 +3625,14 @@ mod tests { use super::{ CommandOutcome, ConcurrentAgentPermitState, InvocationLoop, PeriodicSnapshotAction, ResidentAgentOwnership, ResidentWakeup, catch_invocation_loop_panic, - close_usage_before_delete, coalesce_filesystem_limit_update, + close_usage_before_delete, coalesce_filesystem_limit_update, exit_owner_failure, failed_agent_invocation_outcome, finish_filesystem_limit_unload, periodic_snapshot_failure_outcome, publish_unload_outcome, run_invocation_loop_task, snapshot_action_at, snapshot_baseline_timestamp, spawn_module_owned_unload, successful_agent_invocation_outcome, unload_resident_agent_ownership, wait_for_resident_wakeup, }; + use crate::durable_host::tool::operation::OwnerFailureWinner; use crate::sandbox_filesystem::ScriptedSandboxFilesystem; use crate::services::active_agents::stop_loaded_idle_if_eligible; use crate::services::agent_filesystem::{ @@ -3407,7 +3653,7 @@ mod tests { use golem_common::model::agent::AgentMode; use golem_common::model::oplog::AgentError; use golem_common::model::{OplogIndex, Timestamp}; - use golem_service_base::error::worker_executor::WorkerExecutorError; + use golem_service_base::error::worker_executor::{InterruptKind, WorkerExecutorError}; use std::collections::VecDeque; use std::path::Path; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; @@ -3599,6 +3845,46 @@ mod tests { delete(seal(filesystem)).await.unwrap(); } + #[test] + fn exit_owner_failure_prefers_giving_up() { + let shard_lost = || Some(OwnerFailureWinner::Lifecycle(InterruptKind::ShardLost)); + let recovery_failure = WorkerExecutorError::unknown("recovery failed"); + + assert!(matches!( + exit_owner_failure(shard_lost(), None, None), + OwnerFailureWinner::Lifecycle(InterruptKind::ShardLost) + )); + // A shard that moved outranks both a queued lifecycle interrupt and a recovery failure: + // the agent was neither suspended through the API nor broken, it has a new owner. + assert!(matches!( + exit_owner_failure( + shard_lost(), + Some(InterruptKind::Suspend(Timestamp::now_utc())), + Some(&recovery_failure), + ), + OwnerFailureWinner::Lifecycle(InterruptKind::ShardLost) + )); + + // Without a give-up the previous order stands: the queued interrupt, then the + // recovery failure, then an interrupt stamped now. + assert!(matches!( + exit_owner_failure( + None, + Some(InterruptKind::Suspend(Timestamp::now_utc())), + Some(&recovery_failure), + ), + OwnerFailureWinner::Lifecycle(InterruptKind::Suspend(_)) + )); + assert!(matches!( + exit_owner_failure(None, None, Some(&recovery_failure)), + OwnerFailureWinner::Infrastructure(_) + )); + assert!(matches!( + exit_owner_failure(None, None, None), + OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt(_)) + )); + } + impl Drop for TestStoreOwner { fn drop(&mut self) { self.dropped.store(true, Ordering::Release); diff --git a/golem-worker-executor/src/worker/mod.rs b/golem-worker-executor/src/worker/mod.rs index 63861c6661..139e2e8a0c 100644 --- a/golem-worker-executor/src/worker/mod.rs +++ b/golem-worker-executor/src/worker/mod.rs @@ -78,8 +78,8 @@ use crate::services::golem_config::SnapshotPolicy; use crate::services::linear_memory::{LinearMemoryTracker, SHARED_LINEAR_MEMORY_ERROR}; use crate::services::oplog::plugin::ForwardingOplog; use crate::services::oplog::{ - ArchiveWait, CommitLevel, EphemeralOplog, MultiLayerOplog, Oplog, OplogLifecycleGuard, - OplogOps, downcast_oplog, + ArchiveWait, CommitLevel, EphemeralOplog, MultiLayerOplog, Oplog, OplogError, OplogFence, + OplogLifecycleGuard, OplogOps, downcast_oplog, }; use crate::services::resource_limits::AtomicResourceEntry; use crate::services::resource_usage_metering::ResourceUsageAccount; @@ -154,8 +154,8 @@ use golem_common::model::worker::{ use golem_common::model::{ AgentFingerprint, AgentId, AgentInvocation, AgentInvocationOutput, AgentInvocationPayload, AgentInvocationResult, AgentMetadata, AgentStatusRecord, IdempotencyKey, OwnedAgentId, - PendingInvocationRef, PendingUpdateKind, PendingUpdateRef, RetryPolicyState, Timestamp, - TimestampedAgentInvocation, + PendingInvocationRef, PendingUpdateKind, PendingUpdateRef, RetryPolicyState, ShardAssignment, + ShardEpoch, ShardId, Timestamp, TimestampedAgentInvocation, }; use golem_common::one_shot::OneShotEvent; use golem_common::read_only_lock; @@ -671,6 +671,9 @@ pub struct ResolvedWorkerData { /// Prevents weak-reference background work from starting while an unloaded /// worker is being conditionally removed from `ActiveAgents`. cache_retirement_in_progress: AtomicBool, + /// Set once this executor has given the agent up. One-shot: the first reason wins, and the + /// agent is never revived here. + given_up_reason: std::sync::OnceLock, startup_attempt: StartupAttemptTracker, linear_memory_grant: StdMutex>>>, /// Lifecycle request shared across resident worker generations. A terminal request is retained @@ -994,7 +997,8 @@ impl DurableStreamConsumerJournal for WorkerDurableStreamConsume let (_, changed) = self .state_actor .commit_and_update_state(CommitLevel::Always) - .await; + .await + .map_err(|fence| OplogError::Fenced(fence).to_string())?; if changed { self.state_actor.notify_status_changed(); } @@ -1110,6 +1114,37 @@ fn is_infrastructure_recovery_error(error: &WorkerExecutorError) -> bool { } } +/// Why the agent is given up because of `error`, if the failure is really a lost shard. +/// +/// Such a failure is not this executor's to record or to retry. An append would be refused by the +/// same fence that caused it, and on a shard given up without one it would write an error into an +/// oplog the new owner is already recovering. A retry in place would reopen the oplog at the epoch +/// this executor no longer holds. Every path that fails on a lost shard - startup and replay +/// recovery, streaming-session completion, the dropped-call drain, an interrupt - classifies with +/// this one predicate, so none of them can take the in-place retry another one refuses. +/// +/// `latched` is the fence the oplog holds, for a refusal that reached its caller flattened into +/// some other error; the classified shapes are recognised without one. +pub(crate) fn shard_lost_give_up_reason( + error: &WorkerExecutorError, + latched: Option, +) -> Option { + match latched { + Some(fence) => Some(GiveUpReason::Fenced(Some(Box::new(fence)))), + None if matches!( + error, + WorkerExecutorError::OplogFenced { .. } + | WorkerExecutorError::Interrupted { + kind: InterruptKind::ShardLost + } + ) => + { + Some(GiveUpReason::Fenced(None)) + } + None => None, + } +} + impl Worker { pub(crate) async fn ensure_not_failed + Send + Sync>( deps: &T, @@ -1204,14 +1239,141 @@ impl Worker { .unwrap_or_else(|| "-".to_string()) } - pub(crate) async fn remove_from_active_agents(self: &Arc) { - while !self.deps.active_agents().remove_worker(self, false).await - && self.is_current_cached_owner().await - { - tokio::task::yield_now().await; + /// Records that this executor is giving the agent up. Idempotent; the first reason wins and is + /// the only one logged. + /// + /// Synchronous and lock-free on purpose: the stop path calls it while holding the worker + /// lifecycle lock, where anything that could take that lock again would deadlock. Logging takes + /// no worker lifecycle lock. + pub(crate) fn mark_given_up(&self, reason: GiveUpReason) -> bool { + let first = self.given_up_reason.set(reason).is_ok(); + // Before anything else can run: the status blob, its checkpoint and the recovery-index + // row are the new owner's now, and none of them is fenced. + self.status_flusher.stop_for_give_up(); + self.status_checkpointer.stop_for_give_up(); + if first && let Some(reason) = self.given_up_reason.get() { + // Debug rather than warn: the oplog that latched a fence has already warned with both + // epochs, and a revoke or reassignment is logged by the sweep that gives agents up. + debug!( + agent_id = %self.owned_agent_id, + ?reason, + "Giving the agent up: this executor no longer owns its shard" + ); } + first + } + + pub(crate) fn is_given_up(&self) -> bool { + self.given_up_reason.get().is_some() + } + + /// Marks the agent given up when `error` means its shard was lost, per + /// [`shard_lost_give_up_reason`]. Returns whether the agent is given up, by this failure or + /// for an earlier reason: either way the caller must neither record the failure nor retry in + /// place. Marked rather than stopped, for the same reason as [`Self::mark_given_up`]: the + /// callers unwind to a stop, some of them while holding the worker lifecycle lock. + pub(crate) fn give_up_if_shard_lost(&self, error: &WorkerExecutorError) -> bool { + if let Some(reason) = shard_lost_give_up_reason(error, self.oplog.fence()) { + self.mark_given_up(reason); + } + self.is_given_up() + } + + /// Stops this worker through this handle, whichever generation it is. Nothing public reaches + /// the stop through an arbitrary handle, and a handle kept past its generation is exactly what + /// the tests using this need. + #[cfg(feature = "test-utils")] + #[doc(hidden)] + pub async fn test_stop(&self) { + self.stop_internal( + false, + None, + UnloadRequest::ordinary(UnloadReason::ExplicitStop), + FinalWorkerState::Unloaded { + startup_failure: None, + }, + PendingLiveInvocationDisposition::Fail, + ) + .await; + } + + /// [`GiveUpReason::to_error`] for the reason recorded for this agent, or + /// `ShardingNotReady` when none is recorded yet. + pub(crate) fn give_up_error(&self) -> WorkerExecutorError { + self.given_up_reason + .get() + .map_or(WorkerExecutorError::ShardingNotReady, |reason| { + reason.to_error() + }) + } + + /// What entity bodies are torn down with once this agent has been given up; `None` while it is + /// still this executor's. + pub(crate) fn given_up_owner_failure(&self) -> Option { + self.given_up_reason.get().map(GiveUpReason::owner_failure) + } + + /// Give the agent up: stop it here without writing to its oplog or its status, and drop it + /// from this executor so the worker service resumes it on the shard's owner. + /// + /// Never a restart in place - that would reopen the oplog with the same stale epoch and let + /// this executor keep writing to an agent it no longer owns. + pub(crate) async fn give_up(&self, reason: GiveUpReason) { + self.mark_given_up(reason); + let error = self.give_up_error(); + // Signalled before the stop so a running guest actually leaves wasmtime; the loop then + // exits through `stop_internal`, which is where the agent is dropped. The ack is + // deliberately not awaited: a caller that blocks on it would panic if the worker was + // already stopping and its broadcast sender had gone. + self.set_interrupting_for(InterruptKind::ShardLost, UnloadReason::ShardLost) + .await; + // A deletion already retiring this worker owns its stop and its removal, and may be + // waiting on the invocation this stop would wait for. The mark is enough: the loop's exit + // paths and the removal honour it, so nothing more is written for the agent. + if self.deletion_owns_retirement().await { + return; + } + self.stop_internal( + false, + Some(error.clone()), + UnloadRequest::ordinary(UnloadReason::ShardLost), + FinalWorkerState::Unloaded { + startup_failure: Some(error), + }, + PendingLiveInvocationDisposition::Fail, + ) + .await; + } + + /// Whether `cell` is this worker's published status. Each generation shares its cell with its + /// own worker-state actor and nothing else, so the cell identifies the generation to code that + /// holds it but not the worker. + pub(crate) fn shares_status_cell( + &self, + cell: &Arc>, + ) -> bool { + Arc::ptr_eq(&self.last_known_status, cell) + } + + /// Drops this generation from `ActiveAgents`. A newer generation cached under the same id is + /// left alone: a given-up agent passes through here more than once, and no repeat pass may + /// evict the generation that replaced it. + /// + /// Takes `&self` rather than the `Arc`, because the stop path that removes a given-up + /// generation holds only a reference; the cache supplies the `Arc` it checks identity against. + pub(crate) async fn remove_from_active_agents(&self) { + // A given-up agent's entity bodies are torn down as `ShardLost`: it was not + // interrupted through the Golem API, its shard moved. + let owner_failure = self.given_up_owner_failure().unwrap_or_else(|| { + OwnerFailureWinner::Lifecycle(InterruptKind::Interrupt(Timestamp::now_utc())) + }); + self.deps + .active_agents() + .remove_generation(self, owner_failure) + .await; } + /// Whether this worker is still the generation `ActiveAgents` has cached for its agent id. async fn is_current_cached_owner(&self) -> bool { self.active_agents() .try_get(&self.owned_agent_id) @@ -1219,6 +1381,14 @@ impl Worker { .is_some_and(|worker| std::ptr::eq(worker.as_ref(), self)) } + /// Interrupts and retires this worker as its owner - an environment deletion, for example - + /// draining it and dropping it from `ActiveAgents`. Idempotent and shared: a second caller + /// while retirement is already in flight awaits the same outcome instead of repeating it. + /// + /// An agent given up before or during retirement (its shard moved) is not retired here: + /// [`Self::quiesce_for_owner_retirement`] writes and caches nothing for it, sends its waiters + /// to the shard's new owner and returns the give-up error, and the give-up that marked it + /// removes the generation. pub(crate) async fn interrupt_and_retire( self: &Arc, interrupt: InterruptKind, @@ -1286,6 +1456,12 @@ impl Worker { if let WorkerInstance::CleanupFailed(error) = &*self.instance.lock().await { return Err(error.clone()); } + // Given up before this retirement: the oplog is the shard's new owner's. No terminal is + // claimed, written or cached, and the waiters and the caller are sent to the owner. + if self.is_given_up() { + self.fail_pending_invocations(self.give_up_error()).await; + return Err(self.give_up_error()); + } if interrupt.is_some() { let pending = self.interrupt_signal.lock().await.claim_pending_terminal(); if let Some(pending) = pending { @@ -1295,18 +1471,28 @@ impl Worker { AgentStatus::Running | AgentStatus::Retrying | AgentStatus::Suspended ) { let entry = match pending.kind { - InterruptKind::Interrupt(_) => OplogEntry::interrupted(), - InterruptKind::Suspend(_) => OplogEntry::suspend(), + InterruptKind::Interrupt(_) => Some(OplogEntry::interrupted()), + InterruptKind::Suspend(_) => Some(OplogEntry::suspend()), + // The oplog is the shard's new owner's to write. + InterruptKind::ShardLost => None, InterruptKind::Restart | InterruptKind::Jump => { unreachable!("only terminal interrupts can be claimed") } }; - self.add_and_commit_oplog(entry).await; - if matches!(pending.kind, InterruptKind::Interrupt(_)) - && let Some(key) = &status.current_idempotency_key - { - self.store_invocation_failure(key, &TrapType::Interrupt(pending.kind)) - .await; + if let Some(entry) = entry { + // Refused, the agent has been given up during this retirement: no failure + // is cached for the invocation the new owner resumes, and its waiters are + // sent there before anything removes this generation. + if self.add_and_commit_oplog(entry).await.is_err() { + self.fail_pending_invocations(self.give_up_error()).await; + return Err(self.give_up_error()); + } + if matches!(pending.kind, InterruptKind::Interrupt(_)) + && let Some(key) = &status.current_idempotency_key + { + self.store_invocation_failure(key, &TrapType::Interrupt(pending.kind)) + .await; + } } } } @@ -1812,13 +1998,22 @@ impl Worker { .oplog_service() .lock_lifecycle(&worker.owned_agent_id.agent_id) .await; - let metadata = Self::get_existing_worker_metadata( - &worker.deps, - &mut lifecycle, - &worker.owned_agent_id, - expected_fingerprint, - ) - .await + // The epoch is read first, as in `get_or_create_worker_metadata`: a worker whose shard + // has already left the assignment is refused without touching storage, and the oplog + // opened below asserts the epoch this executor holds. + let metadata = match owned_shard_epoch(&worker.deps, &worker.owned_agent_id.agent_id) { + Ok(shard_epoch) => { + Self::get_existing_worker_metadata( + &worker.deps, + &mut lifecycle, + &worker.owned_agent_id, + expected_fingerprint, + shard_epoch, + ) + .await + } + Err(error) => Err(error), + } .and_then(|metadata| { metadata.ok_or_else(|| { WorkerExecutorError::worker_not_found(worker.owned_agent_id.agent_id()) @@ -1874,7 +2069,11 @@ impl Worker { Self::start_durable_stream_attachment_reconciler(&worker); } drop(instance); + let resolved = published.is_ok(); let _ = sender.send(published); + if resolved { + worker.give_up_if_shard_left_during_construction().await; + } }); (true, completion) } @@ -1894,6 +2093,23 @@ impl Worker { (started, completion.await) } + /// The sweep a delivery runs selects from the resolved agents, so one still being built when + /// the shard left is not in it: it read the assignment before the revoke, opened its oplog at + /// the epoch it was granted, and would stay here, unfenced, until the new owner claims the + /// oplog. This is the sweep's late half for that agent, run once it is published - after the + /// instance guard is released, because giving up takes it. + async fn give_up_if_shard_left_during_construction(self: &Arc) { + let assignment = self.shard_service().try_get_current_assignment(); + if given_up_by_assignment( + assignment.as_ref(), + &self.owned_agent_id.agent_id, + self.oplog().shard_epoch(), + ) { + info!("The agent's shard left this executor while the agent was being created"); + self.give_up(GiveUpReason::ShardNotAssigned).await; + } + } + async fn finish_construction( self: &Arc, start: std::time::Instant, @@ -2119,6 +2335,7 @@ impl Worker { EphemeralInvocationState::Available }), cache_retirement_in_progress: AtomicBool::new(false), + given_up_reason: std::sync::OnceLock::new(), startup_attempt: StartupAttemptTracker::default(), linear_memory_grant: StdMutex::new(None), interrupt_signal: Arc::new(async_lock::Mutex::new(WorkerInterruptState::default())), @@ -2188,6 +2405,10 @@ impl Worker { ) .await?; let status = worker.state_actor.attached_status().await; + // Returned rather than ignored: an agent created at an epoch another executor has + // already claimed is refused here with `OplogFenced`, which its caller retries on the + // shard's owner. Nothing is lost by returning early, since the next load repeats this + // check against the same short oplog. worker .state_actor .append_invocation_if_version( @@ -2197,7 +2418,7 @@ impl Worker { status.invocation_results.revert_generation(), self.instance.clone().lock_owned().await, ) - .await; + .await?; } if worker.last_known_status.load().has_durable_stream_history && !self @@ -2457,6 +2678,14 @@ impl Worker { oom_retry_count: u32, existing_start_attempt: Option, ) -> Result, WorkerExecutorError> { + // A handle kept past a generation this executor has given up must not start it again: the + // start would take permits, could append `Resumed` to an oplog the new owner now writes, + // and a failure in it would publish, by agent id, to the waiters of the generation that + // replaced this one. + if this.is_given_up() { + return Err(this.give_up_error()); + } + { *this.last_resume_request.lock().await = Timestamp::now_utc(); } @@ -2504,7 +2733,7 @@ impl Worker { OplogEntry::resumed(), None, ) - .await; + .await?; } let start_attempt = this.startup_attempt.begin(start_attempt); this.mark_as_loading(start_attempt); @@ -2626,6 +2855,11 @@ impl Worker { if instance.ensure_not_deleting().is_err() || self.oplog.is_retired() { return Ok(None); } + // An agent given up here is archived by the shard's new owner. Moving its entries or + // dropping its cached status from this executor would write to state it no longer owns. + if self.is_given_up() { + return Err(self.give_up_error()); + } if !self.active_agents().contains_worker_generation(self).await { return Err(WorkerExecutorError::runtime( "Archival worker left the active cache; retry with the current owner", @@ -2837,7 +3071,7 @@ impl Worker { { self.before_deletion_stage(WorkerDeletionStage::StreamsCleaned) .await?; - let producer = DurableStreamStore::load_indexed_with_commit( + let producer = match DurableStreamStore::load_indexed_with_commit( self.oplog.clone(), self.owned_agent_id.clone(), self.initial_worker_metadata.fingerprint, @@ -2853,7 +3087,13 @@ impl Worker { self.agent_mode(), ) .await - .map_err(|error| WorkerExecutorError::runtime(error.to_string()))?; + { + Ok(producer) => producer, + Err(error) => { + let error = error.into_worker_executor_error(WorkerExecutorError::runtime); + return Err(self.deletion_step_failed(error).await); + } + }; let activity = crate::services::activity::ActivityGate::new(); let guard = activity.try_enter().unwrap(); let maintenance = std::panic::AssertUnwindSafe(guard.scope(async { @@ -2915,7 +3155,9 @@ impl Worker { producer.wait_durable_drained().await; self.state_actor.drain_lifecycle().await?; self.durable_stream_commit()(None).await; - maintenance?; + if let Err(error) = maintenance { + return Err(self.deletion_step_failed(error).await); + } self.complete_deletion_stage(WorkerDeletionStage::StreamsCleaned) .await; } @@ -2943,14 +3185,27 @@ impl Worker { .await; result.map_err(WorkerExecutorError::runtime)?; } - self.worker_service() + // Attempted even when the agent was given up while the deletion ran: a revoke alone + // hands the shard to nobody, and the fenced delete below is what tells the two cases + // apart. It succeeds while the key is still this executor's, and is refused once + // another executor has taken it. + let removed = self + .worker_service() .remove( &mut lifecycle, &self.owned_agent_id, self.initial_worker_metadata.agent_mode, self.initial_worker_metadata.fingerprint, + self.oplog.shard_epoch(), ) - .await?; + .await; + if let Err(error) = removed { + drop(lifecycle); + return Err(match shard_lost_give_up_reason(&error, None) { + Some(reason) => self.leave_deletion_to_new_owner(reason).await, + None => error, + }); + } self.complete_deletion_stage(WorkerDeletionStage::DurableStateRemoved) .await; } @@ -2972,6 +3227,33 @@ impl Worker { Ok(()) } + /// Ends a deletion whose agent has moved to another executor. Its durable state is the new + /// owner's now, so nothing more is removed from storage: this executor drops only its own cached + /// generation - which `give_up` left to the deletion - and the caller gets the answer that + /// sends the delete to the new owner. + async fn leave_deletion_to_new_owner( + self: &Arc, + reason: GiveUpReason, + ) -> WorkerExecutorError { + self.mark_given_up(reason); + self.active_agents().remove_worker(self, true).await; + self.give_up_error() + } + + /// The error a failed deletion step ends the attempt with. A step that failed because the + /// shard has a new owner - typed, or flattened into another error after the refused write + /// latched this oplog's fence - hands the deletion to that owner, as a refused storage remove + /// does. Any other failure is returned unchanged. + async fn deletion_step_failed( + self: &Arc, + error: WorkerExecutorError, + ) -> WorkerExecutorError { + match shard_lost_give_up_reason(&error, self.oplog.fence()) { + Some(reason) => self.leave_deletion_to_new_owner(reason).await, + None => error, + } + } + async fn before_deletion_stage( &self, stage: WorkerDeletionStage, @@ -3123,7 +3405,23 @@ impl Worker { }; } + /// A start of an agent this executor has given up is never a success, and however it failed, + /// its waiters are told to retry on the shard's new owner. Without this a stop that reports a + /// generic "stopped before startup completed", or the recovery error a lost shard caused, + /// would hand them a failure they surface instead of retrying. + fn given_up_startup_result( + &self, + result: Result<(), WorkerExecutorError>, + ) -> Result<(), WorkerExecutorError> { + if self.is_given_up() { + Err(self.give_up_error()) + } else { + result + } + } + fn publish_startup_result(&self, start_attempt: Uuid, result: Result<(), WorkerExecutorError>) { + let result = self.given_up_startup_result(result); if !self.startup_attempt.complete(start_attempt, &result) { return; } @@ -3157,27 +3455,40 @@ impl Worker { _ => None, }; let is_active = active_attempt == Some(start_attempt); - let result = if is_active { + let mut result = if is_active { Ok(()) } else { Err(WorkerExecutorError::unknown( "Worker stopped before startup completed", )) }; + // The success marker is skipped for an agent this executor has given up: its oplog belongs + // to the shard's new owner, which clears the recovery error itself when it starts the + // agent. Under a fence the append is refused anyway; this also covers a shard revoked or + // reassigned without one. if is_active + && !self.is_given_up() && self .get_non_detached_last_known_status() .await .last_error_kind == Some(OplogErrorKind::Recovery) { - self.add_and_commit_oplog_internal( - &instance_guard, - OplogEntry::recovery_succeeded(), - None, - ) - .await; + // Refused, the start is not a success: the agent has been given up, its waiters are + // told to look for the shard's new owner, and the caller stops it. + if self + .add_and_commit_oplog_internal( + &instance_guard, + OplogEntry::recovery_succeeded(), + None, + ) + .await + .is_err() + { + result = Err(WorkerExecutorError::ShardingNotReady); + } } + let result = self.given_up_startup_result(result); let completed = match &result { Ok(()) => self @@ -3186,14 +3497,24 @@ impl Worker { Err(_) => self.startup_attempt.complete(start_attempt, &result), }; + let started = result.is_ok(); if completed { self.publish_completed_startup_result(start_attempt, result); } drop(instance_guard); - is_active + started } pub(crate) async fn record_recovery_failure(&self, error: &WorkerExecutorError) { + // A recovery that failed because the shard moved is recorded by nobody: the agent is given + // up here and recovered by the shard's new owner. The caller stops the worker right after + // this, and that stop is where the agent is dropped. An agent given up for a reason that + // never reached this error - a revoked or reassigned shard - is not recorded either: the + // oplog would still accept the entry, at the epoch the agent no longer owns in spirit. + if self.give_up_if_shard_lost(error) { + return; + } + let latest_status = self.get_non_detached_last_known_status().await; let previous_error = if latest_status.last_error_kind == Some(OplogErrorKind::Recovery) { Ctx::get_last_error_and_retry_count( @@ -3214,15 +3535,17 @@ impl Worker { let error = recovery_agent_error(error); let retry_policy_state = (!infrastructure_failure && error != AgentError::OutOfMemory) .then_some(RetryPolicyState::Terminal); - self.add_and_commit_oplog(OplogEntry::error( - None, - OplogErrorKind::Recovery, - error, - retry_from, - false, - retry_policy_state, - )) - .await; + // A refusal marks the agent given up, which the caller's stop acts on. + let _ = self + .add_and_commit_oplog(OplogEntry::error( + None, + OplogErrorKind::Recovery, + error, + retry_from, + false, + retry_policy_state, + )) + .await; } pub(crate) fn pending_startup_attempt(&self) -> Option { @@ -4347,18 +4670,54 @@ impl Worker { &self, update_description: UpdateDescription, ) -> Result<(), WorkerExecutorError> { - // Bump + commit under the same worker lifecycle lock. let instance_guard = self.lock_non_stopping_worker().await; + self.enqueue_update_locked(&instance_guard, update_description) + .await + } + + /// Enqueues an update from inside this worker's own invocation loop. Returns `Ok(false)`, + /// enqueuing nothing, when the worker is stopping or has been given up, for the reasons given + /// on [`Self::cancel_invocation_from_loop`]. The manual update that produced the description + /// stays pending in the oplog and runs again in the next generation, here or on the shard's + /// new owner. + pub(crate) async fn enqueue_update_from_loop( + &self, + update_description: UpdateDescription, + ) -> Result { + let instance_guard = self.instance.lock().await; + if self.stopping_or_given_up(&instance_guard) { + return Ok(false); + } + self.enqueue_update_locked(&instance_guard, update_description) + .await?; + Ok(true) + } + + /// Whether the runtime is stopping - on its own or inside a deletion, which wraps the runtime + /// it stops - or the agent has been given up. Checked under the worker lifecycle lock by the + /// loop-side operations, which must not wait for a stop that waits for the loop. + fn stopping_or_given_up(&self, instance_guard: &MutexGuard<'_, WorkerInstance>) -> bool { + matches!( + instance_guard.deletion_runtime(), + WorkerInstance::Stopping(_) + ) || self.is_given_up() + } + + async fn enqueue_update_locked( + &self, + instance_guard: &MutexGuard<'_, WorkerInstance>, + update_description: UpdateDescription, + ) -> Result<(), WorkerExecutorError> { + // Bump + commit under the same worker lifecycle lock. instance_guard.ensure_not_deleting()?; self.bump_read_only_cache_epoch(); - let entry = OplogEntry::pending_update(update_description.clone()); + let entry = OplogEntry::pending_update(update_description); self.add_and_commit_oplog_internal( - &instance_guard, + instance_guard, entry, Some(WorkerCommand::WorkAvailable), ) - .await; - drop(instance_guard); + .await?; Ok(()) } @@ -5047,7 +5406,8 @@ impl Worker { loop { let delta = growth.delta.swap(0, Ordering::AcqRel); if delta > 0 { - self.add_to_oplog(OplogEntry::grow_memory(delta)).await; + self.add_to_oplog_or_give_up(OplogEntry::grow_memory(delta)) + .await; } let current_growth = self.memory_growth.lock().unwrap(); @@ -5077,7 +5437,7 @@ impl Worker { target_revision: ComponentRevision, new_component_size: u64, new_active_plugins: HashSet, - ) { + ) -> Result<(), OplogError> { let done = { let mut growth = self.memory_growth.lock().unwrap(); let entry = OplogEntry::successful_update( @@ -5090,12 +5450,13 @@ impl Worker { self.state_actor .queue_ordered_oplog_entry(self.clone(), entry) }; - if done.await.is_err() { + // A refusal is returned: an update the oplog does not record has not been applied. + done.await.unwrap_or_else(|_| { panic!( "Worker state actor for {} dropped an ordered oplog entry", self.owned_agent_id - ); - } + ) + }) } pub(crate) fn request_memory_limit_interrupt(self: &Arc, memory: LinearMemoryTracker) { @@ -5337,7 +5698,7 @@ impl Worker { status.invocation_results.revert_generation(), instance_guard, ) - .await + .await? { continue; } @@ -5349,7 +5710,7 @@ impl Worker { entry, None, ) - .await; + .await?; } if let Some(idempotency_key) = semantic_idempotency_key { @@ -5939,7 +6300,9 @@ impl Worker { move |_| prepared, ) .await - .map_err(|error| WorkerExecutorError::invalid_request(error.to_string()))?; + .map_err(|error| { + error.into_worker_executor_error(WorkerExecutorError::invalid_request) + })?; attached_during_prepare = true; prepared } else { @@ -5975,7 +6338,7 @@ impl Worker { }, ) .await - .map_err(|error| WorkerExecutorError::runtime(error.to_string()))?; + .map_err(|error| error.into_worker_executor_error(WorkerExecutorError::runtime))?; attached_during_prepare = true; prepared }; @@ -6096,11 +6459,11 @@ impl Worker { ) }), ) - .await; + .await?; streams .commit_consumer_journal() .await - .map_err(WorkerExecutorError::runtime)?; + .map_err(|error| self.runtime_error_unless_fenced(error))?; } for mapping in &persisted_foreign_mappings { if streams @@ -6111,6 +6474,9 @@ impl Worker { continue; } let mut retry_delay = Duration::from_millis(10); + // Not bounded: the pending invocation and its attachment are already committed, so + // giving up would report a failure for an invocation the agent still runs. Only an + // agent this executor no longer owns stops retrying. loop { if self.owner_retirement_requested.is_cancelled() { return Err(WorkerExecutorError::runtime("worker owner is retiring")); @@ -6123,6 +6489,17 @@ impl Worker { .await { Ok(()) => break, + // A refusal is permanent. Retrying it would hold the worker lifecycle lock + // forever, and the give-up that has to take that lock could never stop the + // agent. + Err(_) if self.oplog.fence().is_some() => { + return Err(self.runtime_error_unless_fenced( + "durable foreign topology activation was fenced".to_string(), + )); + } + // A revoke writes nothing, so no fence latches: the mark is all there is, and + // `give_up` is already waiting for this lock. + Err(_) if self.is_given_up() => return Err(self.give_up_error()), Err(error) => { warn!( session = %prepared.attempt.session_key.idempotency_key, @@ -6142,6 +6519,8 @@ impl Worker { .expect("persisted durable session has a commit notification") .send(()); } else if !attached_during_prepare { + // Refused, the invocation is not accepted: the dropped notification tells the waiter + // nothing was committed, and the error keeps `Accepted` from being sent. self.state_actor .commit_and_update_state_notifying( CommitLevel::Always, @@ -6149,7 +6528,8 @@ impl Worker { .take() .expect("legacy durable session has a commit notification"), ) - .await; + .await + .map_err(|fence| self.given_up_by(fence))?; self.state_actor.notify_status_changed(); } if !retained_acceptance && let WorkerInstance::Running(running) = &*instance_guard { @@ -6423,7 +6803,12 @@ impl Worker { accepted_epoch, }) .await - .map_err(|error| WorkerExecutorError::invalid_request(error.to_string()))?; + .map_err(|error| match self.oplog.fence() { + // The attempt reports its errors as text. A refused append behind one has already + // latched the fence, which is reported as such so the caller reroutes to the owner. + Some(fence) => WorkerExecutorError::from(OplogError::Fenced(fence)), + None => WorkerExecutorError::invalid_request(error), + })?; let streams = make_streams(accepted_epoch, attempt.attempt_id)?; for (binding, mapping) in mappings.iter().zip(&materialized_mappings) { @@ -6937,7 +7322,7 @@ impl Worker { Arc::new(move |committed| { let state_actor = state_actor.clone(); Box::pin(async move { - let (_, changed) = if let Some(committed) = committed { + let committed = if let Some(committed) = committed { state_actor .commit_and_update_state_notifying(CommitLevel::Always, committed) .await @@ -6946,7 +7331,10 @@ impl Worker { .commit_and_update_state(CommitLevel::Always) .await }; - if changed { + // A refusal is not reported through this closure: the producer reads it + // back from the oplog's latch, which the refused append set before the + // status actor replied, and the actor has spawned the give-up. + if let Ok((_, true)) = committed { state_actor.notify_status_changed(); } }) @@ -7637,6 +8025,12 @@ impl Worker { let Some(worker) = worker.upgrade() else { break; }; + // Given up here: the shard's new owner recovers and reconciles the + // agent's streams, and a session record appended from this executor + // would land in an oplog that is no longer its to write. + if worker.is_given_up() { + break; + } if worker.cache_retirement_in_progress() { drop(worker); if !wait_for_durable_stream_retry(&shutdown, interval_duration).await { @@ -7701,53 +8095,145 @@ impl Worker { /// Appends an oplog entry without forcing a durable commit. Callers that /// require ordering must await the append before exposing subsequent work. - pub async fn add_to_oplog(&self, entry: OplogEntry) -> OplogIndex { + pub async fn add_to_oplog(&self, entry: OplogEntry) -> Result { self.oplog.add(entry).await } - pub async fn commit_oplog_and_update_state(&self, commit_level: CommitLevel) -> OplogIndex { - let (result, changed) = self.state_actor.commit_and_update_state(commit_level).await; - if changed { - // The notification goes through the worker-state actor's lifecycle queue so that - // this method never waits on (or becomes a queued owner of) the worker lifecycle lock. This - // method runs inside durable-call host futures polled by wasmtime's store event loop - // and on store-keeping wasm fibers, neither of which may block on locks shared with - // the other (see the `state_actor` module docs). - self.state_actor.notify_status_changed(); + /// Appends an entry on a path that has no way to report the failure to its caller. + /// + /// A fenced write means the shard moved while this agent was resident. The agent is marked + /// given up so the stop that follows drops it from this executor rather than writing to an + /// oplog another executor owns now, and `OplogIndex::NONE` is returned for the entry that was + /// not written - the same "no index" value a debugging session's discarded write returns. + /// + /// Marked rather than stopped here on purpose: these callers run under the worker lifecycle + /// lock and inside the wasm store, where `give_up` would deadlock on the lock it already + /// holds. The fence latches on the oplog, so the invocation's next write is refused too and + /// unwinds the loop, which is where the stop belongs. + /// + /// Every other storage failure keeps the fail-stop behaviour it has always had. + pub async fn add_to_oplog_or_give_up(&self, entry: OplogEntry) -> OplogIndex { + match self.oplog.add(entry).await { + Ok(index) => index, + Err(OplogError::Fenced(fence)) => { + self.mark_given_up(GiveUpReason::Fenced(Some(Box::new(fence)))); + OplogIndex::NONE + } + Err(error) => panic!("oplog write: {error}"), + } + } + + /// Commits the buffered entries and folds them into the published status. + /// + /// A commit the storage refused is returned as `OplogError::Fenced`, and the agent is marked + /// given up. It has to be reported rather than folded into "nothing changed": below the + /// commit threshold an add only buffers, so this commit is where a takeover is found, and a + /// caller about to run a side effect, publish a result or acknowledge a request must not do + /// it for entries that never reached the storage. The status actor has already spawned the + /// give-up that drops the agent from this executor. + pub async fn commit_oplog_and_update_state( + &self, + commit_level: CommitLevel, + ) -> Result { + match self.state_actor.commit_and_update_state(commit_level).await { + Ok((index, changed)) => { + if changed { + // The notification goes through the worker-state actor's lifecycle queue so + // that this method never waits on (or becomes a queued owner of) the worker + // lifecycle lock. This method runs inside durable-call host futures polled by + // wasmtime's store event loop and on store-keeping wasm fibers, neither of + // which may block on locks shared with the other (see the `state_actor` module + // docs). + self.state_actor.notify_status_changed(); + } + Ok(index) + } + Err(fence) => Err(self.given_up_by(fence)), } - result } /// Enqueues an actor-owned commit + fold and waits only until the commit is acknowledged. /// Later ordered status reads remain behind the fold on the same FIFO queue. - pub async fn commit_oplog_before_status_update(&self, commit_level: CommitLevel) { - self.state_actor + /// + /// A refused commit is never acknowledged: the actor drops the receipt once the oplog has + /// latched the fence, so a missing acknowledgement with a latched fence is reported as + /// `OplogError::Fenced` (see [`Self::commit_oplog_and_update_state`]). Without one it means + /// the actor stopped, which stays fatal. + pub async fn commit_oplog_before_status_update( + &self, + commit_level: CommitLevel, + ) -> Result<(), OplogError> { + if self + .state_actor .enqueue_commit_and_update_state_notifying(commit_level) .await - .unwrap_or_else(|_| { - panic!( - "Worker state actor for {} stopped before acknowledging commit", - self.owned_agent_id - ) - }); + .is_ok() + { + return Ok(()); + } + match self.oplog.fence() { + Some(fence) => Err(self.given_up_by(fence)), + None => panic!( + "Worker state actor for {} stopped before acknowledging commit", + self.owned_agent_id + ), + } } - // Should only be called from invocation loop - pub async fn add_and_commit_oplog(&self, entry: OplogEntry) -> OplogIndex { - let result = self.add_to_oplog(entry).await; + /// Adds an entry and commits it, reporting a refusal of either as `OplogError::Fenced` (see + /// [`Self::commit_oplog_and_update_state`]). + /// + /// Every other storage failure keeps the fail-stop behaviour of + /// [`Self::add_to_oplog_or_give_up`]. + pub async fn add_and_commit_oplog(&self, entry: OplogEntry) -> Result { + let index = self.add_to_oplog_or_fenced(entry).await?; self.commit_oplog_and_update_state(CommitLevel::Always) - .await; - result + .await?; + Ok(index) } - pub async fn queue_card_revocation(&self, card_id: CardId) -> Option { - self.queue_card_revocations(&[card_id]) - .await + /// Appends an entry, reporting a refusal as `OplogError::Fenced` and marking the agent + /// given up; any other storage failure is fatal, as it always has been. + async fn add_to_oplog_or_fenced(&self, entry: OplogEntry) -> Result { + match self.oplog.add(entry).await { + Ok(index) => Ok(index), + Err(OplogError::Fenced(fence)) => Err(self.given_up_by(fence)), + Err(error) => panic!("oplog write: {error}"), + } + } + + /// A failure that reached its caller flattened into a string, reported as the fence when the + /// oplog has latched one: the worker service retries a lost shard on its new owner, but not an + /// internal error. + fn runtime_error_unless_fenced(&self, error: String) -> WorkerExecutorError { + match self.oplog.fence() { + Some(fence) => self.given_up_by(fence).into(), + None => WorkerExecutorError::runtime(error), + } + } + + /// Marks the agent given up because its oplog refused a write, and returns the refusal for the + /// caller to propagate. + pub(crate) fn given_up_by(&self, fence: OplogFence) -> OplogError { + self.mark_given_up(GiveUpReason::Fenced(Some(Box::new(fence.clone())))); + OplogError::Fenced(fence) + } + + pub async fn queue_card_revocation( + &self, + card_id: CardId, + ) -> Result, OplogError> { + Ok(self + .queue_card_revocations(&[card_id]) + .await? .into_iter() - .next() + .next()) } - pub async fn queue_card_revocations(&self, card_ids: &[CardId]) -> Vec { + pub async fn queue_card_revocations( + &self, + card_ids: &[CardId], + ) -> Result, OplogError> { let boundary_lock = self.card_event_boundary_lock.clone(); let _boundary_guard = boundary_lock.lock().await; self.queue_card_revocations_locked(card_ids).await @@ -7756,7 +8242,7 @@ impl Worker { pub(crate) async fn queue_card_revocations_locked( &self, card_ids: &[CardId], - ) -> Vec { + ) -> Result, OplogError> { let status = self.state_actor.attached_status().await; let pending_revocations = status .pending_card_events @@ -7781,19 +8267,19 @@ impl Worker { let mut queued_event_indices = Vec::with_capacity(card_ids.len()); for card_id in card_ids { queued_event_indices.push( - self.add_to_oplog(OplogEntry::card_event_queued( + self.add_to_oplog_or_fenced(OplogEntry::card_event_queued( None, Box::new(QueuedCardEvent::revoke(card_id)), )) - .await, + .await?, ); } if !queued_event_indices.is_empty() { self.commit_oplog_and_update_state(CommitLevel::Always) - .await; + .await?; } - queued_event_indices + Ok(queued_event_indices) } pub async fn receive_card_transfer( @@ -7837,6 +8323,8 @@ impl Worker { } let boundary_guard = self.card_event_boundary_lock.clone().lock_owned().await; + // A refused entry is not delivered: the sender learns the shard moved instead of treating + // a transfer this oplog never recorded as received. self.state_actor .append_and_commit_attached( OplogEntry::card_event_queued( @@ -7864,13 +8352,18 @@ impl Worker { self.published_authority_generation.clone() } + /// [`Self::add_and_commit_oplog`] for a caller holding the worker lifecycle lock, which sends + /// `wakeup` to the running loop when the commit changed the status. + /// + /// A refusal is returned rather than acknowledged: the management request that wrote the + /// entry must not report it as accepted. async fn add_and_commit_oplog_internal( &self, instance_guard: &MutexGuard<'_, WorkerInstance>, entry: OplogEntry, wakeup: Option, - ) -> OplogIndex { - let result = self.add_to_oplog(entry).await; + ) -> Result { + let index = self.add_to_oplog_or_fenced(entry).await?; // The caller already holds the worker lifecycle lock (and sends the wakeup itself below), so // this must not enqueue a `NotifyStatusChanged` lifecycle job: the commit job is safe to // await while holding the worker lifecycle lock precisely because the status task never takes @@ -7878,7 +8371,8 @@ impl Worker { let (_, changed) = self .state_actor .commit_and_update_state(CommitLevel::Always) - .await; + .await + .map_err(|fence| self.given_up_by(fence))?; if changed && let Some(wakeup) = wakeup @@ -7887,7 +8381,7 @@ impl Worker { running.sender.send(wakeup).unwrap(); }; - result + Ok(index) } async fn activate_plugin_internal( @@ -7909,7 +8403,7 @@ impl Worker { OplogEntry::activate_plugin(plugin_grant_id), Some(WorkerCommand::WorkAvailable), ) - .await; + .await?; drop(instance_guard); Ok(()) @@ -7934,7 +8428,7 @@ impl Worker { OplogEntry::deactivate_plugin(plugin_grant_id), Some(WorkerCommand::WorkAvailable), ) - .await; + .await?; drop(instance_guard); Ok(()) @@ -7981,7 +8475,36 @@ impl Worker { idempotency_key: IdempotencyKey, ) -> Result<(), WorkerExecutorError> { let instance_guard = self.lock_non_stopping_worker().await; + self.cancel_invocation_locked(&instance_guard, idempotency_key) + .await + } + + /// Cancels a pending invocation from inside this worker's own invocation loop. Returns + /// `Ok(false)`, cancelling nothing, when the worker is stopping or has been given up. + /// + /// [`Self::cancel_invocation`] waits for a stop to finish, and a stop from outside the loop + /// finishes only once the loop has exited, so the loop waiting for it would never return. The + /// loop exits instead; the invocation stays pending in the oplog and the next generation + /// skips it again. A given-up agent's oplog is the new owner's to write, so nothing is + /// written for it even before its stop begins. + pub(crate) async fn cancel_invocation_from_loop( + &self, + idempotency_key: IdempotencyKey, + ) -> Result { + let instance_guard = self.instance.lock().await; + if self.stopping_or_given_up(&instance_guard) { + return Ok(false); + } + self.cancel_invocation_locked(&instance_guard, idempotency_key) + .await?; + Ok(true) + } + async fn cancel_invocation_locked( + &self, + instance_guard: &MutexGuard<'_, WorkerInstance>, + idempotency_key: IdempotencyKey, + ) -> Result<(), WorkerExecutorError> { if instance_guard.ensure_not_deleting().is_err() { return Err(WorkerExecutorError::invalid_request( "Cannot cancel invocation on a deleting worker", @@ -7989,13 +8512,11 @@ impl Worker { }; self.add_and_commit_oplog_internal( - &instance_guard, + instance_guard, OplogEntry::cancel_pending_invocation(idempotency_key), Some(WorkerCommand::WorkAvailable), ) - .await; - - drop(instance_guard); + .await?; Ok(()) } @@ -8236,12 +8757,12 @@ impl Worker { } self.oplog .add_durable_stream_batch(Box::new(move |_| entries)) - .await - .map_err(WorkerExecutorError::runtime)?; + .await?; } else { - self.oplog - .add(OplogEntry::revert(dropped_region.clone())) - .await; + // Through the fence-aware helper: a revert is a write like any other, and if the + // shard has a new owner this must surface rather than be discarded. + self.add_to_oplog_or_fenced(OplogEntry::revert(dropped_region.clone())) + .await?; } self.durable_stream_commit()(None).await; self.reattach_worker_status().await; @@ -8369,6 +8890,12 @@ impl Worker { loop { match self.lookup_invocation_result(key).await { LookupResult::Interrupted => break Ok(LookupResult::Interrupted), + // Given up here, so no result for the key will be published on this executor. The + // retry answer is published when the agent is given up but not cached, and a + // receiver that lagged past it, or subscribed after it, finds it here instead. + LookupResult::New | LookupResult::Pending if self.is_given_up() => { + break Ok(LookupResult::Complete(Err(self.give_up_error()))); + } LookupResult::New | LookupResult::Pending => { let waiting = subscription.wait_for(|event| match event { Event::InvocationCompleted { @@ -8401,6 +8928,13 @@ impl Worker { next_ownership_check = tokio::time::Instant::now() + INVOCATION_OWNERSHIP_RECHECK_INTERVAL; + // A key the give-up did not know about yet (enqueued, not yet folded + // into the status it failed from) gets no retry answer published. + // The lookup at the top of the loop answers it. + if self.is_given_up() { + continue; + } + // An agent whose shard has moved is resumed by whoever owns // it now, and its `InvocationCompleted` is published on that // executor's bus. Nothing will ever arrive on ours, and the @@ -8637,6 +9171,28 @@ impl Worker { drop(instance_guard); self.handle_stop_result(stop_result).await; + + // The removal point. Every loop exit and every external stop passes through here, so a + // given-up agent arrives more than once: from its own loop, again from the give-up + // that waited for that loop, and from any stop that arrives through a handle kept past its + // generation. Everything below is scoped to this generation; a pass that finds the entry + // gone or holding a newer generation does nothing. It runs only after the loop has gone, so + // the new owner cannot recover the agent while it is still running here. + // + // Waiters are failed here as well, in memory only. An agent given up from inside its own + // loop - a fence refused in a host call traps with `ShardLost` - stops without failing + // anyone, and while this executor's assignment still names the shard their ownership + // re-check keeps passing. The give-up spawned by the loop's exit commit answers them + // only if it still finds this generation cached, and the loop's own removal can get there + // first. Failing them before the removal, while this generation still holds the entry, + // keeps the failure away from a newer generation's waiters, which match by agent id. Keys + // that already have a result keep it. A generation that left the cache some other way + // first (an idle expiry, an environment unload) is not reached here. + if self.is_given_up() && self.deps.active_agents().is_cached_generation(self).await { + self.fail_pending_invocations(self.give_up_error()).await; + self.remove_from_active_agents().await; + } + if !called_from_invocation_loop && let Some(startup_attempt) = startup_attempt { self.complete_startup(startup_attempt, Err(startup_error)); } @@ -8762,24 +9318,46 @@ impl Worker { fail_pending_invocations.is_some() ); + // Make sure the oplog is committed. Best-effort: a stop must finish. + match self.oplog.commit(CommitLevel::Always).await { + Ok(_) => {} + Err(OplogError::Fenced(fence)) => { + // The shard has a new owner. `mark_given_up` is synchronous and takes + // no lock, so it is safe under the worker lifecycle lock this arm holds - + // calling `give_up` here would deadlock on that same lock. + self.mark_given_up(GiveUpReason::Fenced(Some(Box::new(fence)))); + } + Err(error) => { + warn!(%error, "Committing the oplog while stopping failed"); + } + } + + // After the commit: it can be the first write to find that the shard has a new + // owner, and the waiters are then told to retry there rather than handed the + // stop's own error, which would be cached as a failure the new owner's run of the + // invocation could never correct. + // // TODO: fail pending invocations should be factored out of here and be guaranteed to run // even if there are multiple concurrent stop attempts. if let Some(ref error) = fail_pending_invocations { self.fail_pending_invocations(error.clone()).await; }; - // Make sure the oplog is committed - self.oplog.commit(CommitLevel::Always).await; - // Persist any pending cached-status changes synchronously before the worker leaves // memory, so a subsequent cold load does not have to re-fold oplog entries that were // only reflected in the (deferred) in-memory status. Best-effort: a failure is // logged/metered inside `flush` and re-queued; the blob is reconstructable from the // oplog, so it must not block the stop. - if let Err(err) = self - .status_flusher - .flush(status_flusher::FlushReason::Forced) - .await + // + // Skipped entirely for an agent given up here, whether by the commit above or by a + // revoke or reassignment that latched no fence: this is a key-value write, which + // is NOT fenced, so it would happily overwrite the new owner's newer status blob + // with our stale one. The flusher refuses it too once `mark_given_up` has run. + if !self.is_given_up() + && let Err(err) = self + .status_flusher + .flush(status_flusher::FlushReason::Forced) + .await { debug!("Forced status flush on stop failed (will retry in background): {err}"); } @@ -8930,18 +9508,24 @@ impl Worker { let mut queue = self.queue.write().await; queue.retain(|invocation| !invocation.is_abandoned()); if pending_live_invocations == PendingLiveInvocationDisposition::Fail && !queue.is_empty() { - let status = self.get_attached_last_known_status().await; - let error = Self::ensure_not_failed( - &self.deps, - &self.owned_agent_id, - self.agent_mode(), - &status, - ) - .await - .err() - .unwrap_or_else(|| { - WorkerExecutorError::runtime("Worker stopped with queued resident work") - }); + // A given-up agent's queued work moves to the shard's new owner, so its waiters get + // the retry answer, as in `fail_pending_invocations`. + let error = if self.is_given_up() { + self.give_up_error() + } else { + let status = self.get_attached_last_known_status().await; + Self::ensure_not_failed( + &self.deps, + &self.owned_agent_id, + self.agent_mode(), + &status, + ) + .await + .err() + .unwrap_or_else(|| { + WorkerExecutorError::runtime("Worker stopped with queued resident work") + }) + }; for invocation in queue.drain(..) { invocation.fail(&error); } @@ -8974,6 +9558,19 @@ impl Worker { } async fn fail_pending_invocations(&self, error: WorkerExecutorError) { + // A given-up agent's pending invocations are not failed, they move: the shard's new + // owner runs them. So their waiters are told to retry there, whatever stopped this + // generation, and no result is cached. A cached failure would outlive the stop: a later + // lookup would answer the key with an `InvocationFailed` the caller does not retry, and + // the invocation loop, finding the key complete, would cancel the pending invocation - + // waiting on this very stop to do it, or, with nothing fenced yet, cancelling work the new + // owner still has to run. + let given_up = self.is_given_up(); + let error = if given_up { + self.give_up_error() + } else { + error + }; let queued_items = self.queue.write().await.drain(..).collect::>(); let mut origins = self.external_invocation_origins.write().await; @@ -8993,6 +9590,11 @@ impl Worker { { continue; } + if given_up { + self.publish_completion(idempotency_key, Err(error.clone())); + origins.remove(idempotency_key); + continue; + } invocation_results.insert( idempotency_key.clone(), InvocationResult::Cached { @@ -9071,6 +9673,9 @@ impl Worker { Self::start_if_needed_internal(this, oom_retry_count, start_attempt).await } + /// `shard_epoch` is the epoch the opened oplog asserts, read by the caller with + /// [`owned_shard_epoch`] before anything touches storage. See + /// [`Self::get_or_create_worker_metadata`]. pub(crate) async fn get_existing_worker_metadata< T: HasWorkerService + HasComponentService + HasOplogService + HasConfig + Sync, >( @@ -9078,6 +9683,7 @@ impl Worker { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, expected_fingerprint: Option, + shard_epoch: Option, ) -> Result, WorkerExecutorError> { let Some(metadata) = this.worker_service().get(owned_agent_id).await? else { return Ok(None); @@ -9087,9 +9693,15 @@ impl Worker { { return Ok(None); } - Self::hydrate_existing_worker_metadata(this, lifecycle, owned_agent_id, metadata) - .await - .map(Some) + Self::hydrate_existing_worker_metadata( + this, + lifecycle, + owned_agent_id, + metadata, + shard_epoch, + ) + .await + .map(Some) } async fn hydrate_existing_worker_metadata< @@ -9099,6 +9711,7 @@ impl Worker { lifecycle: &mut OplogLifecycleGuard, owned_agent_id: &OwnedAgentId, metadata: GetWorkerMetadataResult, + shard_epoch: Option, ) -> Result { let component_id = owned_agent_id.component_id(); let GetWorkerMetadataResult { @@ -9168,6 +9781,7 @@ impl Worker { initial_worker_metadata.clone(), read_only_lock::arc_swap::ReadOnlyView::new(current_status.clone()), read_only_lock::std::ReadOnlyLock::new(execution_status.clone()), + shard_epoch, ) .await; @@ -9191,6 +9805,7 @@ impl Worker { + HasConfig + HasOplogService + HasEnvironmentStateService + + HasShardService + Sync, >( this: &T, @@ -9203,6 +9818,14 @@ impl Worker { freshness_disposition: InvocationFreshnessDisposition, creation_mode: WorkerCreationMode, ) -> Result { + // Captured once, here, and cached for the life of the oplog. One live oplog is one + // ownership generation. Re-reading the epoch per write would only let a losing executor + // talk itself back into ownership. A renewal never moves an epoch. A delivery that raises + // the epoch of a shard this executor kept means the shard left and came back: the + // assignment sweep gives the agent up and the open-oplog cache declines the old handle, so + // the next open claims the new epoch. Read before anything else, so a worker whose shard + // has already left the assignment is refused without touching storage. + let shard_epoch = owned_shard_epoch(this, &owned_agent_id.agent_id)?; let component_id = owned_agent_id.component_id(); if creation_mode == WorkerCreationMode::ComponentAgent { @@ -9216,7 +9839,8 @@ impl Worker { let existing = if freshness_disposition == InvocationFreshnessDisposition::KnownFresh { None } else { - Self::get_existing_worker_metadata(this, lifecycle, owned_agent_id, None).await? + Self::get_existing_worker_metadata(this, lifecycle, owned_agent_id, None, shard_epoch) + .await? }; match existing { @@ -9384,6 +10008,7 @@ impl Worker { initial_worker_metadata.clone(), read_only_lock::arc_swap::ReadOnlyView::new(initial_status.clone()), read_only_lock::std::ReadOnlyLock::new(execution_status.clone()), + shard_epoch, ) .await } else { @@ -9396,6 +10021,7 @@ impl Worker { initial_worker_metadata.clone(), read_only_lock::arc_swap::ReadOnlyView::new(initial_status.clone()), read_only_lock::std::ReadOnlyLock::new(execution_status.clone()), + shard_epoch, ) .await }; @@ -9772,6 +10398,130 @@ struct PendingWorkerInterrupt { unload_request: UnloadRequest, } +/// The shard epoch the agent's oplog is opened to assert, read from this executor's current +/// assignment. See [`shard_epoch_to_assert`]. +fn owned_shard_epoch( + this: &T, + agent_id: &AgentId, +) -> Result, WorkerExecutorError> { + shard_epoch_to_assert( + this.shard_service().try_get_current_assignment().as_ref(), + agent_id, + ) +} + +/// The shard epoch an oplog opened for `agent_id` asserts under `assignment`. +/// +/// `Ok(None)` only when there is no assignment at all, before the first registration. +/// +/// An assignment that does not hold the agent's shard is refused with `ShardingNotReady`, which +/// the worker service answers by refreshing its routing and retrying. It does not map to `None`, +/// because admission and this read take separate locks. A revoke can land between them, and its +/// sweep cannot see a worker that is still being built. That worker would otherwise open an oplog +/// that asserts nothing and stay cached, unfenced, across a later re-grant of the shard. +/// +/// The same refusal covers a cleared assignment (lapsed lease, deregistration), which holds no +/// shards at all. +fn shard_epoch_to_assert( + assignment: Option<&ShardAssignment>, + agent_id: &AgentId, +) -> Result, WorkerExecutorError> { + let Some(assignment) = assignment else { + return Ok(None); + }; + let shard_id = ShardId::from_agent_id(agent_id, assignment.number_of_shards); + assignment + .epoch_of(&shard_id) + .map(Some) + .ok_or(WorkerExecutorError::ShardingNotReady) +} + +/// Whether an agent whose oplog asserts `held` has been superseded by a delivery that assigns its +/// shard at `assigned`. +/// +/// - A shard's epoch rises only when it changed owner in between. A kept shard at a higher epoch +/// therefore means another executor may have written to the agent. +/// - An equal epoch is the same ownership generation. +/// - A lower epoch never comes from a newer owner, because the shard manager never lowers an +/// epoch. Giving the agent up would only reopen it below the epoch its oplog row already holds. +/// - `None` on either side is not this rule's business. An ephemeral handle asserts nothing, and +/// an absent shard is the membership check's to handle. +pub(crate) fn epoch_superseded(held: Option, assigned: Option) -> bool { + matches!((held, assigned), (Some(held), Some(assigned)) if held < assigned) +} + +/// Whether a delivered `assignment` takes `agent_id` away from this executor. `held` is the epoch +/// the agent's oplog asserts, `None` when it asserts none or the agent has no oplog yet. +/// +/// True when either: +/// - the assignment does not hold the agent's shard. No assignment at all holds nothing, the same +/// answer `ShardService::check_worker` gives. +/// - it holds the shard at a higher epoch than `held`, per [`epoch_superseded`]. The shard left and +/// came back, so another executor may have written to the agent in between. +/// +/// Every sweep of a delivered assignment selects with this one predicate. A caller that checks +/// agents still being created passes `None`, which leaves only the membership test. +/// +/// Membership and epochs only, never the lease. A lapsed lease refuses new work and leaves running +/// work alone. +pub(crate) fn given_up_by_assignment( + assignment: Option<&ShardAssignment>, + agent_id: &AgentId, + held: Option, +) -> bool { + let Some(assignment) = assignment else { + return true; + }; + let shard_id = ShardId::from_agent_id(agent_id, assignment.number_of_shards); + match assignment.epoch_of(&shard_id) { + None => true, + assigned => epoch_superseded(held, assigned), + } +} + +/// Why this executor is giving an agent up: it no longer owns the agent's shard. +/// +/// Distinct from [`UnloadReason`], which says why an agent left memory. An agent can be unloaded +/// for memory pressure and be back a moment later; a given-up one is gone from this executor +/// and belongs to the shard's new owner. +#[derive(Clone, Debug)] +pub(crate) enum GiveUpReason { + /// A write to the agent's oplog was refused by the storage. Carries the fence when the write + /// path had it to hand; `None` when the loop only saw the classified interrupt. + Fenced(Option>), + /// The shard manager revoked the shard. + ShardRevoked, + /// A delivered assignment no longer holds the agent's shard, or holds it at a higher epoch + /// than the agent's oplog asserts. In the second case the shard came back to this executor, + /// and the agent is reopened here at the new epoch. + ShardNotAssigned, +} + +impl GiveUpReason { + /// What anyone waiting on the agent is told. Every variant is one the worker service answers + /// by refreshing its routing table and retrying, so the invocation lands on the new owner + /// instead of failing. + pub(crate) fn to_error(&self) -> WorkerExecutorError { + match self { + GiveUpReason::Fenced(Some(fence)) => WorkerExecutorError::oplog_fenced( + fence.agent_id.clone(), + fence.expected_epoch.0, + fence.actual_epoch.map(|epoch| epoch.0), + ), + // The loop saw the classified interrupt without the fence details, or the shard was + // taken back explicitly. Either way the caller's move is the same. + GiveUpReason::Fenced(None) + | GiveUpReason::ShardRevoked + | GiveUpReason::ShardNotAssigned => WorkerExecutorError::ShardingNotReady, + } + } + + /// Which `OwnerFailureWinner` entity bodies are torn down with. + pub(crate) fn owner_failure(&self) -> OwnerFailureWinner { + OwnerFailureWinner::Lifecycle(InterruptKind::ShardLost) + } +} + #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum UnloadReason { Deleting, @@ -9786,6 +10536,7 @@ pub(crate) enum UnloadReason { OutOfMemory, Panic, Restart, + ShardLost, Suspend, } @@ -9795,6 +10546,7 @@ impl UnloadReason { InterruptKind::Restart | InterruptKind::Jump => Self::Restart, InterruptKind::Suspend(_) => Self::Suspend, InterruptKind::Interrupt(_) => Self::Interrupt, + InterruptKind::ShardLost => Self::ShardLost, } } } @@ -9840,7 +10592,7 @@ impl PendingWorkerInterrupt { } else { match self.kind { InterruptKind::Restart | InterruptKind::Jump => RetryDecision::Immediate, - InterruptKind::Interrupt(_) => RetryDecision::None, + InterruptKind::Interrupt(_) | InterruptKind::ShardLost => RetryDecision::None, InterruptKind::Suspend(timestamp) => RetryDecision::TryStop(timestamp), } } @@ -10164,12 +10916,15 @@ impl RunningWorker { "Attempting update to revision {component_revision} failed with {error}" ); + // Refused, the update cannot be marked failed, and retrying would find + // the same pending update again: the start fails as a lost shard instead. parent .add_and_commit_oplog(OplogEntry::failed_update( component_revision, Some(error.to_string()), )) - .await; + .await + .map_err(WorkerExecutorError::from)?; // The update is now marked failed in the parent, we can retry. return Box::pin(Self::create_instance(parent, concurrent_agent_permit)) @@ -11219,6 +11974,81 @@ mod tests { use std::path::Path; use test_r::test; + /// Admission and the epoch read are not atomic. An agent whose shard left the assignment in + /// between must be refused, not handed an oplog that asserts nothing. + #[test] + fn an_agent_whose_shard_left_the_assignment_is_refused_an_epoch() { + let agent = AgentId { + component_id: ComponentId(Uuid::new_v4()), + agent_id: "fenced".to_string(), + }; + + assert!(matches!(shard_epoch_to_assert(None, &agent), Ok(None))); + assert!(matches!( + shard_epoch_to_assert( + Some(&ShardAssignment::unexpiring(1, [ShardId::new(0)])), + &agent + ), + Ok(Some(ShardEpoch(0))) + )); + assert!(matches!( + shard_epoch_to_assert(Some(&ShardAssignment::unexpiring(1, [])), &agent), + Err(WorkerExecutorError::ShardingNotReady) + )); + } + + #[test] + fn a_kept_shard_supersedes_an_agent_only_when_its_epoch_rose() { + assert!(epoch_superseded(Some(ShardEpoch(0)), Some(ShardEpoch(1)))); + assert!(!epoch_superseded(Some(ShardEpoch(1)), Some(ShardEpoch(1)))); + // An equal-revision redelivery carrying a lower epoch must not give up a newer handle. + assert!(!epoch_superseded(Some(ShardEpoch(1)), Some(ShardEpoch(0)))); + assert!(!epoch_superseded(None, Some(ShardEpoch(1)))); + assert!(!epoch_superseded(Some(ShardEpoch(0)), None)); + } + + #[test] + fn a_delivery_gives_up_agents_off_its_shards_or_behind_their_shards_epoch() { + let agent = AgentId { + component_id: ComponentId(Uuid::new_v4()), + agent_id: "swept".to_string(), + }; + let at_epoch = |epoch: u64| ShardAssignment { + shard_epochs: HashMap::from([(ShardId::new(0), ShardEpoch(epoch))]), + ..ShardAssignment::unexpiring(1, []) + }; + + // Membership: no assignment, or one without the shard, gives the agent up whatever its + // oplog asserts. + for held in [None, Some(ShardEpoch(0))] { + assert!(given_up_by_assignment(None, &agent, held)); + assert!(given_up_by_assignment( + Some(&ShardAssignment::unexpiring(1, [])), + &agent, + held + )); + } + + // A kept shard gives the agent up only when its epoch rose past the one the oplog asserts. + assert!(!given_up_by_assignment( + Some(&at_epoch(1)), + &agent, + Some(ShardEpoch(1)) + )); + assert!(given_up_by_assignment( + Some(&at_epoch(1)), + &agent, + Some(ShardEpoch(0)) + )); + assert!(!given_up_by_assignment( + Some(&at_epoch(0)), + &agent, + Some(ShardEpoch(1)) + )); + // An agent asserting nothing, such as one still being created, is judged by membership. + assert!(!given_up_by_assignment(Some(&at_epoch(1)), &agent, None)); + } + #[test] fn cancelled_resident_requests_are_pruned_without_dropping_snapshots() { let (sender, receiver) = tokio::sync::oneshot::channel(); @@ -11376,6 +12206,67 @@ mod tests { )); } + /// A lost shard is neither recorded nor retried in place, whichever path it failed: a recovery + /// `Error` entry, like an in-place retry of a streaming-session completion, would write to or + /// reopen an oplog whose owner has changed. Every path classifies with the same predicate, so + /// the shapes one of them recognises are recognised by all. + #[test] + fn a_failure_that_is_a_lost_shard_is_given_up_on_every_path() { + let agent_id = AgentId { + component_id: ComponentId::new(), + agent_id: "fenced".to_string(), + }; + let fence = OplogFence { + agent_id: agent_id.clone(), + expected_epoch: ShardEpoch(2), + actual_epoch: Some(ShardEpoch(3)), + writer_conflict: false, + }; + + // A latched fence wins over whatever the refusal was flattened into on its way out, and + // its details are kept for the error the waiters are given. + assert!(matches!( + shard_lost_give_up_reason( + &WorkerExecutorError::runtime("durable stream commit failed"), + Some(fence.clone()) + ), + Some(GiveUpReason::Fenced(Some(latched))) if *latched == fence + )); + assert!(matches!( + shard_lost_give_up_reason( + &WorkerExecutorError::oplog_fenced(agent_id, 2, Some(3)), + None + ), + Some(GiveUpReason::Fenced(None)) + )); + // Without a latched fence: a shard revoked or reassigned interrupts the agent with + // `ShardLost` and writes nothing, so no fence ever latches. + assert!(matches!( + shard_lost_give_up_reason( + &WorkerExecutorError::Interrupted { + kind: InterruptKind::ShardLost + }, + None + ), + Some(GiveUpReason::Fenced(None)) + )); + + // Every other failure is still the agent's own, to be recorded or retried. + for kind in [ + InterruptKind::Interrupt(Timestamp::now_utc()), + InterruptKind::Suspend(Timestamp::now_utc()), + InterruptKind::Restart, + InterruptKind::Jump, + ] { + assert!( + shard_lost_give_up_reason(&WorkerExecutorError::Interrupted { kind }, None) + .is_none(), + "{kind:?} is not a lost shard" + ); + } + assert!(shard_lost_give_up_reason(&WorkerExecutorError::runtime("boom"), None).is_none()); + } + #[test] fn pending_manual_update_keeps_storage_key_but_has_no_semantic_key() { let target_revision = ComponentRevision::new(2).unwrap(); @@ -12270,6 +13161,16 @@ mod tests { decision(InterruptKind::Suspend(suspend_timestamp), false), RetryDecision::TryStop(suspend_timestamp) ); + // A lost shard is terminal here: a retry in place would reopen the oplog with the same + // stale epoch, and the agent belongs to the shard's new owner now. + assert_eq!( + decision(InterruptKind::ShardLost, false), + RetryDecision::None + ); + assert_eq!( + UnloadReason::from_interrupt(InterruptKind::ShardLost), + UnloadReason::ShardLost + ); // Permit reacquisition overrides the kind-based decision for every kind. for kind in [ @@ -12277,6 +13178,7 @@ mod tests { InterruptKind::Jump, InterruptKind::Interrupt(Timestamp::now_utc()), InterruptKind::Suspend(Timestamp::now_utc()), + InterruptKind::ShardLost, ] { assert_eq!( decision(kind, true), @@ -12286,6 +13188,61 @@ mod tests { } } + #[test] + fn a_given_up_agent_reports_an_error_its_caller_can_retry() { + let agent_id = AgentId { + component_id: golem_common::model::component::ComponentId::new(), + agent_id: "given_up".to_string(), + }; + let fence = OplogFence { + agent_id, + expected_epoch: golem_common::model::ShardEpoch(3), + actual_epoch: Some(golem_common::model::ShardEpoch(4)), + writer_conflict: false, + }; + + // A fence the write path saw in full names both epochs, so an operator reading the log + // can tell which generation lost. + assert!(matches!( + GiveUpReason::Fenced(Some(Box::new(fence))).to_error(), + WorkerExecutorError::OplogFenced { + expected_epoch: 3, + actual_epoch: Some(4), + .. + } + )); + + // Every other shape gives the same answer a lapsed lease does: the worker service + // refreshes its routing table and retries on the owner. None of them may look like a + // plain invocation failure, or the caller would give up instead of moving. + for reason in [ + GiveUpReason::Fenced(None), + GiveUpReason::ShardRevoked, + GiveUpReason::ShardNotAssigned, + ] { + assert!( + matches!(reason.to_error(), WorkerExecutorError::ShardingNotReady), + "{reason:?} must be retriable on the new owner" + ); + } + } + + #[test] + fn giving_an_agent_up_never_looks_like_an_api_interrupt() { + // Entity bodies are torn down as `ShardLost`, not `Interrupt`: the agent was not + // interrupted through the Golem API, its shard moved. + for reason in [ + GiveUpReason::Fenced(None), + GiveUpReason::ShardRevoked, + GiveUpReason::ShardNotAssigned, + ] { + assert!(matches!( + reason.owner_failure(), + OwnerFailureWinner::Lifecycle(InterruptKind::ShardLost) + )); + } + } + #[test] fn interrupt_terminality_matrix() { fn terminal(kind: InterruptKind) -> bool { @@ -12301,6 +13258,7 @@ mod tests { assert!(!terminal(InterruptKind::Jump)); assert!(terminal(InterruptKind::Interrupt(Timestamp::now_utc()))); assert!(terminal(InterruptKind::Suspend(Timestamp::now_utc()))); + assert!(terminal(InterruptKind::ShardLost)); } #[test] diff --git a/golem-worker-executor/src/worker/state_actor.rs b/golem-worker-executor/src/worker/state_actor.rs index 92bda3301b..ee0c556485 100644 --- a/golem-worker-executor/src/worker/state_actor.rs +++ b/golem-worker-executor/src/worker/state_actor.rs @@ -55,11 +55,12 @@ use super::status::{ }; use super::status_flusher::{AgentStatusFlusher, FlushReason}; use super::{ - PendingMemoryGrowth, UnloadReason, Worker, WorkerCommand, WorkerInstance, WorkerStatusMetric, + GiveUpReason, PendingMemoryGrowth, UnloadReason, Worker, WorkerCommand, WorkerInstance, + WorkerStatusMetric, }; use crate::services::linear_memory::LinearMemoryTracker; -use crate::services::oplog::{CommitLevel, Oplog}; -use crate::services::{All, HasConfig, HasSchedulerService}; +use crate::services::oplog::{CommitLevel, Oplog, OplogError, OplogFence}; +use crate::services::{All, HasActiveAgents, HasConfig, HasSchedulerService}; use crate::workerctx::WorkerCtx; use arc_swap::ArcSwap; use chrono::Utc; @@ -174,17 +175,20 @@ pub(crate) struct OwnerCommitController { enum StatusJob { Stop, /// Commits the oplog and folds the newly committed entries into the published status. - /// Replies with the current oplog index after the commit and whether the status changed. + /// Replies with the current oplog index after the commit and whether the status changed, or + /// with the fence when the storage refused the commit. /// The reply deliberately does not depend on the worker lifecycle lock; if the caller wants the /// invocation loop notified about the change, it enqueues a lifecycle job afterwards. CommitAndUpdateState { level: CommitLevel, committed: Option>, notify_after_fold: bool, - done: oneshot::Sender<(OplogIndex, bool)>, + done: oneshot::Sender>, }, /// Appends an entry and completes its commit + fold transaction even if the caller is /// cancelled. The caller-acquired guards remain owned by this job until the transaction ends. + /// Replies with the refusal when the oplog has a new owner, so the caller never reports an + /// entry as delivered that was not written. AppendAndCommitAttached { entry: Box, _worker_keepalive: Arc, @@ -198,7 +202,7 @@ enum StatusJob { expected_result_generation: u64, expected_revert_generation: u64, instance_guard: OwnedMutexGuard, - done: oneshot::Sender, + done: oneshot::Sender>, }, /// Returns the published status after reattaching it when a jump or revert detached it. /// Serialization on the status queue prevents observing an in-flight status transition. @@ -238,7 +242,7 @@ enum LifecycleJob { OrderedOplogEntry { worker: Arc>, entry: Box, - done: oneshot::Sender<()>, + done: oneshot::Sender>, }, MemoryLimitExceeded { worker: Arc>, @@ -274,6 +278,16 @@ impl Drop for WorkerStateActor { } } +/// The error a refused append or commit replies with. The give-up is spawned inside the actor, +/// so the caller only needs an error it will not mistake for a delivered entry. +fn fenced_error(fence: &OplogFence) -> WorkerExecutorError { + WorkerExecutorError::oplog_fenced( + fence.agent_id.clone(), + fence.expected_epoch.0, + fence.actual_epoch.map(|epoch| epoch.0), + ) +} + impl WorkerStateActor { #[allow(clippy::too_many_arguments)] pub fn new( @@ -321,15 +335,22 @@ impl WorkerStateActor { notify_after_fold, done, } => { - let changed = state.commit_and_update_state(level, committed).await; - let index = state.oplog.current_oplog_index().await; - if changed && notify_after_fold { - queue_status_notification( - &status_lifecycle_jobs, - &status_notification_queued, - ); - } - let _ = done.send((index, changed)); + complete_status_job( + async { + let changed = + state.commit_and_update_state(level, committed).await?; + let index = state.oplog.current_oplog_index().await; + if changed && notify_after_fold { + queue_status_notification( + &status_lifecycle_jobs, + &status_notification_queued, + ); + } + Ok((index, changed)) + }, + done, + ) + .await; } StatusJob::AppendAndCommitAttached { entry, @@ -340,17 +361,30 @@ impl WorkerStateActor { } => { complete_status_job( async { - state.oplog.add(*entry).await; - state - .commit_and_update_state(CommitLevel::Always, None) - .await; - state.ensure_status_attached().await; - if state.detached.load(Ordering::Acquire) { - Err(WorkerExecutorError::runtime( - "Committed worker status could not be reconstructed", - )) - } else { - Ok(()) + match state.oplog.add(*entry).await { + Ok(_) => { + if let Err(fence) = state + .commit_and_update_state(CommitLevel::Always, None) + .await + { + return Err(fenced_error(&fence)); + } + state.ensure_status_attached().await; + if state.detached.load(Ordering::Acquire) { + Err(WorkerExecutorError::runtime( + "Committed worker status could not be reconstructed", + )) + } else { + Ok(()) + } + } + // The shard has a new owner: give the agent up and leave no + // further trace in an oplog that is no longer ours. + Err(OplogError::Fenced(fence)) => { + state.give_up_fenced_agent(fence.clone()); + Err(fenced_error(&fence)) + } + Err(error) => panic!("oplog write: {error}"), } }, done, @@ -375,17 +409,31 @@ impl WorkerStateActor { expected_result_generation, expected_revert_generation, ) { - return false; + return Ok(false); } drop(status); - state.oplog.add(*entry).await; - state + // Returned rather than reported as a moved version: the caller + // retries on `false`, and a fenced oplog refuses every retry. + if let Err(error) = state.oplog.add(*entry).await { + if let OplogError::Fenced(fence) = &error { + state.give_up_fenced_agent(fence.clone()); + } + return Err(error); + } + // The entry is only buffered until this commit, which is where a + // takeover is found. The key has not reached the status, so nothing + // that fails pending invocations can answer its caller: the enqueue + // itself has to be refused. + if let Err(fence) = state .commit_and_update_state(CommitLevel::Always, None) - .await; + .await + { + return Err(OplogError::Fenced(fence)); + } if let WorkerInstance::Running(running) = &*instance_guard { running.sender.send(WorkerCommand::WorkAvailable).unwrap(); } - true + Ok(true) }, done, ) @@ -443,8 +491,7 @@ impl WorkerStateActor { entry, done, } => { - worker.add_and_commit_oplog(*entry).await; - let _ = done.send(()); + let _ = done.send(worker.add_and_commit_oplog(*entry).await.map(|_| ())); } LifecycleJob::MemoryLimitExceeded { worker, memory } => { worker @@ -502,11 +549,15 @@ impl WorkerStateActor { } /// Commits the oplog and folds the new entries into the published status. Returns the - /// current oplog index after the commit and whether the status changed. + /// current oplog index after the commit and whether the status changed, or the fence when the + /// storage refused the commit; the refusal has already spawned the agent's give-up. /// /// If the caller's future is dropped while awaiting the reply, the commit still runs to /// completion on the status task (the same semantics as the oplog actor's own jobs). - pub async fn commit_and_update_state(&self, level: CommitLevel) -> (OplogIndex, bool) { + pub async fn commit_and_update_state( + &self, + level: CommitLevel, + ) -> Result<(OplogIndex, bool), OplogFence> { self.commit .run_status_job(|done| StatusJob::CommitAndUpdateState { level, @@ -521,7 +572,7 @@ impl WorkerStateActor { &self, level: CommitLevel, committed: oneshot::Sender<()>, - ) -> (OplogIndex, bool) { + ) -> Result<(OplogIndex, bool), OplogFence> { self.commit .run_status_job(|done| StatusJob::CommitAndUpdateState { level, @@ -585,7 +636,7 @@ impl WorkerStateActor { expected_result_generation: u64, expected_revert_generation: u64, instance_guard: OwnedMutexGuard, - ) -> bool { + ) -> Result { self.commit .run_status_job(|done| StatusJob::AppendInvocationIfVersion { entry: Box::new(entry), @@ -704,7 +755,7 @@ impl WorkerStateActor { &self, worker: Arc>, entry: OplogEntry, - ) -> oneshot::Receiver<()> { + ) -> oneshot::Receiver> { let (done, done_rx) = oneshot::channel(); if self .lifecycle_jobs @@ -725,7 +776,10 @@ impl WorkerStateActor { } impl OwnerCommitController { - pub async fn commit_and_update_state(&self, level: CommitLevel) -> (OplogIndex, bool) { + pub async fn commit_and_update_state( + &self, + level: CommitLevel, + ) -> Result<(OplogIndex, bool), OplogFence> { self.run_status_job(|done| StatusJob::CommitAndUpdateState { level, committed: None, @@ -776,18 +830,62 @@ async fn complete_status_job(transaction: impl Future, done: ones } impl StatusState { + /// Gives the agent up after a background oplog write was refused because its shard moved. + /// + /// Spawned rather than awaited: this runs on the status task, which must never take the + /// worker's instance lock (callers holding that lock await status jobs), and the stop inside + /// [`Worker::give_up`] does take it. Handing the stop to an independent task keeps that + /// discipline while still dropping the agent from this executor - which a bare + /// `mark_given_up` would not do, because on a background path nothing else is unwinding + /// to carry the stop out. + /// + /// Only the generation this actor belongs to is given up, identified by the status cell the + /// two share. By the time the task runs that generation may be gone and a newer one cached + /// under the same id, which is left alone: at a stale epoch its own open latches the fence and + /// gives it up, and at a re-granted epoch it is legitimately this executor's. + fn give_up_fenced_agent(&self, fence: OplogFence) { + let active_agents = self.deps.active_agents(); + let owned_agent_id = self.owned_agent_id.clone(); + let status_cell = self.last_known_status.clone(); + tokio::spawn(async move { + if let Some(worker) = active_agents.try_get_cached(&owned_agent_id).await + && worker.shares_status_cell(&status_cell) + { + worker + .give_up(GiveUpReason::Fenced(Some(Box::new(fence)))) + .await; + } + }); + } + /// The commit + status-fold transaction. Commits the oplog, then either folds the newly /// committed entries into the published status or marks the status detached when it can no /// longer be incrementally computed (e.g. after a revert or a snapshot update). Returns /// whether the published status (or its detachment) changed. + /// + /// A commit the storage refused because the shard has a new owner is returned as the fence + /// rather than folded into "unchanged": a caller whose entry was only buffered until this + /// commit must not report it as written. async fn commit_and_update_state( &self, commit_level: CommitLevel, committed: Option>, - ) -> bool { + ) -> Result { // Sample before committing: a later sample could include new, uncommitted appends. + // Reading the index is not a write, so a fenced oplog still answers it; the sample is + // only consumed on the path where the commit below succeeded. let appended_through = self.oplog.current_oplog_index().await; - let mut new_entries = self.oplog.commit(commit_level).await; + let mut new_entries = match self.oplog.commit(commit_level).await { + Ok(entries) => entries, + Err(OplogError::Fenced(fence)) => { + // Nothing was committed and nothing more can be. The `committed` sender is + // dropped rather than signalled: a fenced commit is not a commit, and every + // awaiter already reads a dropped sender as "no commit observed". + self.give_up_fenced_agent(fence.clone()); + return Err(fence); + } + Err(error) => panic!("oplog write: {error}"), + }; if let Some(committed) = committed { let _ = committed.send(()); } @@ -829,7 +927,14 @@ impl StatusState { // A bounded receipt cache may have dropped part of a long invocation. The // completion receipt has already been sent; make the deferred tail readable // from storage before taking this exceptional reconstruction path. - self.oplog.commit(CommitLevel::Always).await; + match self.oplog.commit(CommitLevel::Always).await { + Ok(_) => {} + Err(OplogError::Fenced(fence)) => { + self.give_up_fenced_agent(fence.clone()); + return Err(fence); + } + Err(error) => panic!("oplog write: {error}"), + } } try_fold_status_from( &self.deps, @@ -890,11 +995,14 @@ impl StatusState { .fetch_add(authority_change_count, Ordering::Release); } - changed + Ok(changed) } async fn reattach(&self) { - self.commit_and_update_state(CommitLevel::Always, None) + // A refused commit has already given the agent up; the status is still recomputed from + // what was committed before it. + let _ = self + .commit_and_update_state(CommitLevel::Always, None) .await; self.ensure_status_attached().await; diff --git a/golem-worker-executor/src/worker/status/tests.rs b/golem-worker-executor/src/worker/status/tests.rs index be61e9b02b..17182a0213 100644 --- a/golem-worker-executor/src/worker/status/tests.rs +++ b/golem-worker-executor/src/worker/status/tests.rs @@ -2932,6 +2932,7 @@ impl OplogService for TestCase { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2945,6 +2946,7 @@ impl OplogService for TestCase { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2958,6 +2960,7 @@ impl OplogService for TestCase { _initial_worker_metadata: AgentMetadata, _last_known_status: read_only_lock::arc_swap::ReadOnlyView, _execution_status: read_only_lock::std::ReadOnlyLock, + _shard_epoch: Option, ) -> Arc { unreachable!() } @@ -2975,7 +2978,8 @@ impl OplogService for TestCase { _lifecycle: &mut crate::services::oplog::OplogLifecycleGuard, _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, - ) { + _expected_epoch: Option, + ) -> Result<(), crate::services::oplog::OplogError> { unreachable!() } diff --git a/golem-worker-executor/src/worker/status_checkpointer.rs b/golem-worker-executor/src/worker/status_checkpointer.rs index edf1f910e4..ef3c3f59bf 100644 --- a/golem-worker-executor/src/worker/status_checkpointer.rs +++ b/golem-worker-executor/src/worker/status_checkpointer.rs @@ -89,6 +89,8 @@ pub struct StatusCheckpointer { /// Set once the owning worker starts deleting. After this, no checkpoint is written, so an /// in-flight write cannot resurrect the checkpoint after `remove_cached_status` deletes it. delete_started: AtomicBool, + /// Set once this executor gives the agent up; see [`Self::stop_for_give_up`]. + given_up: AtomicBool, /// Serializes checkpoint writes and guards the persisted baseline. state: Mutex, @@ -111,10 +113,24 @@ impl StatusCheckpointer { min_oplog_delta, worker_service, delete_started: AtomicBool::new(false), + given_up: AtomicBool::new(false), state: Mutex::new(CheckpointState { last_written: None }), } } + /// Whether no checkpoint may be written: the worker is being deleted, or given up. + fn writes_stopped(&self) -> bool { + self.delete_started.load(Ordering::Acquire) || self.given_up.load(Ordering::Acquire) + } + + /// Stops every later checkpoint write for a generation this executor has given up: the + /// checkpoint belongs to the shard's new owner now. Synchronous, like + /// [`crate::worker::status_flusher::AgentStatusFlusher::stop_for_give_up`], and for the same + /// reason does not wait out a write already in progress. + pub fn stop_for_give_up(&self) { + self.given_up.store(true, Ordering::Release); + } + /// Prevents any future checkpoint write from resurrecting the checkpoint after it is deleted. /// /// Mirrors [`crate::worker::status_flusher::AgentStatusFlusher::begin_delete`]: setting the flag @@ -147,7 +163,7 @@ impl StatusCheckpointer { /// Best-effort: a write failure is logged and metered, the baseline is left unchanged, and the /// worker continues. The oplog remains the source of truth. pub async fn maybe_checkpoint(&self, status: &AgentStatusRecord, reason: CheckpointReason) { - if self.is_ephemeral || !self.enabled || self.delete_started.load(Ordering::Acquire) { + if self.is_ephemeral || !self.enabled || self.writes_stopped() { return; } @@ -155,7 +171,7 @@ impl StatusCheckpointer { // Re-check after taking the lock: `begin_delete` may have set the flag while we waited for // it (it takes this same lock as a barrier), so once we hold the lock the flag is final. - if self.delete_started.load(Ordering::Acquire) { + if self.writes_stopped() { return; } @@ -287,6 +303,7 @@ mod tests { _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, _fingerprint: AgentFingerprint, + _expected_epoch: Option, ) -> Result<(), WorkerExecutorError> { Ok(()) } diff --git a/golem-worker-executor/src/worker/status_flusher.rs b/golem-worker-executor/src/worker/status_flusher.rs index cfc6c2bb29..8ad2371746 100644 --- a/golem-worker-executor/src/worker/status_flusher.rs +++ b/golem-worker-executor/src/worker/status_flusher.rs @@ -119,6 +119,8 @@ pub struct AgentStatusFlusher { /// Set once the worker starts deleting; prevents a concurrent background flush from resurrecting /// the blob after `remove_cached_status` has deleted it. delete_started: AtomicBool, + /// Set once this executor gives the agent up; see [`Self::stop_for_give_up`]. + given_up: AtomicBool, } impl AgentStatusFlusher { @@ -152,9 +154,27 @@ impl AgentStatusFlusher { }), dirty: AtomicBool::new(false), delete_started: AtomicBool::new(false), + given_up: AtomicBool::new(false), }) } + /// Whether nothing more may be written for this worker: it is being deleted, or given up. + fn writes_stopped(&self) -> bool { + self.delete_started.load(Ordering::Acquire) || self.given_up.load(Ordering::Acquire) + } + + /// Stops every later write for a generation this executor has given up, the blob and the + /// recovery-index row alike: both belong to the shard's new owner now, and a write from here + /// could overwrite its status or drop the row its crash recovery relies on. + /// + /// Synchronous, so `Worker::mark_given_up` can call it under the worker lifecycle lock. Unlike + /// [`Self::begin_delete`] it does not wait out a flush already past its early-out; that write + /// was under way before the give-up, like any other write racing the takeover. + pub fn stop_for_give_up(&self) { + self.given_up.store(true, Ordering::Release); + self.dirty.store(false, Ordering::Release); + } + /// Called from the hot path whenever the in-memory status changed. Updates the recovery index /// synchronously (only when the tracking predicate transitions) and then either marks the /// worker dirty for the background sweeper or, when background flushing is disabled, flushes the @@ -164,7 +184,7 @@ impl AgentStatusFlusher { previous_status: &AgentStatusRecord, new_status: &AgentStatusRecord, ) { - if self.is_ephemeral { + if self.is_ephemeral || self.given_up.load(Ordering::Acquire) { return; } @@ -213,7 +233,7 @@ impl AgentStatusFlusher { /// flag is the source of truth; the queue entry is just a wakeup, so we only enqueue on the /// clean→dirty transition. fn mark_dirty(&self) { - if self.is_ephemeral || self.delete_started.load(Ordering::Acquire) { + if self.is_ephemeral || self.writes_stopped() { return; } if !self.dirty.swap(true, Ordering::AcqRel) { @@ -243,7 +263,7 @@ impl AgentStatusFlusher { let mut baseline = self.baseline.lock().await; // Authoritative early-outs under the lock. - if self.delete_started.load(Ordering::Acquire) { + if self.writes_stopped() { self.dirty.store(false, Ordering::Release); return Ok(()); } @@ -293,7 +313,7 @@ impl AgentStatusFlusher { crate::metrics::workers::record_agent_status_flush_failed(reason.as_str()); // Restore the dirty flag and re-enqueue so the sweeper retries. self.dirty.store(true, Ordering::Release); - if !self.delete_started.load(Ordering::Acquire) { + if !self.writes_stopped() { self.queue.enqueue(self.queue_id, self.self_weak.clone()); } Err(err) @@ -513,6 +533,7 @@ mod tests { _owned_agent_id: &OwnedAgentId, _agent_mode: AgentMode, _fingerprint: AgentFingerprint, + _expected_epoch: Option, ) -> Result<(), WorkerExecutorError> { Ok(()) } diff --git a/golem-worker-executor/src/workerctx/default.rs b/golem-worker-executor/src/workerctx/default.rs index f4429898d7..d4fb31e636 100644 --- a/golem-worker-executor/src/workerctx/default.rs +++ b/golem-worker-executor/src/workerctx/default.rs @@ -658,7 +658,7 @@ impl UpdateManagement for Context { &self, target_revision: ComponentRevision, details: Option, - ) { + ) -> Result<(), WorkerExecutorError> { self.durable_ctx .on_worker_update_failed(target_revision, details) .await @@ -669,7 +669,7 @@ impl UpdateManagement for Context { target_revision: ComponentRevision, new_component_size: u64, new_active_plugins: HashSet, - ) { + ) -> Result<(), WorkerExecutorError> { self.durable_ctx .on_worker_update_succeeded(target_revision, new_component_size, new_active_plugins) .await diff --git a/golem-worker-executor/src/workerctx/mod.rs b/golem-worker-executor/src/workerctx/mod.rs index 0f1397d0bc..4bb8dce2cc 100644 --- a/golem-worker-executor/src/workerctx/mod.rs +++ b/golem-worker-executor/src/workerctx/mod.rs @@ -517,20 +517,22 @@ pub trait UpdateManagement { /// Marks the end of a snapshot function call. This can be used to re-enable persistence fn end_call_snapshotting_function(&mut self); - /// Called when an update attempt has failed + /// Called when an update attempt has failed. Fails when the oplog refused to record the + /// failure: the agent has been given up, and must not be rebuilt on its old revision here. async fn on_worker_update_failed( &self, target_revision: ComponentRevision, details: Option, - ); + ) -> Result<(), WorkerExecutorError>; - /// Called when an update attempt succeeded + /// Called when an update attempt succeeded. Fails when the oplog refused to record the + /// update: the agent has been given up, and the update must not be reported as applied. async fn on_worker_update_succeeded( &self, target_revision: ComponentRevision, new_component_size: u64, new_active_plugins: HashSet, - ); + ) -> Result<(), WorkerExecutorError>; } /// Operations not requiring an active worker context, but still depending on the diff --git a/golem-worker-executor/tests/active_agents.rs b/golem-worker-executor/tests/active_agents.rs index 05e9af2e56..1efe162077 100644 --- a/golem-worker-executor/tests/active_agents.rs +++ b/golem-worker-executor/tests/active_agents.rs @@ -58,6 +58,7 @@ async fn abandoned_deletion_finishes_after_the_invocation_loop_exits( epoch: 1, }], revision: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -105,6 +106,7 @@ async fn abandoned_deletion_finishes_after_the_invocation_loop_exits( .revoke_shards(RevokeShardsRequest { shard_ids: vec![ShardId { value: 0 }], revision: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -131,6 +133,7 @@ async fn abandoned_deletion_finishes_after_the_invocation_loop_exits( epoch: 2, }], revision: 2, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -172,6 +175,7 @@ async fn shard_retirement_removes_old_owner_without_removing_its_replacement( epoch: 1, }], revision: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -222,6 +226,7 @@ async fn shard_retirement_removes_old_owner_without_removing_its_replacement( number_of_shards: 1, shard_epochs: vec![], revision, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -234,6 +239,7 @@ async fn shard_retirement_removes_old_owner_without_removing_its_replacement( .revoke_shards(RevokeShardsRequest { shard_ids: vec![ShardId { value: 0 }], revision, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -289,6 +295,7 @@ async fn shard_retirement_removes_old_owner_without_removing_its_replacement( epoch: cycle as u64 + 2, }], revision: revision + 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -374,6 +381,7 @@ async fn a_push_of_zero_shards_is_refused_rather_than_applied( }], revision: 5, number_of_shards: 0, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -443,6 +451,7 @@ async fn a_revoke_older_than_the_last_delivery_does_not_sweep_agents( }], revision: 5, number_of_shards: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -457,6 +466,7 @@ async fn a_revoke_older_than_the_last_delivery_does_not_sweep_agents( .revoke_shards(RevokeShardsRequest { shard_ids: vec![shard], revision: 3, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -476,6 +486,7 @@ async fn a_revoke_older_than_the_last_delivery_does_not_sweep_agents( .revoke_shards(RevokeShardsRequest { shard_ids: vec![shard], revision: 5, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -493,6 +504,99 @@ async fn a_revoke_older_than_the_last_delivery_does_not_sweep_agents( Ok(()) } +/// A delivery that keeps a shard but raises its epoch means the shard left this executor and +/// came back, so another executor may have written to its agents in between. An agent still +/// holding the older epoch's oplog must be given up and reopened at the new epoch; a delivery at +/// the epoch it already holds must leave it alone. +/// +/// Driven over the wire for the same reason as the stale-revoke test above: the sweep lives in +/// the gRPC handler. +#[test] +#[timeout("120s")] +#[tracing::instrument] +async fn a_delivery_that_raises_a_kept_shards_epoch_gives_its_agents_up( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, + _tracing: &Tracing, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let parsed_agent_id = agent_id!("Clock", "epoch-raise-owner"); + let agent_id = executor + .start_agent(&component.id, parsed_agent_id.clone()) + .await?; + executor + .invoke_and_await_agent(&component, &parsed_agent_id, "healthcheck", data_value!()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &agent_id); + assert!(executor.worker_is_loaded(&owned_agent_id).await); + + // The single-shard bootstrap holds shard 0 at epoch 0, so the agent's oplog asserts epoch 0. + let shard = ShardId { value: 0 }; + let mut client = executor.client.clone(); + let push = |epoch: u64, revision: u64| AssignShardsRequest { + shard_epochs: vec![ShardEpochEntry { + shard_id: Some(shard), + epoch, + }], + revision, + number_of_shards: 1, + incarnation_id: String::new(), + }; + + // The handler sweeps on every applied push, changed or not, so this does run the sweep. + let same_epoch = client.assign_shards(push(0, 5)).await?.into_inner(); + assert!(matches!( + same_epoch.result, + Some(assign_shards_response::Result::Success(_)) + )); + tokio::time::sleep(Duration::from_millis(200)).await; + assert!( + executor.worker_is_loaded(&owned_agent_id).await, + "a push at the epoch the agent already holds must not give it up" + ); + + let raised = client.assign_shards(push(1, 6)).await?.into_inner(); + assert!(matches!( + raised.result, + Some(assign_shards_response::Result::Success(_)) + )); + // The agent is idle, so assignment recovery does not track it and cannot reopen it while + // this waits. + wait_until("the superseded agent to be given up", || async { + !executor.worker_is_loaded(&owned_agent_id).await + }) + .await?; + + executor + .invoke_and_await_agent(&component, &parsed_agent_id, "healthcheck", data_value!()) + .await?; + assert!(executor.worker_is_loaded(&owned_agent_id).await); + + // The reopen asserts epoch 1. Had it been handed the epoch-0 handle back, this push would + // sweep it as superseded. + let kept = client.assign_shards(push(1, 7)).await?.into_inner(); + assert!(matches!( + kept.result, + Some(assign_shards_response::Result::Success(_)) + )); + tokio::time::sleep(Duration::from_millis(200)).await; + assert!( + executor.worker_is_loaded(&owned_agent_id).await, + "the agent reopened after the raise must hold the new epoch and survive a push of it" + ); + + drop(client); + drop(executor); + Ok(()) +} + #[test] #[timeout("120s")] #[tracing::instrument] @@ -625,3 +729,97 @@ async fn ttl_eviction_removes_the_evicted_owners_card_interests( ); Ok(()) } + +/// A revoke's sweep selects from the resolved agents, so one still being created when its shard +/// leaves is not in it. It read the assignment before the revoke and opens its oplog at the epoch +/// it was granted, so it must be given up once it is published - not left cached here, unfenced, +/// until the shard's new owner claims the oplog. Prepared rather than started: a started agent's +/// invocation loop checks ownership on its own, and this is the agent that has no loop to. +#[test] +#[timeout("120s")] +#[tracing::instrument] +async fn an_agent_whose_shard_leaves_while_it_is_being_created_is_given_up_once_published( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, + _tracing: &Tracing, +) -> anyhow::Result<()> { + use golem_api_grpc::proto::golem::workerexecutor::v1::{ + CreateWorkerRequest, create_worker_response, + }; + + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = golem_common::model::AgentId { + component_id: component.id, + agent_id: agent_id!("Clock", "created-while-the-shard-leaves").to_string(), + }; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &agent_id); + // The harness learns the executor's agent registry from the first agent that runs, and the + // agent under test never does. + let warm_up = agent_id!("Clock", "created-while-the-shard-leaves-warm-up"); + executor.start_agent(&component.id, warm_up.clone()).await?; + executor + .invoke_and_await_agent(&component, &warm_up, "healthcheck", data_value!()) + .await?; + + // Creation pauses with the oplog open at the epoch the assignment held, before the agent is + // published. + let mut gate = executor + .gate_next_agent_initialization_enqueue(&agent_id) + .await; + let creation = tokio::spawn({ + let mut client = executor.client.clone(); + let request = CreateWorkerRequest { + agent_id: Some(agent_id.clone().into()), + component_owner_account_id: Some(context.account_id.into()), + environment_id: Some(context.default_environment_id.into()), + env: std::collections::HashMap::new(), + config: Vec::new(), + ignore_already_existing: false, + auth_ctx: Some(executor.auth_ctx().into()), + principal: None, + invocation_context: None, + }; + async move { client.prepare_worker(request).await } + }); + tokio::time::timeout(Duration::from_secs(20), gate.entered()) + .await + .map_err(|_| anyhow::anyhow!("creation never reached the initialization enqueue"))?; + + let mut client = executor.client.clone(); + let revoked = client + .revoke_shards(RevokeShardsRequest { + shard_ids: vec![ShardId { value: 0 }], + revision: 1, + incarnation_id: String::new(), + }) + .await? + .into_inner(); + assert!(matches!( + revoked.result, + Some(revoke_shards_response::Result::Success(_)) + )); + + drop(gate); + let created = tokio::time::timeout(Duration::from_secs(20), creation).await??; + assert!( + matches!( + created?.into_inner().result, + Some(create_worker_response::Result::Success(_)) + ), + "the creation had started before the shard left, and completes" + ); + + // Nothing else touches the agent: no invocation, no second delivery. + wait_until( + "the agent created on a shard that left to be given up", + || async { !executor.worker_is_cached(&owned_agent_id).await }, + ) + .await?; + Ok(()) +} diff --git a/golem-worker-executor/tests/api.rs b/golem-worker-executor/tests/api.rs index 13382ab86f..6b4e70a8fb 100644 --- a/golem-worker-executor/tests/api.rs +++ b/golem-worker-executor/tests/api.rs @@ -47,12 +47,13 @@ use golem_worker_executor::services::events::Event; use golem_worker_executor::services::worker_enumeration::WorkerEnumerationService; use golem_worker_executor::services::worker_proxy::{WorkerProxy, WorkerProxyError}; use golem_worker_executor::worker::{ - INVOCATION_OWNERSHIP_RECHECK_INTERVAL, WorkerDeletionHook, WorkerDeletionStage, + INVOCATION_OWNERSHIP_RECHECK_INTERVAL, Worker, WorkerDeletionHook, WorkerDeletionStage, }; use golem_worker_executor_test_utils::{ LastUniqueId, PrecompiledComponent, TestContext, TestExecutorOverrides, TestWorkerExecutor, - WorkerExecutorTestDependencies, fake_ownership, registry_test_card, start, start_customized, - start_with_overrides, start_with_redis_storage, + WorkerExecutorTestDependencies, agent_oplog_length, fake_ownership, registry_test_card, start, + start_customized, start_with_overrides, start_with_redis_storage, + take_agent_oplog_over_at_epoch, }; use pretty_assertions::assert_eq; use redis::Commands; @@ -2058,6 +2059,7 @@ async fn shard_assignment_fails_when_a_recovered_worker_cannot_be_activated( .revoke_shards(RevokeShardsRequest { shard_ids: vec![shard], revision: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -2093,6 +2095,7 @@ async fn shard_assignment_fails_when_a_recovered_worker_cannot_be_activated( }], number_of_shards: 1, revision: 2, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -2116,6 +2119,7 @@ async fn shard_assignment_fails_when_a_recovered_worker_cannot_be_activated( }], number_of_shards: 1, revision: 3, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -7223,10 +7227,15 @@ async fn resource_limits_initialized_for_component_owner_not_caller( /// Control for [`a_caller_is_answered_when_its_agents_shard_is_taken_away`]. /// /// An agent's shard leaves this executor and comes straight back, and the -/// promise it is parked on is then completed here. What this pins, and the test -/// below cannot, is that revoking a shard interrupts the agents on it with -/// `InterruptKind::Restart`, and that interrupt on its own does not strand the -/// caller. So the test below is measuring the handoff and not the interrupt. +/// promise it is parked on is then completed here. A revoke gives the agent up +/// rather than restarting it in place - a restart would reopen its oplog at an +/// epoch this executor no longer holds - so the caller is answered at once with +/// an error it can retry, and the shard returning a moment later does not +/// un-answer it. What has to survive the round trip is the work: the agent is +/// this executor's again, the invocation it was running finishes here, and the +/// retry worker-service makes under the same idempotency key is handed that +/// result instead of running it a second time. So the test below is measuring +/// the handoff and not the giving-up. /// /// It does *not* prove the `select!` is cancel-safe, though it did have to stop /// claiming that twice. Only the `wait_for` future is dropped on a tick; @@ -7246,7 +7255,7 @@ async fn a_caller_is_answered_when_its_agents_shard_comes_back( ) -> anyhow::Result<()> { let context = TestContext::new(last_unique_id); let executor = start(deps, &context).await?; - let parked = park_a_caller_on_a_promise( + let mut parked = park_a_caller_on_a_promise( &executor, &context, host_api_tests, @@ -7259,17 +7268,37 @@ async fn a_caller_is_answered_when_its_agents_shard_comes_back( revoke_shard_zero(&executor).await?; assign_shard_zero(&executor).await?; - // Sit here long enough for the caller's ownership re-check to run several - // times before the result exists, so the answer has to survive the re-check - // firing repeatedly and finding nothing wrong. + // Answered by the giving-up, not left to the ownership re-check: the revoke reached this + // executor, so nothing here waits to find out that the agent moved. + let answer = parked + .answer_within( + Duration::from_secs(20), + "caller parked in invoke_and_await was never answered, although the revoke had \ + already given its agent up here", + ) + .await?; + let error = answer.expect_err( + "the agent was given up when its shard was revoked, so the parked call cannot have \ + been handed a value", + ); + let rendered = format!("{error:#}"); + assert!( + rendered.contains("ShardingNotReady") || rendered.contains("Sharding not ready"), + "the caller has to be told to retry rather than handed a failure it would surface to \ + the user; instead it got: {rendered}" + ); + + // Sit here long enough for the re-check to have run several times, so the retry below is + // answered by an agent that survived the window rather than one that happened to be quick. sleep(INVOCATION_OWNERSHIP_RECHECK_INTERVAL * 3).await; parked.complete_the_promise(&executor, vec![42]).await?; let value = parked - .value_within( - Duration::from_secs(30), - "caller was not answered even though the shard came back and the promise \ + .retry_within( + &executor, + Duration::from_secs(60), + "the retry was never answered even though the shard came back and the promise \ was completed on this executor", ) .await?; @@ -7279,6 +7308,42 @@ async fn a_caller_is_answered_when_its_agents_shard_comes_back( elements: vec![SchemaValue::U8(42)] } ); + + // The value alone cannot tell the retry being handed the parked call's own answer apart from + // a second, independent run of `await_promise` that happened to read the same completed + // promise: both return `[42]`. Count the method's oplog pair instead - the idempotency key + // must have deduplicated the retry into the original invocation, so there can only be one. + use golem_common::model::oplog::{PublicAgentInvocation, PublicOplogEntry}; + let oplog = executor + .get_oplog(&parked.agent_id, OplogIndex::INITIAL) + .await?; + let started = oplog + .iter() + .filter(|entry| match &entry.entry { + PublicOplogEntry::AgentInvocationStarted(params) => matches!( + ¶ms.invocation, + PublicAgentInvocation::AgentMethodInvocation(m) + if m.method_name.replace('-', "_") == "await_promise" + ), + _ => false, + }) + .count(); + let finished = oplog + .iter() + .filter(|entry| match &entry.entry { + PublicOplogEntry::AgentInvocationFinished(params) => params + .method_name + .as_deref() + .is_some_and(|name| name.replace('-', "_") == "await_promise"), + _ => false, + }) + .count(); + assert_eq!( + (started, finished), + (1, 1), + "the retry under the same idempotency key must be answered from the recorded run, not \ + by executing await_promise a second time" + ); Ok(()) } @@ -7295,9 +7360,13 @@ async fn a_caller_is_answered_when_its_agents_shard_comes_back( /// never disturbed, so nothing below the application layer had anything to /// notice. /// -/// What it should get is an error of the `InvalidShardId` family, which is -/// already what worker-service needs to invalidate its routing table and retry -/// against the new owner. That path exists and works; nothing used to reach it. +/// What it should get is an error worker-service answers by invalidating its +/// routing table and retrying against the new owner. Two errors carry that +/// meaning, and which one arrives depends on how the agent was lost: a revoke +/// that reaches this executor gives the agent up and answers its callers with +/// `ShardingNotReady` at once, while a shard that moves without a revoke +/// arriving is caught by the periodic ownership re-check, which reports +/// `InvalidShardId`. Either is a retry; silence is not. #[test] #[tracing::instrument] #[timeout("2m")] @@ -7309,7 +7378,7 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away( ) -> anyhow::Result<()> { let context = TestContext::new(last_unique_id); let executor = start(deps, &context).await?; - let parked = + let mut parked = park_a_caller_on_a_promise(&executor, &context, host_api_tests, "promise-shard-taken") .await?; @@ -7336,7 +7405,9 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away( answer.expect_err("nobody completed the promise, so the only honest answer is an error"); let rendered = format!("{error:#}"); assert!( - rendered.contains("InvalidShardId"), + rendered.contains("ShardingNotReady") + || rendered.contains("Sharding not ready") + || rendered.contains("InvalidShardId"), "the caller has to be told the shard moved, because that is what makes \ worker-service invalidate its routing table and retry against the new \ owner; instead it got: {rendered}" @@ -7478,6 +7549,11 @@ async fn a_caller_is_not_given_up_on_while_the_shard_assignment_is_missing( /// The buffer is shrunk to 16 so a burst of 64 every 50ms is enough to keep /// the receiver behind; with the default 100000 the flood would have to be /// that much larger to say the same thing. +/// +/// The agent is moved by [`fake_ownership`] rather than by a real revoke. A +/// revoke that reaches this executor gives the agent up and answers its callers +/// itself, so the re-check this test exists for would never run and the flood +/// would prove nothing. #[test] #[tracing::instrument] #[timeout("2m")] @@ -7488,14 +7564,12 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away_while_the_even #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, ) -> anyhow::Result<()> { let context = TestContext::new(last_unique_id); - let overrides = TestExecutorOverrides { - configure: Some(Arc::new(|config| { - config.limits.invocation_result_broadcast_capacity = 16; - })), - ..Default::default() - }; + let (mut overrides, controls) = fake_ownership(); + overrides.configure = Some(Arc::new(|config| { + config.limits.invocation_result_broadcast_capacity = 16; + })); let executor = start_with_overrides(deps, &context, overrides).await?; - let parked = park_a_caller_on_a_promise( + let mut parked = park_a_caller_on_a_promise( &executor, &context, host_api_tests, @@ -7503,12 +7577,11 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away_while_the_even ) .await?; - // Taken while the agent is still resident; revoking its shard drops it. let events = executor.event_bus(&parked.agent_id).await?; - info!("Revoking the shard for good, then flooding the event bus with somebody else's news"); + info!("Reporting the agent as moved, then flooding the event bus with somebody else's news"); - revoke_shard_zero(&executor).await?; + controls.pretend_the_agent_moved(); let somebody_else = AgentId { component_id: parked.agent_id.component_id, @@ -7531,6 +7604,10 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away_while_the_even } }); + // Left unread while the flood runs, so the buffer overflows under it however + // the runtime schedules the two. Reading it straight away races the flood: + // a reader on another worker thread can keep pace with it and never lag. + sleep(Duration::from_millis(200)).await; let overflowed = tokio::time::timeout(Duration::from_secs(2), probe.wait_for(|_| None::<()>)).await; if !matches!(overflowed, Ok(Err(RecvError::Lagged(_)))) { @@ -7561,6 +7638,746 @@ async fn a_caller_is_answered_when_its_agents_shard_is_taken_away_while_the_even rendered.contains("InvalidShardId"), "the caller has to be told the shard moved; instead it got: {rendered}" ); + // Without this the test also passes when the answer came from somewhere other than the + // re-check, which is the one path the flood is here to starve. + assert!( + controls.agent_moved_reports() >= 1, + "the ownership re-check never ran while the bus was lagging, so nothing here says the \ + deadline arm survives a starved subscription" + ); + Ok(()) +} + +/// A stop that reaches a generation already given up must leave the generation that replaced it +/// alone. +/// +/// A given-up agent passes through the stop's removal more than once - from its own loop and +/// again from the give-up that waited for it - and a handle kept past its generation can stop it +/// once more. Removal keyed by agent id alone would let any of those passes evict whatever was +/// cached under that id by then: here, the newer generation the shard's return created. +#[test] +#[tracing::instrument] +#[timeout("2m")] +async fn a_stop_through_a_given_up_generation_leaves_the_next_generation_cached( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "stale-generation-stop"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + let stale = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is not cached after its first invocation"))? + .primary(); + + revoke_shard_zero(&executor).await?; + tokio::time::timeout(Duration::from_secs(10), async { + while executor.worker_is_cached(&owned_agent_id).await { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .map_err(|_| anyhow!("the agent stayed cached after its shard was revoked"))?; + + assign_shard_zero(&executor).await?; + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + let fresh = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is not cached after the shard came back"))? + .primary(); + assert!( + !Arc::ptr_eq(&stale, &fresh), + "the shard's return must have created a new generation, or this test proves nothing" + ); + + // The stale generation is already unloaded and given up, so this goes straight to the + // removal. + stale.test_stop().await; + + assert!( + executor.worker_is_cached(&owned_agent_id).await, + "a stop through the given-up generation evicted the generation that replaced it" + ); + let cached = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is no longer cached"))? + .primary(); + assert!( + Arc::ptr_eq(&cached, &fresh), + "the cached generation changed under a stop through a stale handle" + ); + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + Ok(()) +} + +/// A retry scheduled before the shard moved must not resume the agent afterwards. +/// +/// A crash schedules the loop's own restart. If the shard is revoked in that window, the agent has +/// been given up, and the loop's backstop (`is_given_up()` ahead of the retry decision, +/// invocation_loop.rs) has to take the given-up exit instead: no restart here, no `Resumed` written +/// to an oplog the new owner is taking over, and the caller told to reroute. Without the backstop +/// the retry would win the race and resume an agent this executor no longer owns. +#[test] +#[tracing::instrument] +#[timeout(120000)] +async fn a_retry_scheduled_before_a_revoke_does_not_resume_the_agent( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "retry-after-revoke"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + + let invocation = { + let executor = executor.clone(); + let component = component.clone(); + let agent_id = agent_id.clone(); + tokio::spawn( + async move { + executor + .invoke_and_await_agent(&component, &agent_id, "interruption", data_value!()) + .await + } + .in_current_span(), + ) + }; + tokio::time::sleep(Duration::from_secs(5)).await; + + // The crash schedules the loop's restart; the revoke lands while it is pending. + let _ = executor.simulated_crash(&worker_id).await; + revoke_shard_zero(&executor).await?; + + // Given up, so the pending retry must not bring it back: the agent leaves the cache and stays + // out of it. + tokio::time::timeout(Duration::from_secs(30), async { + while executor.worker_is_cached(&owned_agent_id).await { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .map_err(|_| anyhow!("the agent stayed cached after its shard was revoked"))?; + + let result = tokio::time::timeout(Duration::from_secs(30), invocation) + .await + .map_err(|_| anyhow!("the caller was never answered after the revoke"))??; + assert!( + result.is_err(), + "the invocation must be handed back to the caller to reroute, not completed by an \ + executor that no longer owns the shard" + ); + + sleep(Duration::from_secs(2)).await; + assert!( + !executor.worker_is_cached(&owned_agent_id).await, + "a retry scheduled before the revoke resumed an agent this executor had given up" + ); + + // The shard coming back is what may start it again, from the oplog, as a new generation. + assign_shard_zero(&executor).await?; + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + Ok(()) +} + +/// The other half of the stale-handle contract: a handle kept past its generation must not be able +/// to *start* it either. +/// +/// The stop-side test above pins that a stale handle cannot evict the generation that replaced it. +/// This one pins the guard at the other end (`start_if_needed_internal`, worker/mod.rs): a start +/// through a given-up generation would take permits, could append `Resumed` to an oplog the new +/// owner is now writing, and would publish its failures, by agent id, to the waiters of the +/// generation that replaced it. The caller gets a retriable error instead, and the live generation +/// is untouched. +#[test] +#[tracing::instrument] +async fn a_start_through_a_given_up_generation_is_refused( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "stale-generation-start"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + let stale = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is not cached after its first invocation"))? + .primary(); + + revoke_shard_zero(&executor).await?; + tokio::time::timeout(Duration::from_secs(10), async { + while executor.worker_is_cached(&owned_agent_id).await { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .map_err(|_| anyhow!("the agent stayed cached after its shard was revoked"))?; + + assign_shard_zero(&executor).await?; + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + let fresh = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is not cached after the shard came back"))? + .primary(); + assert!( + !Arc::ptr_eq(&stale, &fresh), + "the shard's return must have created a new generation, or this test proves nothing" + ); + + let refused = Worker::start_if_needed(stale.clone()).await; + match refused { + Err(WorkerExecutorError::ShardingNotReady | WorkerExecutorError::OplogFenced { .. }) => {} + Err(other) => panic!( + "a start through a given-up generation must be answered with something the caller can \ + retry on the new owner, got {other}" + ), + Ok(_) => panic!("a start through a given-up generation was allowed"), + } + + let cached = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is no longer cached"))? + .primary(); + assert!( + Arc::ptr_eq(&cached, &fresh), + "the cached generation changed under a start through a stale handle" + ); + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + Ok(()) +} + +/// A caller awaiting an invocation whose oplog write was refused inside a host call must be told +/// to reroute. +/// +/// A fence found in a host call surfaces as a `ShardLost` trap: the agent marks itself +/// given up and its loop stops without failing anyone. The executor's own assignment still +/// names the shard - this is the zombie, and nobody has told it - so the waiter's ownership +/// re-check keeps passing. Two things can answer the caller: the give-up spawned when the loop's +/// exit commit is refused, and the loop's own stop. The spawned one misses the caller whenever the +/// loop removes the agent before it looks, and nothing in a test can hold it back, so this pins +/// the outcome rather than that ordering. +/// +/// The takeover lands while the guest is parked in `poll` inside `sleep_for`, after +/// `subscribe_duration` has committed through the status actor. The first write after it is then +/// the gated monotonic-clock hook's own commit when the guest reads the elapsed time: a refusal in +/// the host call, not in the status actor. +#[test] +#[tracing::instrument] +#[timeout("2m")] +async fn a_caller_waiting_on_an_invocation_fenced_inside_a_host_call_is_told_to_reroute( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + use golem_common::model::oplog::PublicOplogEntry; + + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "fenced-inside-a-host-call"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + + let mut caller = { + let executor = executor.clone(); + let component = component.clone(); + let agent_id = agent_id.clone(); + tokio::spawn( + async move { + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(6.0f64)) + .await + } + .in_current_span(), + ) + }; + executor + .wait_for_status(&worker_id, AgentStatus::Running, Duration::from_secs(10)) + .await?; + + tokio::time::timeout(Duration::from_secs(10), async { + loop { + let oplog = executor.get_oplog(&worker_id, OplogIndex::INITIAL).await?; + let subscribed = oplog.iter().any(|entry| match &entry.entry { + PublicOplogEntry::End(end) => oplog.iter().any(|start| { + start.oplog_index == end.start_index + && matches!( + &start.entry, + PublicOplogEntry::Start(params) + if params.function_name == "monotonic_clock::subscribe_duration" + ) + }), + _ => false, + }); + if subscribed { + return anyhow::Ok(()); + } + sleep(Duration::from_millis(50)).await; + } + }) + .await + .map_err(|_| anyhow!("the guest never subscribed to its sleep"))??; + // Lets `subscribe_duration`'s own commit finish, so the guest is parked in `poll` with about + // five seconds of sleep left and nothing else is due to write. + sleep(Duration::from_secs(1)).await; + // Baseline for the no-further-progress check below: from here on, the guest's only next + // durable operation is the monotonic-clock read that must be refused. + let oplog_before_takeover = executor.get_oplog(&worker_id, OplogIndex::INITIAL).await?; + + let mut clock = executor + .gate_next_monotonic_clock_start(&owned_agent_id) + .await?; + take_agent_oplog_over_at_epoch(deps, &context, &owned_agent_id, 1).await?; + + let answer = match tokio::time::timeout(Duration::from_secs(30), &mut caller).await { + Ok(joined) => joined?, + Err(_) => { + caller.abort(); + bail!( + "the caller was never answered, although its agent's oplog has a new owner and \ + this executor gave the agent up" + ); + } + }; + info!(result = ?answer, "caller was answered"); + let error = + answer.expect_err("the invocation's writes were refused, so it cannot have a value"); + let rendered = format!("{error:#}"); + assert!( + rendered.contains("ShardingNotReady") || rendered.contains("Sharding not ready"), + "the caller has to be given the error worker-service reroutes on; instead it got: \ + {rendered}" + ); + + tokio::time::timeout(Duration::from_secs(10), async { + while executor.worker_is_cached(&owned_agent_id).await { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .map_err(|_| anyhow!("the fenced agent stayed cached on this executor"))?; + + // `entered()` only fires once the gated commit *succeeds* (`OwnerExecution:: + // test_after_monotonic_clock_start` sends it after the commit, not before), so its timing out + // here does not by itself prove the guest ever reached the gate: the same timeout would be + // observed if the agent had been interrupted by something else first, well before the + // monotonic-clock read. Prove reachability directly from the oplog instead: nothing may follow + // `subscribe_duration`'s `End` on this executor once the shard is taken away, since the + // guest's only next durable operation is the monotonic-clock read the fence must have refused. + let oplog_after = executor.get_oplog(&worker_id, OplogIndex::INITIAL).await?; + assert_eq!( + oplog_after.len(), + oplog_before_takeover.len(), + "no oplog entries may follow `subscribe_duration`'s `End` once the shard is taken away; \ + the guest's next durable call is the monotonic-clock read the fence must have refused" + ); + assert!( + tokio::time::timeout(Duration::from_millis(500), clock.entered()) + .await + .is_err(), + "the gated clock commit went through, so the refusal was not the host call's" + ); + Ok(()) +} + +/// An invocation enqueued onto an oplog that has a new owner must be refused, not accepted. +/// +/// Enqueueing buffers the pending-invocation entry and commits it through the status actor, and +/// that commit is where the takeover is found. A refusal folded into "status unchanged" would +/// report the enqueue a success for a key that never reached the status, and neither the give-up +/// the refusal spawns nor the stop's removal fails keys the status does not hold - so nothing +/// would ever answer the caller. +#[test] +#[tracing::instrument] +#[timeout("2m")] +async fn an_invocation_enqueued_onto_a_fenced_oplog_is_refused_rather_than_accepted( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "fenced-enqueue"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + let idle = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the agent is not cached after its first invocation"))? + .primary(); + + take_agent_oplog_over_at_epoch(deps, &context, &owned_agent_id, 1).await?; + + // The refusal has to come from the enqueue's own commit. Had anything written to the idle + // agent first, the give-up that write spawned could evict it, and the invocation would then + // be refused at a fresh generation's open instead, which proves nothing about the enqueue. + let cached = executor + .active_agent(&owned_agent_id) + .await + .ok_or_else(|| anyhow!("the idle agent left the cache before the enqueue reached it"))? + .primary(); + assert!( + Arc::ptr_eq(&idle, &cached), + "the generation that opened the oplog at the old epoch is no longer the cached one" + ); + + let answer = tokio::time::timeout( + Duration::from_secs(10), + executor.invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)), + ) + .await + .map_err(|_| { + anyhow!( + "the caller was never answered: the invocation was accepted onto an oplog whose \ + pending entry the storage refused" + ) + })?; + info!(result = ?answer, "caller was answered"); + let error = answer.expect_err("the pending entry was never committed, so there is no value"); + let rendered = format!("{error:#}"); + assert!( + // Refused before acceptance, the fence arrives as a rejection carrying the error the + // worker service reroutes on. + rendered.contains("Sharding not ready") || rendered.contains("fenced"), + "the caller has to be given the error worker-service reroutes on; instead it got: \ + {rendered}" + ); + + tokio::time::timeout(Duration::from_secs(10), async { + while executor.worker_is_cached(&owned_agent_id).await { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .map_err(|_| anyhow!("the fenced agent stayed cached on this executor"))?; + Ok(()) +} + +#[test] +#[tracing::instrument] +async fn a_deletion_that_outlived_the_shard_leaves_the_agent_to_its_new_owner( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let name = agent_id!("Clocks", "deleted-after-its-shard-moved"); + let worker_id = executor.start_agent(&component.id, name.clone()).await?; + executor + .invoke_and_await_agent(&component, &name, "sleep_for", data_value!(0.0f64)) + .await?; + let owned = OwnedAgentId::new(context.default_environment_id, &worker_id); + + let hook = Arc::new(DeletionStageHook::new( + owned.clone(), + Some(WorkerDeletionStage::DurableStateRemoved), + None, + )); + executor.set_worker_deletion_hook(hook.clone()); + let deleting = tokio::spawn({ + let executor = executor.clone(); + let worker_id = worker_id.clone(); + async move { executor.delete_worker(&worker_id).await }.in_current_span() + }); + tokio::time::timeout(Duration::from_secs(10), hook.wait_until_gated()) + .await + .map_err(|_| anyhow!("the deletion never reached the removal of the durable state"))?; + + // The deletion was accepted while this executor owned the shard. Before it gets to the agent's + // state the shard moves on, and the new owner takes the oplog over. + take_agent_oplog_over_at_epoch(deps, &context, &owned, 1).await?; + let entries = agent_oplog_length(deps, &context, &owned).await?; + assert!(entries > 0); + hook.release(); + + let result = tokio::time::timeout(Duration::from_secs(30), deleting).await??; + let rendered = format!( + "{:#}", + result.expect_err("the deletion belongs to the shard's new owner now") + ); + assert!( + rendered.contains("ShardingNotReady") + || rendered.contains("Sharding not ready") + || rendered.contains("fenced"), + "the caller has to be sent to the new owner; instead it got: {rendered}" + ); + assert_eq!( + agent_oplog_length(deps, &context, &owned).await?, + entries, + "a deletion that lost the shard removes nothing from the new owner's oplog" + ); + assert!( + executor.active_agent(&owned).await.is_none(), + "this executor still holds an agent it has given up" + ); + Ok(()) +} + +/// A deletion removes the agent's oplog first and its cached status after. When the second step +/// fails, the retry has to finish the job: the oplog delete it repeats asserts this executor's +/// epoch against a key whose record the first attempt already removed, and that has to count as +/// deleted rather than as a key another executor took over. +#[test] +#[tracing::instrument] +#[timeout("2m")] +async fn a_deletion_retried_after_it_removed_the_oplog_finishes_the_cleanup( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + use golem_worker_executor::storage::keyvalue::KeyValueStorageError; + use golem_worker_executor::storage::keyvalue::fault_injecting::{ + FaultInjectingKeyValueStorage, KeyValueStorageFaults, + }; + + let context = TestContext::new(last_unique_id); + let faults = KeyValueStorageFaults::default(); + let executor = start_with_overrides( + deps, + &context, + TestExecutorOverrides { + wrap_key_value_storage: Some(Arc::new({ + let faults = faults.clone(); + move |storage| Arc::new(FaultInjectingKeyValueStorage::new(storage, faults.clone())) + })), + ..TestExecutorOverrides::default() + }, + ) + .await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let name = agent_id!("Clocks", "deletion-retried-after-the-oplog-went"); + let worker_id = executor.start_agent(&component.id, name.clone()).await?; + executor + .invoke_and_await_agent(&component, &name, "sleep_for", data_value!(0.0f64)) + .await?; + let owned = OwnedAgentId::new(context.default_environment_id, &worker_id); + + // Armed once the deletion reaches the durable state, so the failure lands between the oplog + // delete and the removal of the cached status rather than in an earlier stage. + let hook = Arc::new(DeletionStageHook::new( + owned.clone(), + Some(WorkerDeletionStage::DurableStateRemoved), + None, + )); + executor.set_worker_deletion_hook(hook.clone()); + let deleting = tokio::spawn({ + let executor = executor.clone(); + let worker_id = worker_id.clone(); + async move { executor.delete_worker(&worker_id).await }.in_current_span() + }); + tokio::time::timeout(Duration::from_secs(10), hook.wait_until_gated()) + .await + .map_err(|_| anyhow!("the deletion never reached the removal of the durable state"))?; + faults.fail( + "remove", + 1, + KeyValueStorageError::Other("injected: key-value storage unavailable".to_string()), + ); + hook.release(); + + let first = tokio::time::timeout(Duration::from_secs(30), deleting).await??; + assert!( + first.is_err(), + "the injected failure has to fail the first attempt" + ); + assert_eq!( + agent_oplog_length(deps, &context, &owned).await?, + 0, + "the first attempt failed after it had removed the oplog" + ); + + executor.delete_worker(&worker_id).await?; + assert!(executor.get_worker_metadata(&worker_id).await.is_err()); + Ok(()) +} + +/// A stop can be the first write to find that the shard has a new owner. Its final commit is +/// then refused, and the waiters of the invocations it cuts short have to be told to retry on the +/// new owner - not handed the stop's own error, which is cached as a failure the new owner's run +/// of the invocation never corrects. +/// +/// The invocation is paused with its `AgentInvocationStarted` buffered and not yet committed, +/// so the stop's commit is not empty: an empty commit queries nothing and finds no fence. +#[test] +#[tracing::instrument] +#[timeout("2m")] +async fn a_stop_that_finds_the_fence_on_its_own_commit_tells_the_caller_to_reroute( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + _tracing: &Tracing, + #[tagged_as("host_api_tests")] host_api_tests: &PrecompiledComponent, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + let component = executor + .component_dep(&context.default_environment_id, host_api_tests) + .store() + .await?; + let agent_id = agent_id!("Clocks", "fenced-on-the-stops-commit"); + let worker_id = executor + .start_agent(&component.id, agent_id.clone()) + .await?; + let owned_agent_id = OwnedAgentId::new(context.default_environment_id, &worker_id); + executor + .invoke_and_await_agent(&component, &agent_id, "sleep_for", data_value!(0.0f64)) + .await?; + + let mut started = executor + .gate_next_invocation_started(&owned_agent_id) + .await?; + let idempotency_key = IdempotencyKey::fresh(); + let mut caller = { + let executor = executor.clone(); + let component = component.clone(); + let agent_id = agent_id.clone(); + let idempotency_key = idempotency_key.clone(); + tokio::spawn( + async move { + executor + .invoke_and_await_agent_with_key( + &component, + &agent_id, + &idempotency_key, + "sleep_for", + data_value!(0.0f64), + ) + .await + } + .in_current_span(), + ) + }; + tokio::time::timeout(Duration::from_secs(10), started.entered()) + .await + .map_err(|_| anyhow!("the invocation never buffered its start entry"))?; + + take_agent_oplog_over_at_epoch(deps, &context, &owned_agent_id, 1).await?; + // An external stop carrying an error of its own, which the waiters must not be given. + let deletion = { + let executor = executor.clone(); + let worker_id = worker_id.clone(); + tokio::spawn(async move { executor.delete_worker(&worker_id).await }.in_current_span()) + }; + // The stop waits for the paused guest; let it go once the stop is under way. + sleep(Duration::from_millis(500)).await; + started.release(); + let _ = tokio::time::timeout(Duration::from_secs(30), deletion).await; + + let answer = match tokio::time::timeout(Duration::from_secs(30), &mut caller).await { + Ok(joined) => joined?, + Err(_) => { + caller.abort(); + bail!("the caller was never answered"); + } + }; + info!(result = ?answer, "caller was answered"); + let error = answer.expect_err("the invocation moved to the shard's new owner"); + let rendered = format!("{error:#}"); + assert!( + rendered.contains("Sharding not ready") || rendered.contains("fenced"), + "the caller has to be told to retry on the new owner; instead it got: {rendered}" + ); + assert!( + !rendered.contains("deleted"), + "the stop's own error reached the caller: {rendered}" + ); + + // Nothing was cached for the key. A retry on this executor is turned away at admission - the + // agent is being deleted here - but never answered from a cached failure, which comes back as + // an invocation failure rather than a rejection. + let retried = executor + .invoke_and_await_agent_with_key( + &component, + &agent_id, + &idempotency_key, + "sleep_for", + data_value!(0.0f64), + ) + .await; + if let Err(error) = &retried { + let rendered = format!("{error:#}"); + assert!( + !(rendered.contains("invocation failed") && rendered.contains("deleted")), + "the retry was answered from a cached failure: {rendered}" + ); + } Ok(()) } @@ -7576,6 +8393,7 @@ async fn revoke_shard_zero(executor: &TestWorkerExecutor) -> anyhow::Result<()> .revoke_shards(RevokeShardsRequest { shard_ids: vec![ShardId { value: 0 }], revision: 1, + incarnation_id: String::new(), }) .await?; Ok(()) @@ -7596,6 +8414,7 @@ async fn assign_shard_zero(executor: &TestWorkerExecutor) -> anyhow::Result<()> }], number_of_shards: 1, revision: 2, + incarnation_id: String::new(), }) .await?; Ok(()) @@ -7631,17 +8450,25 @@ async fn park_a_caller_on_a_promise( let promise_data = crate::raw_params(vec![promise_id_value.clone()]); + // Parked under an explicit key so a test can reissue the call the way worker-service reissues + // one it was told to reroute: the retry lands on this same invocation instead of starting a + // second run of it. + let idempotency_key = IdempotencyKey::fresh(); + let executor_clone = executor.clone(); let component_clone = component.clone(); let agent_id_clone = agent_id.clone(); + let key_clone = idempotency_key.clone(); + let params = promise_data.clone(); let fiber = tokio::spawn( async move { executor_clone - .invoke_and_await_agent( + .invoke_and_await_agent_with_key( &component_clone, &agent_id_clone, + &key_clone, "await_promise", - promise_data, + params, ) .await } @@ -7656,6 +8483,10 @@ async fn park_a_caller_on_a_promise( agent_id: worker_id, promise_id: promise_id_value, caller: fiber, + component, + parsed_agent_id: agent_id, + idempotency_key, + promise_data, }) } @@ -7665,13 +8496,18 @@ struct ParkedCaller { agent_id: AgentId, promise_id: SchemaValue, caller: JoinHandle>, + /// What it takes to reissue the parked call under its own idempotency key. + component: ComponentDto, + parsed_agent_id: golem_common::model::agent::ParsedAgentId, + idempotency_key: IdempotencyKey, + promise_data: golem_common::schema::TypedSchemaValue, } impl ParkedCaller { /// Waits for the parked call to come back. The outer result says whether it /// was answered at all; the inner one is what it was told. async fn answer_within( - mut self, + &mut self, patience: Duration, gave_up: &str, ) -> anyhow::Result> { @@ -7704,9 +8540,42 @@ impl ParkedCaller { Ok(()) } + /// Reissues the parked call under its original idempotency key, the way + /// worker-service reissues one an executor told it to reroute, and returns + /// the value that retry is given. + /// + /// The key is what makes this a retry rather than a second run: an + /// invocation already recorded under it is not executed again, so the answer + /// here is the answer to the call that was parked. + async fn retry_within( + &self, + executor: &TestWorkerExecutor, + patience: Duration, + gave_up: &str, + ) -> anyhow::Result { + tokio::time::timeout( + patience, + executor.invoke_and_await_agent_with_key( + &self.component, + &self.parsed_agent_id, + &self.idempotency_key, + "await_promise", + self.promise_data.clone(), + ), + ) + .await + .map_err(|_| anyhow!("{gave_up}"))?? + .into_return_value() + .ok_or_else(|| anyhow!("expected return value")) + } + /// Waits for the parked call and returns the value it was given, failing if /// it was not answered in time or was answered with an error. - async fn value_within(self, patience: Duration, gave_up: &str) -> anyhow::Result { + async fn value_within( + mut self, + patience: Duration, + gave_up: &str, + ) -> anyhow::Result { self.answer_within(patience, gave_up) .await?? .into_return_value() diff --git a/golem-worker-executor/tests/hot_update.rs b/golem-worker-executor/tests/hot_update.rs index 9f491850e6..6259afbd91 100644 --- a/golem-worker-executor/tests/hot_update.rs +++ b/golem-worker-executor/tests/hot_update.rs @@ -1595,6 +1595,107 @@ async fn manual_update_on_idle( Ok(()) } +/// A stop arriving while a manual update is in flight must not deadlock either side. +/// +/// The update is enqueued from the invocation loop, and a stop taking the same worker down could +/// wait on the loop that was waiting to enqueue. +/// The enqueue is non-blocking now (`enqueue_update_from_loop`), so both finish. The test is +/// written as a race rather than a fixed order - either outcome is legal, a hang is not - and the +/// timeout is the assertion. +#[test] +#[tracing::instrument] +#[timeout(120000)] +async fn a_stop_racing_a_manual_update_never_deadlocks( + last_unique_id: &LastUniqueId, + deps: &WorkerExecutorTestDependencies, + #[tagged_as("agent_update_v2")] agent_update_v2: &PrecompiledComponent, + _tracing: &Tracing, +) -> anyhow::Result<()> { + let context = TestContext::new(last_unique_id); + let executor = start(deps, &context).await?; + + let http_server = TestHttpServer::start().await; + let mut env = HashMap::new(); + env.insert("PORT".to_string(), http_server.port().to_string()); + + let component = executor + .component_dep(&context.default_environment_id, agent_update_v2) + .store() + .await?; + let agent_id = agent_id!("UpdateTest"); + let worker_id = executor + .start_agent_with(&component.id, agent_id.clone(), env, Vec::new()) + .await?; + let mut _log_output_guards = Vec::new(); + _log_output_guards.push(executor.log_output_scoped(&worker_id).await?); + + let updated_component = executor + .update_component(&component.id, "it_agent_update_v3_release") + .await?; + + executor + .invoke_and_await_agent(&component, &agent_id, "f1", data_value!(0u64)) + .await?; + + // Both are issued without awaiting the first: the update goes through the loop-side enqueue + // while the interrupt takes the worker down under it. + let update = { + let executor = executor.clone(); + let worker_id = worker_id.clone(); + let revision = updated_component.revision; + spawn( + async move { + executor + .manual_update_worker(&worker_id, revision, false) + .await + } + .in_current_span(), + ) + }; + let stop = { + let executor = executor.clone(); + let worker_id = worker_id.clone(); + spawn(async move { executor.interrupt(&worker_id).await }.in_current_span()) + }; + + let (update, stop) = tokio::time::timeout(Duration::from_secs(60), async { + tokio::join!(update, stop) + }) + .await + .map_err(|_| { + anyhow::anyhow!("a stop racing a manual update deadlocked: neither call came back") + })?; + // Either may fail on its own terms - the worker is being stopped - but neither may hang, and + // the executor has to stay usable afterwards. + let _ = update?; + let _ = stop?; + + // The worker is still answerable, and on a revision that is one of the two legal outcomes - + // the method set differs between them, so the probe is the metadata rather than a call. + let metadata = tokio::time::timeout( + Duration::from_secs(60), + executor.get_worker_metadata(&worker_id), + ) + .await + .map_err(|_| anyhow::anyhow!("the agent never answered again after the race"))??; + assert!( + metadata.component_revision == updated_component.revision + || metadata.component_revision == ComponentRevision::INITIAL, + "the update either landed or did not, but the revision must be one of the two, got {:?}", + metadata.component_revision + ); + assert!( + !matches!(metadata.status, AgentStatus::Failed), + "a stop racing an update must not fail the agent, got {:?}", + metadata.status + ); + executor.check_oplog_is_queryable(&worker_id).await?; + + drop(executor); + http_server.abort(); + Ok(()) +} + #[test] #[tracing::instrument] async fn manual_update_on_idle_without_save_snapshot( diff --git a/golem-worker-executor/tests/indexed_storage.rs b/golem-worker-executor/tests/indexed_storage.rs index 355506a104..be19b62aa4 100644 --- a/golem-worker-executor/tests/indexed_storage.rs +++ b/golem-worker-executor/tests/indexed_storage.rs @@ -16,6 +16,7 @@ use async_trait::async_trait; use bytes::Bytes; use golem_common::config::{DbPostgresConfig, RedisConfig}; use golem_common::model::AgentId; +use golem_common::model::ShardEpoch; use golem_common::model::agent::AgentMode; use golem_common::model::component::ComponentId; use golem_common::redis::RedisPool; @@ -29,7 +30,7 @@ use golem_worker_executor::storage::indexed::redis::RedisIndexedStorage; use golem_worker_executor::storage::indexed::sqlite::SqliteIndexedStorage; use golem_worker_executor::storage::indexed::{ IndexedStorage, IndexedStorageError, IndexedStorageLabelledApi, IndexedStorageMetaNamespace, - IndexedStorageNamespace, ScanResume, + IndexedStorageNamespace, ScanResume, WriterId, }; use golem_worker_executor_test_utils::WorkerExecutorTestDependencies; use pretty_assertions::assert_eq; @@ -43,6 +44,15 @@ use uuid::Uuid; #[async_trait] trait GetIndexedStorage: Debug { async fn get_indexed_storage(&self) -> Arc; + + /// Two handles onto the SAME store, writing as two different processes - what a shard manager + /// that lost its state can produce by minting one epoch twice. + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ); } struct InMemoryIndexedStorageWrapper; @@ -59,6 +69,18 @@ impl GetIndexedStorage for InMemoryIndexedStorageWrapper { let kvs = InMemoryIndexedStorage::new(); Arc::new(kvs) } + + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ) { + let store = InMemoryIndexedStorage::new(); + let first = store.for_writer(WriterId(Uuid::new_v4())); + let second = store.for_writer(WriterId(Uuid::new_v4())); + (Arc::new(first), Arc::new(second)) + } } #[test_dep(scope = Shared, tagged_as = "in_memory")] @@ -81,23 +103,42 @@ impl Debug for RedisIndexedStorageWrapper { #[async_trait] impl GetIndexedStorage for RedisIndexedStorageWrapper { async fn get_indexed_storage(&self) -> Arc { - let random_prefix = Uuid::new_v4(); - let redis_pool = RedisPool::configured(&RedisConfig { + Arc::new(RedisIndexedStorage::new( + self.pool(&Uuid::new_v4().to_string()).await, + )) + } + + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ) { + let prefix = Uuid::new_v4().to_string(); + let first = + RedisIndexedStorage::new(self.pool(&prefix).await).for_writer(WriterId(Uuid::new_v4())); + let second = + RedisIndexedStorage::new(self.pool(&prefix).await).for_writer(WriterId(Uuid::new_v4())); + (Arc::new(first), Arc::new(second)) + } +} + +impl RedisIndexedStorageWrapper { + async fn pool(&self, key_prefix: &str) -> RedisPool { + RedisPool::configured(&RedisConfig { host: self.redis.public_host(), port: self.redis.public_port(), database: 0, tracing: false, pool_size: 1, retries: Default::default(), - key_prefix: random_prefix.to_string(), + key_prefix: key_prefix.to_string(), username: None, password: None, tls: false, }) .await - .unwrap(); - let kvs = RedisIndexedStorage::new(redis_pool); - Arc::new(kvs) + .unwrap() } } @@ -148,6 +189,35 @@ impl GetIndexedStorage for SqliteIndexedStorageWrapper { let sis = SqliteIndexedStorage::configured(&config).await.unwrap(); Arc::new(sis) } + + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ) { + let tempdir = tempfile::tempdir().unwrap(); + let database = tempdir + .path() + .join("indexed.db") + .to_string_lossy() + .into_owned(); + self.tempdirs.lock().unwrap().push(tempdir); + let config = golem_common::config::DbSqliteConfig { + database, + max_connections: 10, + foreign_keys: false, + }; + let first = SqliteIndexedStorage::configured(&config) + .await + .unwrap() + .for_writer(WriterId(Uuid::new_v4())); + let second = SqliteIndexedStorage::configured(&config) + .await + .unwrap() + .for_writer(WriterId(Uuid::new_v4())); + (Arc::new(first), Arc::new(second)) + } } #[test_dep(scope = Shared, tagged_as = "sqlite")] @@ -185,6 +255,22 @@ impl GetIndexedStorage for MultiSqliteIndexedStorageWrapper { let storage = MultiSqliteIndexedStorage::new(&path, 10, true); Arc::new(storage) } + + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ) { + let tempdir = tempfile::tempdir().unwrap(); + let path = tempdir.path().to_path_buf(); + self.tempdirs.lock().unwrap().push(tempdir); + let first = + MultiSqliteIndexedStorage::new(&path, 10, true).for_writer(WriterId(Uuid::new_v4())); + let second = + MultiSqliteIndexedStorage::new(&path, 10, true).for_writer(WriterId(Uuid::new_v4())); + (Arc::new(first), Arc::new(second)) + } } #[test_dep(scope = Shared, tagged_as = "multi_sqlite")] @@ -204,9 +290,10 @@ impl Debug for PostgresIndexedStorageWrapper { } } -#[async_trait] -impl GetIndexedStorage for PostgresIndexedStorageWrapper { - async fn get_indexed_storage(&self) -> Arc { +impl PostgresIndexedStorageWrapper { + /// A fresh database, and the config that reaches it. Separated from `get_indexed_storage` so a + /// second storage can be opened onto the same database as a different writer. + async fn fresh_database(&self) -> IndexedStoragePostgresConfig { let db_name = format!("idx_{}", Uuid::new_v4().simple()); let admin_pool = sqlx::postgres::PgPoolOptions::new() @@ -234,18 +321,42 @@ impl GetIndexedStorage for PostgresIndexedStorageWrapper { acquire_timeout: None, }; - let config = IndexedStoragePostgresConfig { + IndexedStoragePostgresConfig { postgres, drop_prefix_delete_batch_size: 1024, max_concurrent_ops: None, - }; + } + } +} +#[async_trait] +impl GetIndexedStorage for PostgresIndexedStorageWrapper { + async fn get_indexed_storage(&self) -> Arc { + let config = self.fresh_database().await; let storage = PostgresIndexedStorage::configured(&config) .await .expect("Cannot create postgres indexed storage"); Arc::new(storage) } + + async fn get_two_writers( + &self, + ) -> ( + Arc, + Arc, + ) { + let config = self.fresh_database().await; + let first = PostgresIndexedStorage::configured(&config) + .await + .expect("Cannot create postgres indexed storage") + .for_writer(WriterId(Uuid::new_v4())); + let second = PostgresIndexedStorage::configured(&config) + .await + .expect("Cannot create postgres indexed storage") + .for_writer(WriterId(Uuid::new_v4())); + (Arc::new(first), Arc::new(second)) + } } #[test_dep(scope = Shared, tagged_as = "postgres")] @@ -356,6 +467,7 @@ async fn staged_publication_preserves_atomic_visibility( key, id, vec![id as u8], + None, ) .await .unwrap(); @@ -393,6 +505,7 @@ async fn staged_publication_preserves_atomic_visibility( .map(|id| (id, Bytes::from(vec![value, id as u8]))) .collect::>() .into(), + None, ) .await .unwrap(); @@ -495,6 +608,7 @@ async fn staged_publication_preserves_atomic_visibility( key, first_id, vec![61], + None, ) .await .unwrap(); @@ -530,7 +644,16 @@ async fn staged_publication_preserves_atomic_visibility( .is_none() ); storage - .append("test", "stage", "entry", staged.clone(), key, 1, vec![23]) + .append( + "test", + "stage", + "entry", + staged.clone(), + key, + 1, + vec![23], + None, + ) .await .unwrap(); assert!( @@ -599,6 +722,7 @@ async fn staged_publication_preserves_atomic_visibility( "race", 1, vec![51], + None, ) .await .unwrap(); @@ -619,7 +743,8 @@ async fn staged_publication_preserves_atomic_visibility( visible.clone(), "ordinary", 1, - vec![77] + vec![77], + None, ), ); let published = published.unwrap(); @@ -641,103 +766,120 @@ async fn postgres_singleton_append_many_preserves_storage_contract( ) { let storage = storage.get_indexed_storage().await; let value = Bytes::from_static(&[0, 255, 17, 3]); - for ns in [primary, compressed] { - storage - .append_many("svc", "api", "entity", &ns.ns, "singleton", Arc::from([])) - .await - .unwrap(); - assert!( - !storage - .exists("svc", "api", ns.ns.clone(), "singleton") - .await - .unwrap() - ); - storage - .append_many( - "svc", - "api", - "entity", - &ns.ns, - "singleton", - Arc::from([(17, value.clone())]), - ) - .await - .unwrap(); - assert_eq!( + // Once asserting no epoch, which is a lone autocommit INSERT, and once asserting the recorded + // one, which goes through the fenced transaction. A caller must not be able to tell the two + // paths apart. + for (key, shard_epoch) in [ + ("singleton", None), + ("fenced-singleton", Some(ShardEpoch(1))), + ] { + for ns in [primary, compressed] { + if let Some(epoch) = shard_epoch { + storage + .set_key_epoch("svc", "api", ns.ns.clone(), key, epoch) + .await + .unwrap(); + } storage - .read("svc", "api", "entity", ns.ns.clone(), "singleton", 0, 100) + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([]), + shard_epoch, + ) .await - .unwrap(), - vec![(17, value.to_vec())] - ); - assert_eq!( + .unwrap(); + assert!( + !storage + .exists("svc", "api", ns.ns.clone(), key) + .await + .unwrap() + ); storage - .length("svc", "api", ns.ns.clone(), "singleton") + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(17, value.clone())]), + shard_epoch, + ) .await - .unwrap(), - 1 - ); - assert_eq!( - storage - .last("svc", "api", "entity", ns.ns.clone(), "singleton") + .unwrap(); + assert_eq!( + storage + .read("svc", "api", "entity", ns.ns.clone(), key, 0, 100) + .await + .unwrap(), + vec![(17, value.to_vec())] + ); + assert_eq!( + storage + .length("svc", "api", ns.ns.clone(), key) + .await + .unwrap(), + 1 + ); + assert_eq!( + storage + .last("svc", "api", "entity", ns.ns.clone(), key) + .await + .unwrap(), + Some((17, value.to_vec())) + ); + let (_, keys) = storage + .scan_stable("svc", "api", ns.meta.clone(), Some(key), None, 10) .await - .unwrap(), - Some((17, value.to_vec())) - ); - let (_, keys) = storage - .scan_stable("svc", "api", ns.meta.clone(), Some("singleton"), None, 10) - .await - .unwrap(); - assert_eq!(keys, vec!["singleton".to_string()]); + .unwrap(); + assert_eq!(keys, vec![key.to_string()]); + assert!(matches!( + storage + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(u64::MAX, value.clone())]), + shard_epoch, + ) + .await, + Err(IndexedStorageError::Other(_)) + )); + assert_eq!( + storage + .length("svc", "api", ns.ns.clone(), key) + .await + .unwrap(), + 1 + ); + } assert!(matches!( storage .append_many( "svc", "api", "entity", - &ns.ns, - "singleton", - Arc::from([(u64::MAX, value.clone())]) + &primary.ns, + key, + Arc::from([(17, Bytes::from_static(b"replacement"))]), + shard_epoch, ) .await, - Err(IndexedStorageError::Other(_)) + Err(IndexedStorageError::Conflict(_)) )); assert_eq!( storage - .length("svc", "api", ns.ns.clone(), "singleton") + .read("svc", "api", "entity", primary.ns.clone(), key, 0, 100) .await .unwrap(), - 1 + vec![(17, value.to_vec())] ); } - assert!(matches!( - storage - .append_many( - "svc", - "api", - "entity", - &primary.ns, - "singleton", - Arc::from([(17, Bytes::from_static(b"replacement"))]) - ) - .await, - Err(IndexedStorageError::Conflict(_)) - )); - assert_eq!( - storage - .read( - "svc", - "api", - "entity", - primary.ns.clone(), - "singleton", - 0, - 100 - ) - .await - .unwrap(), - vec![(17, value.to_vec())] - ); } #[test] @@ -755,6 +897,7 @@ async fn postgres_append_many_rolls_back_across_statement_chunks( "atomic", 1025, b"original".to_vec(), + None, ) .await .unwrap(); @@ -764,7 +907,7 @@ async fn postgres_append_many_rolls_back_across_statement_chunks( .into(); assert!(matches!( storage - .append_many("svc", "api", "entity", &ns.ns, "atomic", pairs) + .append_many("svc", "api", "entity", &ns.ns, "atomic", pairs, None) .await, Err(IndexedStorageError::Conflict(_)) )); @@ -791,7 +934,7 @@ async fn exists_append( let value1 = "value1".as_bytes().to_vec(); let result1 = is.exists("svc", "api", ns.ns.clone(), key1).await.unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); let result2 = is.exists("svc", "api", ns.ns.clone(), key1).await.unwrap(); @@ -813,9 +956,18 @@ async fn namespaces_are_separate( let key1 = "key1"; let value1 = "value1".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns1.ns.clone(), key1, 1, value1) - .await - .unwrap(); + is.append( + "svc", + "api", + "entity", + ns1.ns.clone(), + key1, + 1, + value1, + None, + ) + .await + .unwrap(); let result = is.exists("svc", "api", ns2.ns.clone(), key1).await.unwrap(); assert_eq!(result, false); @@ -844,6 +996,7 @@ async fn can_append_and_get( key1, 1, value1.clone(), + None, ) .await .unwrap(); @@ -855,6 +1008,7 @@ async fn can_append_and_get( key1, 2, value2.clone(), + None, ) .await .unwrap(); @@ -866,6 +1020,7 @@ async fn can_append_and_get( key1, 3, value3.clone(), + None, ) .await .unwrap(); @@ -892,11 +1047,11 @@ async fn append_cannot_overwrite( let value1 = "value1".as_bytes().to_vec(); let value2 = "value2".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); let result1 = is - .append("svc", "api", "entity", ns.ns.clone(), key1, 1, value2) + .append("svc", "api", "entity", ns.ns.clone(), key1, 1, value2, None) .await; assert!(result1.is_err()); @@ -924,6 +1079,7 @@ async fn append_can_skip( key1, 4, value1.clone(), + None, ) .await .unwrap(); @@ -935,6 +1091,7 @@ async fn append_can_skip( key1, 8, value2.clone(), + None, ) .await .unwrap(); @@ -962,11 +1119,11 @@ async fn length( let value2 = "value2".as_bytes().to_vec(); let result1 = is.length("svc", "api", ns.ns.clone(), key1).await.unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 4, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 4, value1, None) .await .unwrap(); let result2 = is.length("svc", "api", ns.ns.clone(), key1).await.unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 8, value2) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 8, value2, None) .await .unwrap(); let result3 = is.length("svc", "api", ns.ns.clone(), key1).await.unwrap(); @@ -1018,10 +1175,10 @@ async fn scan_with_no_pattern_single_paged( let value1 = "value1".as_bytes().to_vec(); let value2 = "value2".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2) + is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2, None) .await .unwrap(); @@ -1069,6 +1226,7 @@ async fn scan_with_no_pattern_paginated( key1, 1, value1.clone(), + None, ) .await .unwrap(); @@ -1080,6 +1238,7 @@ async fn scan_with_no_pattern_paginated( key1, 2, value2.clone(), + None, ) .await .unwrap(); @@ -1091,6 +1250,7 @@ async fn scan_with_no_pattern_paginated( key2, 1, value2.clone(), + None, ) .await .unwrap(); @@ -1102,6 +1262,7 @@ async fn scan_with_no_pattern_paginated( key3, 3, value3.clone(), + None, ) .await .unwrap(); @@ -1203,12 +1364,30 @@ async fn scan_stable_resumes_past_deleted_keys( .collect(); for (swept, below, key) in &planted { - is.append("svc", "api", "entity", swept.clone(), key, 1, b"v".to_vec()) - .await - .unwrap(); - is.append("svc", "api", "entity", below.clone(), key, 1, b"v".to_vec()) - .await - .unwrap(); + is.append( + "svc", + "api", + "entity", + swept.clone(), + key, + 1, + b"v".to_vec(), + None, + ) + .await + .unwrap(); + is.append( + "svc", + "api", + "entity", + below.clone(), + key, + 1, + b"v".to_vec(), + None, + ) + .await + .unwrap(); } // Take a page, delete its keys here and in the layer below, and resume from the token. @@ -1312,6 +1491,7 @@ async fn multi_sqlite_scan_stable_crosses_its_files_a_page_at_a_time() { key, 1, b"v".to_vec(), + None, ) .await .unwrap(); @@ -1375,9 +1555,18 @@ async fn multi_sqlite_scan_stable_sees_files_created_after_a_walk() { agent_mode: AgentMode::Durable, }; let key = format!("key-{name}"); - is.append("svc", "api", "entity", namespace, &key, 1, b"v".to_vec()) - .await - .unwrap(); + is.append( + "svc", + "api", + "entity", + namespace, + &key, + 1, + b"v".to_vec(), + None, + ) + .await + .unwrap(); key } @@ -1430,6 +1619,7 @@ async fn last_id_matches_last_without_the_value( &key, id, format!("value-{id}").into_bytes(), + None, ) .await .unwrap(); @@ -1466,13 +1656,13 @@ async fn scan_with_prefix_pattern_single_paged( let value2 = "value2".as_bytes().to_vec(); let value3 = "value3".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2) + is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key3, 1, value3) + is.append("svc", "api", "entity", ns.ns.clone(), key3, 1, value3, None) .await .unwrap(); @@ -1520,12 +1710,21 @@ async fn sql_scan_prefix_is_literal_bounded_and_deletion_safe( ]; for key in keys { storage - .append("svc", "api", "entity", ns.ns.clone(), key, 1, vec![1]) + .append("svc", "api", "entity", ns.ns.clone(), key, 1, vec![1], None) .await .unwrap(); } storage - .append("svc", "api", "entity", ns.ns.clone(), "aa", 2, vec![2]) + .append( + "svc", + "api", + "entity", + ns.ns.clone(), + "aa", + 2, + vec![2], + None, + ) .await .unwrap(); @@ -1637,13 +1836,13 @@ async fn scan_with_prefix_pattern_paginated( let value2 = "value2".as_bytes().to_vec(); let value3 = "value3".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2) + is.append("svc", "api", "entity", ns.ns.clone(), key2, 1, value2, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key3, 1, value3) + is.append("svc", "api", "entity", ns.ns.clone(), key3, 1, value3, None) .await .unwrap(); @@ -1702,7 +1901,7 @@ async fn exists_append_delete( let value1 = "value1".as_bytes().to_vec(); let result1 = is.exists("svc", "api", ns.ns.clone(), key1).await.unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 1, value1, None) .await .unwrap(); is.delete("svc", "api", ns.ns.clone(), key1).await.unwrap(); @@ -1725,9 +1924,18 @@ async fn delete_is_per_namespace( let key1 = "key1"; let value1 = "value1".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns1.ns.clone(), key1, 1, value1) - .await - .unwrap(); + is.append( + "svc", + "api", + "entity", + ns1.ns.clone(), + key1, + 1, + value1, + None, + ) + .await + .unwrap(); is.delete("svc", "api", ns2.ns.clone(), key1).await.unwrap(); let result = is.exists("svc", "api", ns1.ns.clone(), key1).await.unwrap(); @@ -1777,6 +1985,7 @@ async fn first( key1, 5, value1.clone(), + None, ) .await .unwrap(); @@ -1788,6 +1997,7 @@ async fn first( key1, 7, value2.clone(), + None, ) .await .unwrap(); @@ -1826,6 +2036,7 @@ async fn last( key1, 5, value1.clone(), + None, ) .await .unwrap(); @@ -1837,6 +2048,7 @@ async fn last( key1, 7, value2.clone(), + None, ) .await .unwrap(); @@ -1875,6 +2087,7 @@ async fn closest_low( key1, 5, value1.clone(), + None, ) .await .unwrap(); @@ -1886,6 +2099,7 @@ async fn closest_low( key1, 7, value2.clone(), + None, ) .await .unwrap(); @@ -1924,6 +2138,7 @@ async fn closest_match( key1, 5, value1.clone(), + None, ) .await .unwrap(); @@ -1935,6 +2150,7 @@ async fn closest_match( key1, 7, value2.clone(), + None, ) .await .unwrap(); @@ -1973,6 +2189,7 @@ async fn closest_mid( key1, 5, value1.clone(), + None, ) .await .unwrap(); @@ -1984,6 +2201,7 @@ async fn closest_mid( key1, 7, value2.clone(), + None, ) .await .unwrap(); @@ -2014,10 +2232,10 @@ async fn closest_high( .closest("svc", "api", "entity", ns.ns.clone(), key1, 10) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 5, value1) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 5, value1, None) .await .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 7, value2) + is.append("svc", "api", "entity", ns.ns.clone(), key1, 7, value2, None) .await .unwrap(); let result2 = is @@ -2052,6 +2270,7 @@ async fn drop_prefix_no_match( key1, 10, value1.clone(), + None, ) .await .unwrap(); @@ -2063,6 +2282,7 @@ async fn drop_prefix_no_match( key1, 11, value2.clone(), + None, ) .await .unwrap(); @@ -2074,6 +2294,7 @@ async fn drop_prefix_no_match( key1, 12, value3.clone(), + None, ) .await .unwrap(); @@ -2112,6 +2333,7 @@ async fn drop_prefix_partial( key1, 10, value1.clone(), + None, ) .await .unwrap(); @@ -2123,6 +2345,7 @@ async fn drop_prefix_partial( key1, 11, value2.clone(), + None, ) .await .unwrap(); @@ -2134,6 +2357,7 @@ async fn drop_prefix_partial( key1, 12, value3.clone(), + None, ) .await .unwrap(); @@ -2164,15 +2388,42 @@ async fn drop_prefix_full( let value2 = "value2".as_bytes().to_vec(); let value3 = "value3".as_bytes().to_vec(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 10, value1) - .await - .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 11, value2) - .await - .unwrap(); - is.append("svc", "api", "entity", ns.ns.clone(), key1, 12, value3) - .await - .unwrap(); + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key1, + 10, + value1, + None, + ) + .await + .unwrap(); + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key1, + 11, + value2, + None, + ) + .await + .unwrap(); + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key1, + 12, + value3, + None, + ) + .await + .unwrap(); is.drop_prefix("svc", "api", ns.ns.clone(), key1, 20) .await @@ -2184,3 +2435,834 @@ async fn drop_prefix_full( assert_eq!(result, vec![]); } + +// --------------------------------------------------------------------------------------------- +// The shard-epoch fence. +// +// Every test below runs against all five backends. The ones that cannot fence (redis, in-memory) +// must behave exactly as they did before the epoch argument existed - accept the write and ignore +// the epoch - so each test asserts both halves rather than being skipped for them. +// --------------------------------------------------------------------------------------------- + +fn assert_fenced( + result: Result<(), IndexedStorageError>, + expected_epoch: u64, + actual_epoch: Option, +) { + match result { + Err(IndexedStorageError::Fenced { + expected, actual, .. + }) => { + assert_eq!(expected, ShardEpoch(expected_epoch), "expected epoch"); + assert_eq!(actual, actual_epoch.map(ShardEpoch), "stored epoch"); + } + other => panic!("expected a Fenced error, got {other:?}"), + } +} + +#[test] +#[tracing::instrument] +async fn append_with_the_recorded_epoch_is_accepted( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-match"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(7)) + .await + .unwrap(); + is.append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([ + (1, Bytes::from_static(b"a")), + (2, Bytes::from_static(b"b")), + (3, Bytes::from_static(b"c")), + ]), + Some(ShardEpoch(7)), + ) + .await + .unwrap(); + + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 3 + ); +} + +#[test] +#[tracing::instrument] +async fn a_stale_epoch_append_is_refused_and_writes_nothing( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-stale"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(8)) + .await + .unwrap(); + let result = is + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([ + (1, Bytes::from_static(b"a")), + (2, Bytes::from_static(b"b")), + (3, Bytes::from_static(b"c")), + ]), + Some(ShardEpoch(7)), + ) + .await; + + let length = is.length("svc", "api", ns.ns.clone(), key).await.unwrap(); + assert_fenced(result, 7, Some(8)); + // The whole batch is rolled back, not the tail of it. + assert_eq!(length, 0, "a refused batch must leave no entry behind"); +} + +#[test] +#[tracing::instrument] +async fn another_writer_at_the_same_epoch_is_refused( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // A shard manager that lost its state mints from zero again and can hand a live owner's epoch + // to somebody else. The epoch alone cannot separate them, so the row's writer does: the owner + // holds it, and the newcomer is refused at the open rather than sharing the generation. + let (owner, newcomer) = is.get_two_writers().await; + let key = "fence-two-writers"; + + owner + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(4)) + .await + .unwrap(); + + let claim = newcomer + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(4)) + .await; + + match claim { + Err(IndexedStorageError::Fenced { + expected, + actual, + writer_conflict, + .. + }) => { + assert_eq!(expected, ShardEpoch(4), "expected epoch"); + assert_eq!(actual, Some(ShardEpoch(4)), "stored epoch"); + assert!( + writer_conflict, + "the epochs match, so the refusal has to name the writer as the reason - that is \ + what tells the shard manager to mint past this epoch rather than leave it shared" + ); + } + other => panic!("expected a Fenced error, got {other:?}"), + } + + // And the newcomer cannot write behind the owner's back either. + let append = newcomer + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(1, Bytes::from_static(b"a"))]), + Some(ShardEpoch(4)), + ) + .await; + match append { + Err(IndexedStorageError::Fenced { + writer_conflict, .. + }) => assert!(writer_conflict), + other => panic!("expected a Fenced error, got {other:?}"), + } + assert_eq!( + owner + .length("svc", "api", ns.ns.clone(), key) + .await + .unwrap(), + 0, + "a refused append writes nothing" + ); + + // The owner is untouched by the attempt. + owner + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(1, Bytes::from_static(b"a"))]), + Some(ShardEpoch(4)), + ) + .await + .unwrap(); +} + +#[test] +#[tracing::instrument] +async fn the_same_writer_re_opens_at_the_same_epoch( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // The ordinary case the writer column must not break: one process re-opening an oplog it + // already holds, at the epoch it already holds, which happens on every cache eviction. + let is = is.get_indexed_storage().await; + let key = "fence-reopen"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(4)) + .await + .unwrap(); + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(4)) + .await + .unwrap(); + is.append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(1, Bytes::from_static(b"a"))]), + Some(ShardEpoch(4)), + ) + .await + .unwrap(); +} + +#[test] +#[tracing::instrument] +async fn a_newcomer_minted_above_the_collision_takes_the_oplog_over( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // The repair the refusal above sets off: the newcomer reports the collision, the shard manager + // mints past it, and the higher epoch takes the oplog over - at which point the old owner is + // the one being refused. + let (owner, newcomer) = is.get_two_writers().await; + let key = "fence-re-mint"; + + owner + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(4)) + .await + .unwrap(); + newcomer + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + newcomer + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(1, Bytes::from_static(b"a"))]), + Some(ShardEpoch(5)), + ) + .await + .unwrap(); + + let refused = owner + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(2, Bytes::from_static(b"b"))]), + Some(ShardEpoch(4)), + ) + .await; + match refused { + Err(IndexedStorageError::Fenced { + actual, + writer_conflict, + .. + }) => { + assert_eq!(actual, Some(ShardEpoch(5))); + assert!( + !writer_conflict, + "this one is an ordinary takeover, not two writers on one epoch" + ); + } + other => panic!("expected a Fenced error, got {other:?}"), + } +} + +#[test] +#[tracing::instrument] +async fn an_append_ahead_of_the_recorded_epoch_is_refused( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-ahead"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + + // The check is equality, not "at least": a record behind the asserted epoch means the open + // that should have raised it never ran, so the write is not ours to make. Both call shapes, so + // a single-entry path that parts from the batch path cannot quietly relax the check. + let batch = is + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([ + (1, Bytes::from_static(b"a")), + (2, Bytes::from_static(b"b")), + (3, Bytes::from_static(b"c")), + ]), + Some(ShardEpoch(6)), + ) + .await; + let batch_length = is.length("svc", "api", ns.ns.clone(), key).await.unwrap(); + let single = is + .append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 4, + b"d".to_vec(), + Some(ShardEpoch(6)), + ) + .await; + let length = is.length("svc", "api", ns.ns.clone(), key).await.unwrap(); + + assert_fenced(batch, 6, Some(5)); + assert_eq!( + batch_length, 0, + "a refused batch must leave no entry behind" + ); + assert_fenced(single, 6, Some(5)); + assert_eq!(length, 0, "a refused append must leave no entry behind"); +} + +#[test] +#[tracing::instrument] +async fn an_append_without_a_recorded_epoch_is_refused( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-absent"; + + // No upsert. Epoch 0 is a perfectly valid epoch, so this also pins that an absent row is not + // silently treated as zero. + let result = is + .append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 1, + b"a".to_vec(), + Some(ShardEpoch(0)), + ) + .await; + + let length = is.length("svc", "api", ns.ns.clone(), key).await.unwrap(); + assert_fenced(result, 0, None); + assert_eq!(length, 0); +} + +#[test] +#[tracing::instrument] +async fn an_unfenced_append_ignores_the_recorded_epoch( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-none"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(9)) + .await + .unwrap(); + // `None` asserts nothing: it is what the archive layers and generic callers pass. + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 1, + b"a".to_vec(), + None, + ) + .await + .unwrap(); + + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 1 + ); +} + +#[test] +#[tracing::instrument] +async fn the_recorded_epoch_only_ever_climbs( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-monotonic"; + + // Rising and repeated epochs are accepted ... + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(9)) + .await + .unwrap(); + + // ... a falling one is not, or a zombie could re-open at its stale epoch and un-fence itself + // against the current owner. + let result = is + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(8)) + .await; + + assert_fenced(result, 8, Some(9)); + // and the rejected upsert left the record alone + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 1, + b"a".to_vec(), + Some(ShardEpoch(9)), + ) + .await + .unwrap(); + assert_fenced( + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 2, + b"b".to_vec(), + Some(ShardEpoch(8)), + ) + .await, + 8, + Some(9), + ); +} + +#[test] +#[tracing::instrument] +async fn deleting_the_recorded_epoch_fences_later_writes( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-deleted"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(3)) + .await + .unwrap(); + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, None) + .await + .unwrap(); + // Idempotent: deleting again is not an error. + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, None) + .await + .unwrap(); + + let result = is + .append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 1, + b"a".to_vec(), + Some(ShardEpoch(3)), + ) + .await; + + assert_fenced(result, 3, None); +} + +#[test] +#[tracing::instrument] +async fn a_deleted_record_does_not_remember_its_epoch( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-forgotten"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(9)) + .await + .unwrap(); + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, None) + .await + .unwrap(); + + // The documented limit of the fence: the forward-only rule lives on the record, so once the + // record is gone a lower epoch than the one it held is recorded and written through. Closing + // this needs the delete to keep the epoch, which changes this test on purpose. + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(8)) + .await + .unwrap(); + is.append( + "svc", + "api", + "entity", + ns.ns.clone(), + key, + 1, + b"a".to_vec(), + Some(ShardEpoch(8)), + ) + .await + .unwrap(); + + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 1 + ); +} + +async fn append_fenced( + is: &Arc, + ns: &IndexedStorageNamespace, + key: &str, + ids: &[u64], + epoch: Option, +) -> Result<(), IndexedStorageError> { + let pairs: Vec<(u64, Bytes)> = ids + .iter() + .map(|id| (*id, Bytes::from(id.to_string()))) + .collect(); + is.append_many("svc", "api", "entity", ns, key, Arc::from(pairs), epoch) + .await +} + +#[test] +#[tracing::instrument] +async fn the_recorded_writer_deletes_the_key_and_its_record( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-delete-owner"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + append_fenced(&is, &ns.ns, key, &[1, 2], Some(ShardEpoch(5))) + .await + .unwrap(); + + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, Some(ShardEpoch(5))) + .await + .unwrap(); + + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 0 + ); + // The record went with the entries, so the old epoch writes nothing back. + assert_fenced( + append_fenced(&is, &ns.ns, key, &[3], Some(ShardEpoch(5))).await, + 5, + None, + ); +} + +#[test] +#[tracing::instrument] +async fn a_delete_by_a_writer_that_lost_the_key_deletes_nothing( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // A deletion that outlived its writer's hold on the key: the new holder took it at a higher + // epoch and wrote to it. The stale delete must leave both the record and the entries alone. + let (stale, owner) = is.get_two_writers().await; + let key = "fence-delete-stale"; + + stale + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + append_fenced(&stale, &ns.ns, key, &[1], Some(ShardEpoch(5))) + .await + .unwrap(); + owner + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(6)) + .await + .unwrap(); + append_fenced(&owner, &ns.ns, key, &[2], Some(ShardEpoch(6))) + .await + .unwrap(); + + assert_fenced( + stale + .delete_with_epoch("svc", "api", ns.ns.clone(), key, Some(ShardEpoch(5))) + .await, + 5, + Some(6), + ); + + assert_eq!( + owner + .length("svc", "api", ns.ns.clone(), key) + .await + .unwrap(), + 2, + "a refused delete removes no entry" + ); + append_fenced(&owner, &ns.ns, key, &[3], Some(ShardEpoch(6))) + .await + .expect("a refused delete leaves the new holder's record in place"); +} + +#[test] +#[tracing::instrument] +async fn a_delete_asserting_an_epoch_on_a_key_without_a_record_deletes_nothing( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let is = is.get_indexed_storage().await; + let key = "fence-delete-unrecorded"; + + append_fenced(&is, &ns.ns, key, &[1], None).await.unwrap(); + + assert_fenced( + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, Some(ShardEpoch(1))) + .await, + 1, + None, + ); + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 1 + ); +} + +#[test] +#[tracing::instrument] +async fn a_delete_asserting_an_epoch_on_a_key_that_is_already_gone_is_accepted( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // Neither a record nor entries: nothing is left for a writer that lost the key to destroy, so + // a deletion retried after its own earlier attempt removed the key can finish. + let is = is.get_indexed_storage().await; + let key = "fence-delete-gone"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(3)) + .await + .unwrap(); + append_fenced(&is, &ns.ns, key, &[1, 2], Some(ShardEpoch(3))) + .await + .unwrap(); + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, Some(ShardEpoch(3))) + .await + .unwrap(); + is.delete_with_epoch("svc", "api", ns.ns.clone(), key, Some(ShardEpoch(3))) + .await + .expect("a repeated delete of a key that is already gone is accepted"); + is.delete_with_epoch( + "svc", + "api", + ns.ns.clone(), + "fence-delete-never-written", + Some(ShardEpoch(3)), + ) + .await + .expect("a delete of a key that was never written is accepted"); + + assert!(!is.exists("svc", "api", ns.ns.clone(), key).await.unwrap()); + // Accepting the delete recorded nothing: the old epoch still writes nothing back. + assert_fenced( + append_fenced(&is, &ns.ns, key, &[3], Some(ShardEpoch(3))).await, + 3, + None, + ); +} + +#[test] +#[tracing::instrument] +async fn an_unfenced_delete_removes_the_key_and_its_record( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + let (first, second) = is.get_two_writers().await; + let key = "fence-delete-unfenced"; + + first + .set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(7)) + .await + .unwrap(); + append_fenced(&first, &ns.ns, key, &[1, 2], Some(ShardEpoch(7))) + .await + .unwrap(); + + // Asserting nothing, so it does not matter that another writer holds the record. + second + .delete_with_epoch("svc", "api", ns.ns.clone(), key, None) + .await + .unwrap(); + + assert_eq!( + first + .length("svc", "api", ns.ns.clone(), key) + .await + .unwrap(), + 0 + ); + assert_fenced( + append_fenced(&first, &ns.ns, key, &[3], Some(ShardEpoch(7))).await, + 7, + None, + ); +} + +#[test] +#[tracing::instrument] +async fn an_empty_batch_is_accepted_whatever_epoch_it_asserts( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // Nothing to write is nothing to fence: every backend agrees, stale epoch or not. + let is = is.get_indexed_storage().await; + let key = "fence-empty-batch"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(5)) + .await + .unwrap(); + + append_fenced(&is, &ns.ns, key, &[], Some(ShardEpoch(4))) + .await + .expect("an empty batch writes nothing, so it is not refused"); + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 0 + ); +} + +#[test] +#[tracing::instrument] +async fn a_repeated_id_in_a_staged_batch_is_a_conflict( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, +) { + // A stage is an oplog insert like the visible oplog, in a batch as in a single append: a batch + // re-sent after an indeterminate write collides on its first id. + let is = is.get_indexed_storage().await; + let staged = IndexedStorageNamespace::StagedOpLog { + agent_id: AgentId { + component_id: ComponentId::new(), + agent_id: "staged-conflict".into(), + }, + agent_mode: AgentMode::Durable, + }; + let key = "staged-conflict"; + + append_fenced(&is, &staged, key, &[1, 2], None) + .await + .unwrap(); + + // Only the classification is asserted: an unfenced Redis batch is a pipeline, and its callers + // reconcile a partial one by reading it back. + match append_fenced(&is, &staged, key, &[2, 3], None).await { + Err(IndexedStorageError::Conflict(_)) => {} + other => panic!("expected a Conflict, got {other:?}"), + } +} + +#[test] +#[tracing::instrument] +async fn a_failed_batch_leaves_no_partial_write( + deps: &WorkerExecutorTestDependencies, + #[dimension(is)] is: &Arc, + #[tagged_as("ns1")] ns: &IndexedStorageNamespaces, +) { + // What makes "the fence is checked once per batch" true rather than "once per entry": a + // backend that loops single appends would leave the entries before the failure behind. + let is = is.get_indexed_storage().await; + let key = "batch-atomicity"; + + is.set_key_epoch("svc", "api", ns.ns.clone(), key, ShardEpoch(1)) + .await + .unwrap(); + is.append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([(1, Bytes::from_static(b"a")), (2, Bytes::from_static(b"b"))]), + Some(ShardEpoch(1)), + ) + .await + .unwrap(); + + // id 1 already exists, so the second entry of this batch violates the primary key. + let result = is + .append_many( + "svc", + "api", + "entity", + &ns.ns, + key, + Arc::from([ + (3, Bytes::from_static(b"c")), + (1, Bytes::from_static(b"dup")), + (4, Bytes::from_static(b"d")), + ]), + Some(ShardEpoch(1)), + ) + .await; + + assert!(result.is_err(), "a duplicate id must fail the batch"); + assert_eq!( + is.length("svc", "api", ns.ns.clone(), key).await.unwrap(), + 2, + "the failed batch must not have written its first entry" + ); +} diff --git a/golem-worker-executor/tests/instance_layer.rs b/golem-worker-executor/tests/instance_layer.rs index 4651e8135f..bd3e16ebe8 100644 --- a/golem-worker-executor/tests/instance_layer.rs +++ b/golem-worker-executor/tests/instance_layer.rs @@ -637,7 +637,11 @@ async fn completed_tool_positional_replay_errors( )?; live.await_result(&parent_id).await?; drop(primary); - active_agent.execution().commit(CommitLevel::Always).await; + active_agent + .execution() + .commit(CommitLevel::Always) + .await + .unwrap(); let before = active_agent.execution().oplog().current_oplog_index().await; active_agent @@ -881,7 +885,11 @@ async fn incomplete_tool_config_tail_reauthorizes_without_rejecting_recorded_rep )?; live.await_result(&parent_id).await?; drop(primary); - active_agent.execution().commit(CommitLevel::Always).await; + active_agent + .execution() + .commit(CommitLevel::Always) + .await + .unwrap(); let live_tip = active_agent.execution().oplog().current_oplog_index().await; let live_entries = active_agent @@ -1119,7 +1127,11 @@ async fn run_incomplete_tool_reveal_tail_reauthorization( )?; live.await_result(&parent_id).await?; drop(primary); - active_agent.execution().commit(CommitLevel::Always).await; + active_agent + .execution() + .commit(CommitLevel::Always) + .await + .unwrap(); let live_tip = active_agent.execution().oplog().current_oplog_index().await; let entries = active_agent .execution() @@ -2492,7 +2504,11 @@ async fn filesystem_capable_entity_stream_replays_on_owner_filesystem( )?; let live_result = live.await_result(&parent_id).await?; drop(primary); - active_agent.execution().commit(CommitLevel::Always).await; + active_agent + .execution() + .commit(CommitLevel::Always) + .await + .unwrap(); active_agent .execution() diff --git a/golem-worker-executor/tests/wasi.rs b/golem-worker-executor/tests/wasi.rs index 3499f18f40..6e2c88d306 100644 --- a/golem-worker-executor/tests/wasi.rs +++ b/golem-worker-executor/tests/wasi.rs @@ -2082,6 +2082,7 @@ async fn filesystem_full_replay_survives_lifecycle_transitions_impl( .revoke_shards(RevokeShardsRequest { shard_ids: vec![shard], revision: 1, + incarnation_id: String::new(), }) .await? .into_inner(); @@ -2107,6 +2108,7 @@ async fn filesystem_full_replay_survives_lifecycle_transitions_impl( // round trip does not depend on timing. revision: 1, number_of_shards: 1, + incarnation_id: String::new(), }) .await? .into_inner(); diff --git a/golem-worker-executor/tests/worker_initialization.rs b/golem-worker-executor/tests/worker_initialization.rs index 5e6330a8bd..2ab67f03b8 100644 --- a/golem-worker-executor/tests/worker_initialization.rs +++ b/golem-worker-executor/tests/worker_initialization.rs @@ -188,7 +188,8 @@ async fn register_stream(worker: &Worker) -> anyhow::ResultReplay is deterministic by construction
2

The resident runtime is disposable

-

The Wasmtime Store, the worker task, sockets, channels, caches and the executor process itself may vanish at any instruction boundary. Suspend, evict, reshard, restart and crash are all recovered the same way: throw the instance away, build a new Store, replay the oplog, continue.

+

The Wasmtime Store, the worker task, sockets, channels, caches and the executor process itself may vanish at any instruction boundary. Suspend, evict, restart and crash are all recovered the same way, by the same executor: throw the instance away, build a new Store, replay the oplog, continue. Losing the shard is different: every oplog write asserts this executor's shard epoch inside the storage transaction, and once another executor holds it, the next write is refused rather than accepted. That refusal gives up the agent (InterruptKind::ShardLost) — stopped here with nothing more written, dropped from this executor, never restarted in place — and it is the shard's new owner, not this executor, that builds the Store and replays (§18).

3
@@ -455,7 +455,7 @@

2Architecture & component m
Golem services around the worker executor - Clients call the worker service over HTTP or gRPC. The worker service asks the shard manager which executor owns an agent's shard and forwards the invocation to that executor over gRPC. Inside the executor, a Worker object per agent owns an invocation queue, a Wasmtime Store and a durable host context; the durable host context appends to and replays from the oplog service, which is backed by indexed storage (Redis or SQLite) with archive layers in blob storage. Executors also talk to the registry service for components and metadata, and to each other for RPC. + Clients call the worker service over HTTP or gRPC. The worker service asks the shard manager which executor owns an agent's shard and forwards the invocation to that executor over gRPC. Inside the executor, a Worker object per agent owns an invocation queue, a Wasmtime Store and a durable host context; the durable host context appends to and replays from the oplog service, which is backed by indexed storage (Postgres or SQLite alongside a real shard manager, since only those can fence a write on the shard epoch; Redis cannot and is refused at startup in that topology) with archive layers in blob storage. Executors also talk to the registry service for components and metadata, and to each other for RPC. ClientsCLI, HTTP API, SDKs Worker servicerouting, API gateway @@ -477,7 +477,7 @@

2Architecture & component m Durable host context (DurableWorkerCtx)durability guard · durable call sessions · replay cursorhost APIs: http, key-value, rpc, tools, streams… Servicesscheduler, promises, rpc,events, plugins - Oplog service (multi-layer)primary: indexed storage (Redis or SQLite)archive layers: compressed, then blob storage + Oplog service (multi-layer)primary: indexed storage (Postgres or SQLite; fenced on shard epoch)archive layers: compressed, then blob storage Other executorsdurable RPCstream transportworker proxy @@ -574,7 +574,7 @@

Cross-agent plumbing services/{rpc,promise,scheduler,worke

Storage backends src/storage/{indexed,keyvalue,scheduler}/*

-

The oplog service, key-value service, scheduler and status flusher are written against three small storage traits. Implementations: in-memory (tests), SQLite and multi-SQLite (single binary, local runs), PostgreSQL and Redis (clusters). Decorators exist for retrying, namespace routing and fault injection (used by the tests that simulate storage failure). Indexed storage also offers scan_stable, a key walk that does not skip keys when the caller deletes the ones it was handed, which is what the oplog sweep pages with. Blob storage for oplog archive layers and snapshots is provided by golem-service-base.

+

The oplog service, key-value service, scheduler and status flusher are written against three small storage traits. Implementations: in-memory (tests), SQLite and multi-SQLite (single binary, local runs), PostgreSQL and Redis. A real shard manager moves shards between executors, so its oplog writes must be fenceable on the shard epoch (IndexedStorage::supports_epoch_fencing) or the executor refuses to start; only PostgreSQL and the SQLite-backed indexed storages qualify — Redis remains available for key-value storage and for indexed storage without a real shard manager (single-shard, debugging service), but not for a clustered executor's oplog. Decorators exist for retrying, namespace routing and fault injection (used by the tests that simulate storage failure). Indexed storage also offers scan_stable, a key walk that does not skip keys when the caller deletes the ones it was handed, which is what the oplog sweep pages with. Blob storage for oplog archive layers and snapshots is provided by golem-service-base.

@@ -760,6 +760,8 @@

Append versus commit

CommitLevel::Always waits for durable storage. Deferred still waits for durable oplogs, but on an ephemeral oplog returns after ordered handoff to its bounded asynchronous writer; queue backpressure may still wait. DurableOnly waits only for durable agents and remains a no-op for ephemeral agents. Explicit protocol barriers and acceptance keep their existing storage guarantees: completion's use of Deferred is not a global weakening.

+

A commit can be refused instead of written. Every append and commit asserts this executor's shard epoch inside the storage transaction, and a storage that has recorded a newer epoch — another executor now owns the shard — returns OplogError::Fenced (DurableStreamProducerError::Fenced for durable-stream session records) instead of writing anything. commit_oplog_and_update_state, commit_oplog_before_status_update and add_and_commit_oplog return that refusal rather than swallowing it, so the three guarantees above hold either way: a refused PendingAgentInvocation commit is not acknowledged as accepted, a refused AgentInvocationFinished commit is not published to waiters, and a refused side-effect Start is not treated as durable. The first refusal latches — every later add on that oplog is refused too, without a second storage round trip — and the agent is given up (§18) instead of retrying the write or restarting in place.

+

worker/state_actor.rs::commit_and_update_state samples the appended oplog tip before its explicit commit and discards receipt entries already folded into the published status. Threshold auto-flushes in both primary and ephemeral oplogs, and replica waits, can commit outside the status actor; therefore an empty commit receipt does not prove that no new entries committed. Ephemeral threshold flushes hand batches to a bounded async writer, while read snapshots include handed-off entries plus the buffered tail after the persisted writer watermark. The status path retains at most 32 batch receipts; if that cap creates a gap, exceptional catch-up forces storage after the completion acknowledgment, then reads the missing range. Otherwise a non-contiguous receipt makes the actor reuse status::try_fold_status_from to read committed storage in bounded chunks, hydrate external StreamSession payloads and publish once. Completion itself does not await that fold. A freshness-sensitive status read queued on the actor waits behind it; the persistent status cache remains asynchronous. Admission and authority jobs still await their full folds, and the RunningWorkers recovery index remains synchronously updated by the actor. The unchanged synchronous Create write preserves an ephemeral agent's identity before execution; after executor loss, that identity reconstructs an observation-only owner, never a fresh execution.

Rediscovery

@@ -1975,7 +1977,8 @@

18Interrupt, suspend, restart, evic SuspendInterruptKind::Suspend(ts), used when the guest sleeps, waits for a promise, or has a pending durable RPC long enough to unload. RPC waits check after 30s and retry every 10s while other live work defers voluntary yielding.Suspend (h)On demand: a new invocation, a promise completion, or a durable ScheduledAction::Resume (RPC default: 5s later; mixed waits choose the earliest wakeup) JumpInterruptKind::Jump ("jumping back in time"): atomic-region rollback or set-oplog-index. Entity-local rollback instead appends Jumps at the live tail during reconstruction, preserving interleaved sibling history.Jump { region } (hint)Immediately, with the region skipped; entity-local continuation need not restart the owner Eviction (memory / filesystem pressure)EvictionClass ordering: LoadedIdle first (cheapest), then WarmRunnable (has durable pending invocations). Executing workers and workers with non-durable in-memory work (internal queue, ResumeReplay, interrupt) are never evicted.noneOn the next invocation or scheduled action - Reshardingon_shard_assignment_changed: workers whose shard moved away are unloaded here and reconstructed by the new ownernoneWhen the new owner is asked for the worker + ReshardingThe shard manager's RevokeShards/AssignShards gRPC calls give up (GiveUpReason::ShardRevoked / ShardNotAssigned) every agent whose shard moved away, via InterruptKind::ShardLostnoneNever on this executor. The agent is stopped and dropped here; the new owner is the one that reconstructs, when the worker service sends it a request + Oplog epoch fenceA write is refused because the shard epoch this executor asserted no longer matches storage (OplogError::Fenced / OplogFence); same give-up path as resharding, via GiveUpReason::Fencednone — the fence latches, so nothing later is written eitherNever on this executor, for the same reason. Any invocation still pending here is failed with a retriable error (no cached result), so worker-service retries it against the new owner instead Process death—whatever was committedWhen any executor is asked for the worker diff --git a/golem-worker-service/src/api/invocation_session.rs b/golem-worker-service/src/api/invocation_session.rs index 23db0b06c6..66c148e950 100644 --- a/golem-worker-service/src/api/invocation_session.rs +++ b/golem-worker-service/src/api/invocation_session.rs @@ -28,7 +28,7 @@ use futures::{SinkExt, StreamExt}; use golem_api_grpc::invocation_session_protocol::InvocationSessionState; use golem_api_grpc::proto::golem::schema::SchemaValue as ProtoSchemaValue; use golem_api_grpc::proto::golem::worker::{ - DurableStreamMapping, InputStreamEnd, InputStreamItem, InvocationAccepted, + DurableStreamMapping, InputStreamEnd, InputStreamItem, InvocationAccepted, InvocationRejected, InvocationRejectionReason, InvocationRequest, InvocationResponse, InvocationSessionResult, OutputStreamEnd, OutputStreamError, OutputStreamItem, ResumeOperation, StreamCancel, StreamCancelReason, StreamCancelRole, StreamCursor, StreamMappingRole, input_stream_item, @@ -58,6 +58,7 @@ use golem_common::schema::{ BinaryValuePayload, SchemaGraph, SchemaType, SchemaValue, schema_value_to_proto_with_streams, }; use golem_service_base::clients::registry::RegistryServiceError; +use golem_service_base::error::worker_executor::WorkerExecutorError; use golem_service_base::model::auth::AuthCtx; use poem::web::websocket::{CloseCode, Message, WebSocketStream}; use std::collections::{HashMap, HashSet, VecDeque}; @@ -1857,13 +1858,16 @@ async fn translate_private_response( Ok(vec![frame(text_message(&message)?)]) } invocation_response::Response::Rejected(rejected) => { - let code = rejection_code(rejected.reason); + let code = rejection_code(&rejected); Ok(vec![frame(text_message( &PublicServerMessage::InvocationRejected { attempt_id: Some(attempt_id), code, message: safe_rejection_message(code), - retryable: matches!(code, PublicErrorCode::ResourceExhausted), + retryable: matches!( + code, + PublicErrorCode::ResourceExhausted | PublicErrorCode::RoutingMiss + ), version: 1, }, )?)]) @@ -2619,8 +2623,8 @@ fn public_start_error(error: PublicAgentSessionStartError) -> (PublicErrorCode, } } -fn rejection_code(reason: i32) -> PublicErrorCode { - match InvocationRejectionReason::try_from(reason) { +fn rejection_code(rejected: &InvocationRejected) -> PublicErrorCode { + match InvocationRejectionReason::try_from(rejected.reason) { Ok(InvocationRejectionReason::Validation) => PublicErrorCode::ValidationError, Ok(InvocationRejectionReason::Unauthorized) => PublicErrorCode::Unauthorized, Ok(InvocationRejectionReason::NotFound) => PublicErrorCode::NotFound, @@ -2635,10 +2639,29 @@ fn rejection_code(reason: i32) -> PublicErrorCode { Ok(InvocationRejectionReason::InputConflict) => PublicErrorCode::InputConflict, Ok(InvocationRejectionReason::InputGap) => PublicErrorCode::InputGap, Ok(InvocationRejectionReason::ResourceExhausted) => PublicErrorCode::ResourceExhausted, + // The unary and agent-RPC paths reroute on a routing miss; a session client is told the + // same thing so it can retry instead of surfacing a server fault. + Ok(InvocationRejectionReason::Internal) if is_routing_miss(rejected) => { + PublicErrorCode::RoutingMiss + } _ => PublicErrorCode::InternalError, } } +fn is_routing_miss(rejected: &InvocationRejected) -> bool { + rejected + .worker_error + .clone() + .and_then(|error| WorkerExecutorError::try_from(error).ok()) + .is_some_and(|error| { + // A fenced oplog crosses the wire as `ShardingNotReady`. + matches!( + error, + WorkerExecutorError::InvalidShardId { .. } | WorkerExecutorError::ShardingNotReady + ) + }) +} + fn safe_rejection_message(code: PublicErrorCode) -> String { match code { PublicErrorCode::UnsupportedSubprotocol => "required WebSocket subprotocol is unsupported", @@ -2666,6 +2689,9 @@ fn safe_rejection_message(code: PublicErrorCode) -> String { PublicErrorCode::ProducerError => "stream producer failed", PublicErrorCode::InvocationFailed => "invocation failed", PublicErrorCode::ProtocolError => "invocation protocol failed", + PublicErrorCode::RoutingMiss => { + "the agent's shard is moving between executors; retry the invocation" + } PublicErrorCode::InternalError => "invocation failed", } .to_string() @@ -3195,6 +3221,49 @@ mod tests { .push_back(admission); } + /// A routing miss is an `Internal` rejection whose carried error names a shard or a fence. + /// It gets its own public code, so a client retries it, and any other `Internal` stays one. + #[test] + fn a_rejection_carrying_a_routing_miss_maps_to_a_retryable_public_code() { + let rejected = |worker_error: Option| InvocationRejected { + reason: InvocationRejectionReason::Internal as i32, + error: String::new(), + idempotency_key: None, + agent_id: None, + component_revision: None, + worker_error: worker_error.map(Into::into), + }; + for miss in [ + WorkerExecutorError::InvalidShardId { + shard_id: golem_common::model::ShardId::new(0), + shard_ids: Vec::new(), + }, + WorkerExecutorError::ShardingNotReady, + WorkerExecutorError::OplogFenced { + agent_id: golem_common::model::AgentId { + component_id: golem_common::model::component::ComponentId::new(), + agent_id: "fenced".to_string(), + }, + expected_epoch: 1, + actual_epoch: Some(2), + }, + ] { + assert_eq!( + rejection_code(&rejected(Some(miss))), + PublicErrorCode::RoutingMiss + ); + } + assert_eq!( + rejection_code(&rejected(Some(WorkerExecutorError::unknown("boom")))), + PublicErrorCode::InternalError + ); + assert_eq!( + rejection_code(&rejected(None)), + PublicErrorCode::InternalError + ); + assert_eq!(PublicErrorCode::RoutingMiss.as_str(), "routing-miss"); + } + #[test] async fn input_retries_gaps_and_terminal_high_water_use_public_sequences() { let state = active_input_state(); diff --git a/golem-worker-service/src/service/worker/client.rs b/golem-worker-service/src/service/worker/client.rs index d2d650fcd3..28f43f7482 100644 --- a/golem-worker-service/src/service/worker/client.rs +++ b/golem-worker-service/src/service/worker/client.rs @@ -69,7 +69,6 @@ use golem_service_base::grpc::client::MultiTargetGrpcClient; use golem_service_base::model::auth::AuthCtx; use golem_service_base::model::{ComponentFileSystemNode, FileReadResponse, GetOplogResponse}; use golem_service_base::service::routing_table::{HasRoutingTableService, RoutingTableService}; -use std::future::Future; use std::pin::Pin; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::{collections::HashMap, sync::Arc}; @@ -180,26 +179,57 @@ fn validate_agent_enumeration_count(count: u64) -> Result<(), WorkerExecutorErro pub type InvocationRequestStream = Pin + Send + 'static>>; pub type InvocationResponseStream = Pin> + Send + 'static>>; -type InvocationSessionCall<'a> = Pin< - Box< - dyn Future>, Status>> - + Send - + 'a, - >, ->; - -fn invoke_agent_session_once<'a>( - client: &'a mut WorkerExecutorClient>, - request: Option, -) -> InvocationSessionCall<'a> { - match request { - Some(request) => Box::pin(client.invoke_agent_session(request)), - None => Box::pin(std::future::ready(Err(Status::aborted( - "invocation session request was already consumed", - )))), +/// One dispatch of an invocation session's opening frame, read up to the executor's decision. +#[derive(Debug)] +struct SessionDispatch { + decision: Option, + responses: tonic::Streaming, + /// Feeds the executor the rest of the caller's request stream. Nothing is sent on it before + /// the executor accepts, and a dispatch that is not accepted drops it unused. + tail: mpsc::Sender, +} + +impl SessionDispatch { + fn routing_miss(&self) -> Option { + match &self.decision { + Some(InvocationResponse { + response: Some(invocation_response::Response::Rejected(rejected)), + }) => routing_miss_error(rejected), + _ => None, + } + } + + fn accepted(&self) -> bool { + matches!( + self.decision, + Some(InvocationResponse { + response: Some(invocation_response::Response::Accepted(_)), + }) + ) } } +async fn dispatch_invocation_session( + client: &mut WorkerExecutorClient>, + first: InvocationRequest, +) -> Result { + let (tail, receiver) = mpsc::channel(1); + let mut responses = client + .invoke_agent_session( + futures::stream::once(std::future::ready(first)).chain(ReceiverStream::new(receiver)), + ) + .await? + .into_inner(); + // `tail` stays open while the decision is awaited: an executor rejects a request stream that + // closes before acceptance as a protocol violation. + let decision = responses.message().await?; + Ok(SessionDispatch { + decision, + responses, + tail, + }) +} + #[derive(Debug)] enum OneShotInvocationSessionResult { Success(Box), @@ -212,6 +242,24 @@ fn protocol_failure(details: impl Into) -> OneShotInvocationSessionResul OneShotInvocationSessionResult::ProtocolFailure(details.into()) } +/// The executor error a rejection carries when the request reached an executor that does not +/// own the agent's shard: a stale route, a lapsed lease, or an oplog with a new owner. Not a +/// refusal of the invocation - the caller retries it on the shard's owner after refreshing its +/// routing table - which is why it is read off the typed error rather than +/// [`decode_invocation_rejection`], whose result is a service error nothing retries. +fn routing_miss_error(rejected: &InvocationRejected) -> Option { + if rejected.reason != InvocationRejectionReason::Internal as i32 { + return None; + } + let error: WorkerExecutorError = rejected.worker_error.clone()?.try_into().ok()?; + // A fenced oplog crosses the wire as `ShardingNotReady`. + matches!( + error, + WorkerExecutorError::InvalidShardId { .. } | WorkerExecutorError::ShardingNotReady + ) + .then_some(error) +} + fn protocol_executor_error(details: impl Into) -> WorkerExecutorError { WorkerExecutorError::Unknown { details: details.into(), @@ -2143,7 +2191,18 @@ impl WorkerClient for WorkerExecutorWorkerClient { |outcome| match outcome { OneShotInvocationSessionResult::Success(output) => Ok(*output), OneShotInvocationSessionResult::Rejected(rejected) => { - Err(decode_invocation_rejection(rejected).into()) + // A routing miss is retried on the shard's owner, like the typed failure + // an executor sends for the same condition after accepting. + match routing_miss_error(&rejected) { + Some(error) => { + tracing::debug!( + %error, + "Executor turned the invocation away as a routing miss" + ); + Err(error.into()) + } + None => Err(decode_invocation_rejection(rejected).into()), + } } OneShotInvocationSessionResult::Failure(failure) => { Err(decode_invocation_failure(failure).into()) @@ -2164,41 +2223,81 @@ impl WorkerClient for WorkerExecutorWorkerClient { agent_id: &AgentId, request: InvocationRequestStream, ) -> WorkerResult { - let routing_table = self - .routing_table_service - .get_routing_table() - .await - .map_err(|error| { - WorkerServiceError::InternalCallError( - CallWorkerExecutorError::FailedToGetRoutingTable(error), - ) - })?; - let pod = routing_table.lookup(agent_id).ok_or_else(|| { - WorkerServiceError::InternalCallError(CallWorkerExecutorError::FailedToConnectToPod( - Status::unavailable(format!("no active shard for agent {agent_id}")), - )) + // An executor that does not own the agent's shard rejects the session before accepting + // it, and no input may precede acceptance. So the opening frame alone is dispatched, and + // is dispatched again after the routing table is refreshed, exactly like the unary path; + // the caller's input is attached only to the executor that accepted. A transport failure + // before the decision arrives also sends the opening frame again, with the same attempt + // id and downgraded to MayExist, as the unary path does. Two consequences: input a caller + // sends too early is held until acceptance, so it is the response validator rather than + // the executor that refuses it; and while no executor owns the shard the session waits, + // which keeps a WebSocket session's connection open for as long as that lasts. + let mut request = request; + let first = request.next().await.ok_or_else(|| { + WorkerServiceError::Internal( + "invocation session request ended before start".to_string(), + ) })?; - let request = Arc::new(std::sync::Mutex::new(Some(request))); - let response = self - .worker_executor_clients - .call_without_retry( + let first_dispatch = Arc::new(AtomicBool::new(true)); + + let dispatch = self + .call_worker_executor( + agent_id.clone(), "invoke_agent_session", - pod.uri(self.worker_executor_clients.uses_tls()), move |worker_executor_client| { - let request = request - .lock() - .unwrap_or_else(|poison| poison.into_inner()) - .take(); - invoke_agent_session_once(worker_executor_client, request) + let mut first = first.clone(); + if let Some(invocation_request::Request::Start(start)) = &mut first.request + && start.freshness_disposition + == golem_api_grpc::proto::golem::worker::InvocationFreshnessDisposition::KnownFresh + as i32 + && freshness_disposition_for_dispatch( + InvocationFreshnessDisposition::KnownFresh, + &first_dispatch, + ) == InvocationFreshnessDisposition::MayExist + { + start.freshness_disposition = + golem_api_grpc::proto::golem::worker::InvocationFreshnessDisposition::MayExist + as i32; + } + Box::pin(dispatch_invocation_session(worker_executor_client, first)) + }, + |dispatch| match dispatch.routing_miss() { + Some(error) => { + tracing::debug!( + %error, + "Executor turned the invocation session away as a routing miss" + ); + Err(error.into()) + } + None => Ok(dispatch), }, + WorkerServiceError::InternalCallError, ) - .await - .map_err(|status| { - WorkerServiceError::InternalCallError( - CallWorkerExecutorError::FailedToConnectToPod(status), - ) - })?; - Ok(Box::pin(response.into_inner())) + .await?; + + if dispatch.accepted() { + let tail = dispatch.tail; + tokio::spawn(async move { + loop { + tokio::select! { + item = request.next() => match item { + Some(item) => { + if tail.send(item).await.is_err() { + break; + } + } + None => break, + }, + // The session ended on the executor's side while the caller still had + // input to send; stop holding the caller's stream. + _ = tail.closed() => break, + } + } + }); + } + Ok(Box::pin( + futures::stream::iter(dispatch.decision.map(Ok)).chain(dispatch.responses), + )) } async fn invoke_agent_session_one_shot( @@ -3103,10 +3202,10 @@ mod one_shot_session_tests { #[cfg(test)] mod rejection_mapping_tests { use super::{ - WorkerClient, WorkerExecutorWorkerClient, decode_invocation_rejection, + WorkerClient, WorkerExecutorWorkerClient, WorkerServiceError, decode_invocation_rejection, validate_agent_enumeration_count, }; - use futures::{Stream, stream}; + use futures::{Stream, StreamExt, stream}; use golem_api_grpc::proto::golem::schema::{SchemaValue, schema_value}; use golem_api_grpc::proto::golem::shardmanager::{ IpAddress, Pod as GrpcPod, RoutingTable as GrpcRoutingTable, RoutingTableEntry, ShardId, @@ -3114,8 +3213,9 @@ mod rejection_mapping_tests { }; use golem_api_grpc::proto::golem::worker::v1::{AgentError, agent_error}; use golem_api_grpc::proto::golem::worker::{ + InputStreamEnd, InvocationAccepted, InvocationFreshnessDisposition as WireFreshness, InvocationRejected, InvocationRejectionReason, InvocationRequest, InvocationResponse, - invocation_response, + InvocationStart, invocation_request, invocation_response, }; use golem_api_grpc::proto::golem::workerexecutor::v1::worker_executor_server::{ WorkerExecutor, WorkerExecutorServer, @@ -3137,11 +3237,15 @@ mod rejection_mapping_tests { use golem_service_base::model::auth::AuthCtx; use golem_service_base::model::quota_lease::{PendingReservation, QuotaLease}; use golem_service_base::service::routing_table::{RoutingTableConfig, RoutingTableService}; + use std::collections::BTreeMap; use std::net::Ipv4Addr; use std::pin::Pin; use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; use test_r::test; use tokio::net::TcpListener; + use tokio_stream::wrappers::ReceiverStream; use tokio_stream::wrappers::TcpListenerStream; use tonic::codec::CompressionEncoding; use tonic::{Request, Response, Status}; @@ -3249,6 +3353,10 @@ mod rejection_mapping_tests { _port: u16, _pod_name: Option, _executor_id: uuid::Uuid, + _previous_shard_epochs: std::collections::BTreeMap< + golem_common::model::ShardId, + ShardEpoch, + >, ) -> Result { unreachable!() } @@ -3257,6 +3365,10 @@ mod rejection_mapping_tests { &self, _executor_id: uuid::Uuid, _shard_epochs: std::collections::BTreeMap, + _fenced_shard_epochs: std::collections::BTreeMap< + golem_common::model::ShardId, + ShardEpoch, + >, ) -> Result { unreachable!() } @@ -3308,8 +3420,20 @@ mod rejection_mapping_tests { } } - #[derive(Clone)] - struct RejectingExecutor; + /// Rejects every invocation as `NotFound`, after first rejecting `routing_misses` of them as + /// having reached an executor that does not own the agent's shard. With `accept` it accepts + /// instead of rejecting as `NotFound`, and keeps the session open until its request stream ends. + #[derive(Clone, Default)] + struct RejectingExecutor { + routing_misses: Arc, + accept: bool, + calls: Arc, + /// The opening frame's freshness disposition, one per call in call order. + dispositions: Arc>>, + /// Receives `(call index, frames after the opening one)` once a call's request stream has + /// ended, which happens after the call returned its response stream. + tail_frames: Option>, + } macro_rules! unimplemented_unary { ($name:ident, $request:ty, $response:ty) => { @@ -3474,37 +3598,98 @@ mod rejection_mapping_tests { ) -> Result, Status> { let mut requests = request.into_inner(); let start = requests.message().await?.expect("missing invocation start"); - let (idempotency_key, agent_id) = match start.request { + let (idempotency_key, agent_id, freshness_disposition) = match start.request { Some(golem_api_grpc::proto::golem::worker::invocation_request::Request::Start( start, - )) => (start.idempotency_key, start.agent_id), + )) => ( + start.idempotency_key, + start.agent_id, + start.freshness_disposition, + ), other => panic!("expected invocation start, got {other:?}"), }; - Ok(Response::new(Box::pin(stream::iter([Ok( - InvocationResponse { - response: Some(invocation_response::Response::Rejected( - InvocationRejected { - reason: InvocationRejectionReason::NotFound as i32, - error: "agent not found".to_string(), - idempotency_key, - agent_id, - component_revision: None, - worker_error: None, - }, - )), - }, - )])))) + let call = self.calls.fetch_add(1, Ordering::SeqCst); + self.dispositions + .lock() + .unwrap() + .push(freshness_disposition); + let routing_miss = self + .routing_misses + .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |left| { + left.checked_sub(1) + }) + .is_ok(); + let response = if !routing_miss && self.accept { + invocation_response::Response::Accepted(InvocationAccepted { + agent_id, + idempotency_key, + ..Default::default() + }) + } else { + let (reason, error, worker_error) = if routing_miss { + let error = WorkerExecutorError::InvalidShardId { + shard_id: golem_common::model::ShardId::new(0), + shard_ids: Vec::new(), + }; + ( + InvocationRejectionReason::Internal, + error.to_string(), + Some(error.into()), + ) + } else { + ( + InvocationRejectionReason::NotFound, + "agent not found".to_string(), + None, + ) + }; + invocation_response::Response::Rejected(InvocationRejected { + reason: reason as i32, + error, + idempotency_key, + agent_id, + component_revision: None, + worker_error, + }) + }; + let accepted = matches!(response, invocation_response::Response::Accepted(_)); + + let (responses, receiver) = tokio::sync::mpsc::channel(1); + responses + .send(Ok(InvocationResponse { + response: Some(response), + })) + .await + .expect("the response stream was dropped before it was returned"); + // A rejected session ends at once, as an executor's does. An accepted one stays open + // until the caller's request stream ends, or its forwarded input could be cut off. + let held_open = accepted.then_some(responses); + let tail_frames = self.tail_frames.clone(); + tokio::spawn(async move { + let mut frames = 0; + // An error ends the count as well: a caller drops a rejected session's request + // stream rather than finishing it. + while let Ok(Some(_)) = requests.message().await { + frames += 1; + } + if let Some(tail_frames) = tail_frames { + let _ = tail_frames.send((call, frames)); + } + drop(held_open); + }); + Ok(Response::new(Box::pin(ReceiverStream::new(receiver)))) } } - #[test] - async fn unary_not_found_rejection_preserves_the_public_error_category() { + /// A worker service client whose only executor is `executor`, and an agent to invoke through + /// it. + async fn client_against(executor: RejectingExecutor) -> (WorkerExecutorWorkerClient, AgentId) { let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let port = listener.local_addr().unwrap().port(); tokio::spawn(async move { tonic::transport::Server::builder() .add_service( - WorkerExecutorServer::new(RejectingExecutor) + WorkerExecutorServer::new(executor) .accept_compressed(CompressionEncoding::Gzip) .send_compressed(CompressionEncoding::Gzip), ) @@ -3547,8 +3732,15 @@ mod rejection_mapping_tests { component_id: ComponentId::new(), agent_id: "missing".to_string(), }; + (client, agent_id) + } + + /// Invokes an agent through a worker service whose only executor is `executor`, and returns + /// the error the invocation ends with. + async fn invoke_against(executor: RejectingExecutor) -> WorkerServiceError { + let (client, agent_id) = client_against(executor).await; - let error = client + client .invoke_agent( &agent_id, Some("run".to_string()), @@ -3568,7 +3760,12 @@ mod rejection_mapping_tests { None, ) .await - .unwrap_err(); + .unwrap_err() + } + + #[test] + async fn unary_not_found_rejection_preserves_the_public_error_category() { + let error = invoke_against(RejectingExecutor::default()).await; let public_error: AgentError = error.into(); assert!( @@ -3576,6 +3773,151 @@ mod rejection_mapping_tests { "InvocationRejected(NotFound) must remain a public not-found error, got {public_error:?}" ); } + + #[test] + async fn a_routing_miss_rejection_is_retried_rather_than_surfaced() { + // An executor that has just lost the agent's shard rejects before accepting. The worker + // service has to retry that on the shard's owner - here the same fake, answering the second + // time - rather than fail the invocation with the first rejection. + let executor = RejectingExecutor { + routing_misses: Arc::new(AtomicUsize::new(1)), + ..Default::default() + }; + let calls = executor.calls.clone(); + + let error = invoke_against(executor).await; + + assert_eq!( + calls.load(Ordering::SeqCst), + 2, + "the routing miss must be retried, exactly once" + ); + let public_error: AgentError = error.into(); + assert!( + matches!(public_error.error, Some(agent_error::Error::NotFound(_))), + "the retried call's answer must be the one surfaced, got {public_error:?}" + ); + } + + fn session_start(agent_id: &AgentId) -> InvocationRequest { + InvocationRequest { + request: Some(invocation_request::Request::Start(InvocationStart { + agent_id: Some(agent_id.clone().into()), + method_name: Some("run".to_string()), + idempotency_key: Some( + golem_common::model::IdempotencyKey::new("session-key".to_string()).into(), + ), + freshness_disposition: WireFreshness::KnownFresh as i32, + ..Default::default() + })), + } + } + + #[test] + async fn a_routing_miss_rejection_on_a_session_is_retried_on_the_shard_owner() { + // A streaming session meets the same routing miss as a unary invocation and has to be + // retried the same way, including giving up KnownFresh once a dispatch may have reached + // an executor. + let executor = RejectingExecutor { + routing_misses: Arc::new(AtomicUsize::new(1)), + ..Default::default() + }; + let calls = executor.calls.clone(); + let dispositions = executor.dispositions.clone(); + let (client, agent_id) = client_against(executor).await; + + let responses = client + .invoke_agent_session( + &agent_id, + Box::pin(stream::iter([session_start(&agent_id)])), + ) + .await + .expect("the session was not dispatched"); + let responses = + tokio::time::timeout(Duration::from_secs(30), responses.collect::>()) + .await + .expect("the session's response stream never ended"); + + assert_eq!( + calls.load(Ordering::SeqCst), + 2, + "the routing miss must be retried, exactly once" + ); + assert_eq!( + *dispositions.lock().unwrap(), + vec![ + WireFreshness::KnownFresh as i32, + WireFreshness::MayExist as i32 + ] + ); + match responses.as_slice() { + [ + Ok(InvocationResponse { + response: Some(invocation_response::Response::Rejected(rejected)), + }), + ] => assert_eq!( + rejected.reason, + InvocationRejectionReason::NotFound as i32, + "the retried call's answer must be the one surfaced" + ), + other => panic!("expected only the retried call's rejection, got {other:?}"), + } + } + + #[test] + async fn session_input_reaches_only_the_executor_that_accepted() { + // No input may precede acceptance, so the executor that turned the session away must have + // seen the opening frame alone, and the caller's input must reach the one that accepted. + let (tail_frames, mut ended) = tokio::sync::mpsc::unbounded_channel(); + let executor = RejectingExecutor { + routing_misses: Arc::new(AtomicUsize::new(1)), + accept: true, + tail_frames: Some(tail_frames), + ..Default::default() + }; + let (client, agent_id) = client_against(executor).await; + let request = stream::iter([ + session_start(&agent_id), + InvocationRequest { + request: Some(invocation_request::Request::InputEnd( + InputStreamEnd::default(), + )), + }, + ]); + + // Held until the end: dropping it would cancel the session before its input arrived. + let mut responses = client + .invoke_agent_session(&agent_id, Box::pin(request)) + .await + .expect("the session was not dispatched"); + let first = tokio::time::timeout(Duration::from_secs(30), responses.next()) + .await + .expect("the session sent no first frame") + .expect("the session's response stream ended without a frame"); + assert!( + matches!( + first, + Ok(InvocationResponse { + response: Some(invocation_response::Response::Accepted(_)), + }) + ), + "the caller must see the acceptance first, got {first:?}" + ); + + let mut frames_per_call = BTreeMap::new(); + while frames_per_call.len() < 2 { + let (call, frames) = tokio::time::timeout(Duration::from_secs(30), ended.recv()) + .await + .expect("a dispatch's request stream never ended") + .expect("the executor stopped reporting"); + frames_per_call.insert(call, frames); + } + assert_eq!( + frames_per_call, + BTreeMap::from([(0, 0), (1, 1)]), + "input must reach only the executor that accepted" + ); + } } #[cfg(test)] diff --git a/golem-worker-service/src/service/worker/routing_logic.rs b/golem-worker-service/src/service/worker/routing_logic.rs index 93968044cf..801db760a1 100644 --- a/golem-worker-service/src/service/worker/routing_logic.rs +++ b/golem-worker-service/src/service/worker/routing_logic.rs @@ -551,6 +551,10 @@ impl<'a> RetryState<'a> { op = self.op, "Retry calling executor reached the maximum attempts" ); + // The attempt counter restarts and the loop goes on, so the wait has to happen + // here too, or an executor that keeps refusing is retried as fast as the network + // allows. + sleep(delay).await; Ok(None) } } diff --git a/integration-tests/tests/sharding.rs b/integration-tests/tests/sharding.rs index 79ff76edbd..099f9098fc 100644 --- a/integration-tests/tests/sharding.rs +++ b/integration-tests/tests/sharding.rs @@ -22,6 +22,8 @@ mod tests { use bytes::Bytes; use golem_api_grpc::proto::golem::worker; use golem_client::api::RegistryServiceClient; + #[cfg(unix)] + use golem_common::model::AgentId; use golem_common::model::base64::Base64; use golem_common::model::component::ComponentDto; use golem_common::model::environment_plugin_grant::EnvironmentPluginGrantCreation; @@ -30,6 +32,8 @@ mod tests { OplogProcessorPluginSpec, PluginRegistrationCreation, PluginSpecDto, }; use golem_common::model::{AgentStatus, IdempotencyKey, OplogIndex}; + #[cfg(unix)] + use golem_common::schema::SchemaValue; use golem_common::tracing::{TracingConfig, init_tracing_with_default_debug_env_filter}; use golem_common::{agent_id, data_value}; use golem_test_framework::components::rdb::DbInfo; @@ -277,6 +281,242 @@ mod tests { chaos.await.unwrap(); } + // Pausing an executor is SIGSTOP. + #[cfg(unix)] + #[test] + #[timeout(360000)] + // Not `#[flaky]`, unlike the scenarios above: what this pins is that a duplicate never + // happens, and a retry would hide one that only happens some of the time. + async fn an_executor_paused_until_its_shards_move_cannot_finish_the_invocations_it_started( + deps: &EnvBasedTestDependencies, + cluster_control: &WorkerExecutorClusterControlStub, + _tracing: &Tracing, + ) { + // Under the executor's `suspend_after` (10s by default), so the sleep runs in flight instead + // of suspending the agent: the executors freeze mid-invocation, which is the case a lease + // alone cannot stop - a frozen executor wakes up still holding work it can try to finish. + const DELAY_MILLIS: u64 = 8_000; + + deps.reset(cluster_control, 16).await; + let admin = deps.admin().await; + let (_, env) = admin.app_and_env().await.unwrap(); + let component = admin + .component(&env.id, "it_agent_counters_release") + .name("it:agent-counters") + .store() + .await + .unwrap(); + + let mut agents = Vec::new(); + for i in 1..=8 { + let parsed_agent_id = agent_id!("InstantiationGrowthCounter", format!("fenced-{i}")); + let agent_id = admin + .start_agent(&component.id, parsed_agent_id.clone()) + .await + .unwrap(); + agents.push((parsed_agent_id, agent_id)); + } + + // Read before anything is in flight, so that connecting does not eat into the delay the + // executors have to be frozen inside. + let pool = match deps.rdb().info() { + DbInfo::Postgres(pg) => sqlx::PgPool::connect(&pg.public_connection_string()) + .await + .expect("Failed to connect to Postgres"), + _ => panic!("this test only implements reading the stored owning epochs from Postgres"), + }; + let mut initial_epochs = Vec::new(); + for (parsed_agent_id, agent_id) in &agents { + let epoch = stored_owning_epoch(&pool, agent_id) + .await + .unwrap_or_else(|| { + panic!("{parsed_agent_id}: the owning epoch is recorded before the first entry") + }); + initial_epochs.push(epoch); + } + + let mut invocations = JoinSet::new(); + for (parsed_agent_id, _) in &agents { + let user = deps.admin().await; + let component = component.clone(); + let parsed_agent_id = parsed_agent_id.clone(); + invocations.spawn( + async move { + let result = user + .invoke_and_await_agent_with_key( + &component, + &parsed_agent_id, + &IdempotencyKey::fresh(), + "delayed_increment", + data_value!(DELAY_MILLIS), + ) + .await; + (parsed_agent_id, result) + } + .in_current_span(), + ); + } + + // Every invocation is inside its sleep now, on whichever executor owns its agent. Freezing + // all executors but one leaves the survivor as the only one to take their shards over, so + // nothing here needs to know which executor owned which agent. + tokio::time::sleep(Duration::from_secs(2)).await; + let started = cluster_control.started_indices().await; + let (survivor, frozen) = started + .split_first() + .expect("the reset starts every executor"); + info!("Pausing worker executors {frozen:?}, keeping {survivor}"); + for idx in frozen { + cluster_control.pause(*idx).await; + } + + // Long enough for the shard manager to have taken the frozen executors' shards away and + // granted them to the survivor at a higher epoch, so that it recovers and finishes their + // invocations itself, and past the delay, so that the frozen executors' own sleeps are over + // the moment they wake. What normally moves the shards is the shard manager's health check, + // which unregisters an executor once its probes and their retries have gone unanswered, + // usually well before its lease runs out; and the worker service's keep-alive drops the + // calls stuck on a frozen executor, so that they are retried against the new owner. The + // lease is only the upper bound on the move: at the default 60s, renewed every 20s and + // reaped on a 20s tick, an unrenewed lease is gone within 80s of the pause. That bound + // keeps a thawed executor from waking up as the owner; it does not stretch the callers' + // retries, which count on the health check. + tokio::time::sleep(Duration::from_secs(90)).await; + + // Thawed, the frozen executors carry on from exactly where they stopped, still holding + // invocations the survivor now owns. For each one, either the executor gives the agent up + // when it re-registers, or it finishes the sleep first and the fence refuses its write. + // Which of the two each agent took is a race this cannot steer, and both are safe only + // because the survivor recorded a higher epoch first, which the epoch check below asserts. + // The refusal itself is pinned deterministically by golem-worker-executor's oplog and + // indexed-storage tests; in a live cluster it is visible as + // `oplog_epoch_fence_total{op="append",outcome="refused"}`. + info!("Resuming worker executors {frozen:?}"); + for idx in frozen { + cluster_control.resume(*idx).await; + } + + while let Some(joined) = + tokio::time::timeout(Duration::from_secs(180), invocations.join_next()) + .await + .expect("Timed out waiting for the invocations to finish") + { + let (parsed_agent_id, result) = joined.unwrap(); + let value = result + .unwrap_or_else(|err| panic!("{parsed_agent_id}: invocation failed: {err:?}")) + .into_return_value() + .unwrap_or_else(|| panic!("{parsed_agent_id}: expected a return value")); + assert_eq!( + value, + SchemaValue::U32(1), + "{parsed_agent_id}: the delayed increment must be applied exactly once" + ); + } + + let mut owner_changed = false; + for ((parsed_agent_id, agent_id), initial) in agents.iter().zip(&initial_epochs) { + let current = stored_owning_epoch(&pool, agent_id) + .await + .unwrap_or_else(|| { + panic!("{parsed_agent_id}: the owning epoch is no longer recorded") + }); + assert!( + current >= *initial, + "{parsed_agent_id}: the stored epoch went down from {initial} to {current}" + ); + owner_changed |= current > *initial; + } + pool.close().await; + assert!( + owner_changed, + "no agent's shard changed owner during the test, so the run exercised neither the \ + give-up nor the fence" + ); + + assert_every_started_executor_serves(cluster_control, "while the invocations finished") + .await; + + for (parsed_agent_id, agent_id) in &agents { + assert_eq!( + count_completions_of(&admin, agent_id, "delayed_increment").await, + 1, + "{parsed_agent_id}: exactly one executor may record the invocation's completion" + ); + // The agent's state agrees: one increment from the delayed call, one from this. + let next = admin + .invoke_and_await_agent(&component, parsed_agent_id, "increment", data_value!()) + .await + .unwrap() + .into_return_value() + .unwrap_or_else(|| panic!("{parsed_agent_id}: expected a return value")); + assert_eq!( + next, + SchemaValue::U32(2), + "{parsed_agent_id}: the delayed increment must not have been applied twice" + ); + } + + // Again after the follow-up calls: they are the first new traffic after the thaw, and the + // first chance for a re-registered executor to recover agents on a shard it was given back. + assert_every_started_executor_serves( + cluster_control, + "while the agents were invoked again", + ) + .await; + } + + /// Asserts that no started executor has died: a write the fence let through would land in an + /// oplog the survivor is writing too, and a conflicting append there is fatal to the executor + /// that makes it - which can be the rightful owner. Checked through the health endpoint rather + /// than process liveness: an aborting process stays "running" until the OS has finished + /// writing its crash report, which can outlast this test. Each call is a single probe, so it + /// only covers what happened before it. + #[cfg(unix)] + async fn assert_every_started_executor_serves( + cluster_control: &WorkerExecutorClusterControlStub, + during: &str, + ) { + for idx in cluster_control.started_indices().await { + assert!( + cluster_control.is_serving(idx).await, + "worker executor {idx} stopped serving {during}" + ); + } + } + + /// The owning epoch the executors' indexed storage holds for `agent_id`'s oplog. Read from the + /// storage itself because the public oplog does not carry the shard epoch. + #[cfg(unix)] + async fn stored_owning_epoch(pool: &sqlx::PgPool, agent_id: &AgentId) -> Option { + sqlx::query_scalar( + "SELECT epoch FROM golem_worker_executor_indexed.indexed_key_epoch \ + WHERE namespace = 'durable-worker-oplog' AND key = $1", + ) + .bind(agent_id.to_redis_key()) + .fetch_optional(pool) + .await + .expect("Failed to read indexed_key_epoch") + } + + /// Counts the completions `agent_id`'s oplog records for `method`. Scoped to one method on + /// purpose: an agent's own initialization completes asynchronously after `start_agent` returns, + /// so any count taken over every method races it. + #[cfg(unix)] + async fn count_completions_of(user: &impl TestDsl, agent_id: &AgentId, method: &str) -> usize { + user.get_oplog(agent_id, OplogIndex::INITIAL) + .await + .unwrap() + .into_iter() + .filter(|entry| { + matches!( + &entry.entry, + PublicOplogEntry::AgentInvocationFinished(params) + if params.method_name.as_deref() == Some(method) + ) + }) + .count() + } + async fn coordinated_scenario( deps: &EnvBasedTestDependencies, cluster_control: &WorkerExecutorClusterControlStub, diff --git a/local-run/start.sh b/local-run/start.sh index 49fd6c1e6f..305a3cae8f 100644 --- a/local-run/start.sh +++ b/local-run/start.sh @@ -16,8 +16,10 @@ fi LOCAL_RUN_DIR="${GOLEM_DIR}/local-run" -rm -rf "${LOCAL_RUN_DIR}/data/shard-manager" -mkdir -pv "${LOCAL_RUN_DIR}/data/redis" "${LOCAL_RUN_DIR}/data/shard-manager" "${LOCAL_RUN_DIR}/logs" +# Wipe the executor's indexed storage along with the shard manager's state: the oplog epochs it +# records were minted by that state and are ahead of everything a fresh one mints. +rm -rf "${LOCAL_RUN_DIR}/data/shard-manager" "${LOCAL_RUN_DIR}/data/worker-executor" +mkdir -pv "${LOCAL_RUN_DIR}/data/redis" "${LOCAL_RUN_DIR}/data/shard-manager" "${LOCAL_RUN_DIR}/data/worker-executor" "${LOCAL_RUN_DIR}/logs" # start redis # Redis persistence isn't needed for local-run, and misconfigured snapshotting can force Redis into @@ -137,6 +139,10 @@ GOLEM__SHARD_MANAGER__HOST="localhost" \ GOLEM__SHARD_MANAGER__PORT=${SHARD_MANAGER_GRPC_PORT} \ GOLEM__SHARD_MANAGER__RETRIES__MAX_ATTEMPTS=10 \ GOLEM__SHARD_MANAGER__RETRIES__MIN_DELAY=1s \ +GOLEM__INDEXED_STORAGE__TYPE="Sqlite" \ +GOLEM__INDEXED_STORAGE__CONFIG__DATABASE="../local-run/data/worker-executor/golem_indexed.sqlite" \ +GOLEM__INDEXED_STORAGE__CONFIG__MAX_CONNECTIONS=10 \ +GOLEM__INDEXED_STORAGE__CONFIG__FOREIGN_KEYS=false \ ../target/debug/worker-executor & worker_executor_pid=$!