CASSSIDECAR-374: Implement durable operational job tracker by andresbeckruiz · Pull Request #359 · apache/cassandra-sidecar

andresbeckruiz · 2026-06-10T20:34:25Z

CASSSIDECAR-374

Original PR made against CASSSIDECAR-373 branch with review comments: andresbeckruiz#3.

Changes

DurableOperationalJobTracker: Implementation of OperationalJobTracker that persists job state via StorageProvider. Live local jobs are cached in a local ConcurrentHashMap, completed jobs are served from storage
OperationalJobManager: Separate job tracking from execution — the job is registered in the tracker (and persisted) before execution begins, ensuring durable trackers always have a record before the job runs
Unit tests for DurableOperationalJobTracker and persist-before-execute verification in OperationalJobManagerTest
Integration test for DurableOperationalJobTracker with Cassandra-backed StorageProvider

Future Work

CASSSIDECAR-378: Add support for creation and coordination of local Sidecar Jobs

pauloricardomg

I noticed the durable job tracker is never initialized, do we plan to do this in a follow-up PR ? Is the idea that the operator will choose whether to use the durable or the in memory job tracker ?

pauloricardomg · 2026-06-12T16:10:34Z

+    @Nullable
+    public UUID nodeId()
+    {
+        return null;


Why did we not include this field in the cluster ops schema?

Also why is this field needed, I don't see it being used anywhere.

nodeId was not added to the cluster ops schema because for cluster-wide records, there's no single associated node. Per-node state is tracked separately in the cluster_ops_node_state table, and the nodes included in a job are recorded in the node_execution_order column.

nodeId is part of the OperationalJobInfo interface, which was designed to capture the common read-only fields from OperationalJob. nodeId already existed for single-node jobs (drain, decommission, move). Since OperationalJobRecord now implements this interface, it returns null since cluster-wide records aren't associated with a single node.

pauloricardomg · 2026-06-12T16:11:15Z

+    @NotNull
+    public List<UUID> nodesExecuting()
+    {
+        return Collections.emptyList();


Is there any reason why we're not populating these nodes* fields from the cluster_ops_node_state table ?

Populated these fields in d8d047e

pauloricardomg · 2026-06-12T19:03:15Z

+     * @param job the operational job to convert
+     * @return a new record capturing the job's current state
+     */
+    public static OperationalJobRecord fromOperationalJob(OperationalJob job)


When a job completes, the DurableOperationalJobTracker evicts it from the live map. After that, get(jobId) returns
an OperationalJobRecord from storage. Let's trace what data is available at each stage:

While the job is live (e.g., NodeDecommissionJob): nodeId() → UUID of the node being decommissioned startTime() → when execution began nodesPending() → [nodeId] initially, then [] after execution starts nodesExecuting()→ [nodeId] while running nodesSucceeded()→ [nodeId] after success nodesFailed() → [nodeId] after failure

The factory that persists the job only captures 3 fields:

// OperationalJobRecord.fromOperationalJob(job) return new OperationalJobRecord(job.jobId(), job.operationType(), job.status()); // ^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^^^^^ // that's it — nodeId, startTime, node lists are all discarded

After completion, get() returns the record with stubbed implementations:

nodeId() → null (always) startTime() → null (constructor sets it to null) nodesPending() → emptyList (always) nodesExecuting() → emptyList (always) nodesSucceeded() → emptyList (always) nodesFailed() → emptyList (always)

So an API client observing a decommission job would see:

DURING execution: AFTER completion (from storage): ───────────────────────────────── ───────────────────────────────── { { "jobId": "abc-123", "jobId": "abc-123", "operation": "decommission", "operation": "DECOMMISSION", "jobStatus": "RUNNING", "jobStatus": "SUCCEEDED", "startTime": "2026-06-12T...", "startTime": null, "nodesExecuting": ["node-uuid"], "nodesExecuting": [], "nodesPending": [], "nodesPending": [], "nodesSucceeded": [], "nodesSucceeded": [], "nodesFailed": [] "nodesFailed": [] } }

All the provenance — which node was decommissioned, when it started, whether it succeeded per-node — is lost once the job leaves the
local map. The record in Cassandra only has (jobId, operationType, status).

The record in Cassandra is intended to start with this information. The nodes list information concern is addressed in d8d047e.

For start time, the OperationalJobCoordinator introduced in CASSSIDECAR-377 will be responsible for setting this when calling updateJobStatus to set the status to RUNNING. See:

cassandra-sidecar/server/src/main/java/org/apache/cassandra/sidecar/db/ClusterOpsDatabaseAccessor.java

Lines 117 to 122 in c578297

else if (status == OperationalJobStatus.RUNNING && shouldSetStartTime(clusterName, jobId))

{

statement = tableSchema.updateStatusWithStartTime()

.bind(status.name(), lastUpdate, lastUpdate,

clusterName, jobId, operationType.name());

}

pauloricardomg · 2026-06-12T19:20:13Z

+            storageProvider.persistJob(OperationalJobRecord.fromOperationalJob(job));
+
+            job.asyncResult().onComplete(ar -> {
+                liveJobs.remove(job.jobId());


liveJobs removal should come after updateTerminalStatus to avoid RUNNING → CREATED → SUCCEEDED. when polling status between removal and update

Addressed in adee1db

pauloricardomg · 2026-06-12T19:23:20Z

+        return liveJobs.computeIfAbsent(jobId, id -> {
+            OperationalJob job = mappingFunction.apply(id);
+
+            storageProvider.persistJob(OperationalJobRecord.fromOperationalJob(job));


what happens if !storageProvider.isAvailable ?

Added guard in 464f03e

pauloricardomg · 2026-06-12T19:23:52Z

+                                        TaskExecutorPool executor)
+    {
+        this.liveJobs = new ConcurrentHashMap<>(serviceConfiguration.operationalJobTrackerSize());
+        this.storageProvider = storageProvider;


who is expected to initialize the storage provider ? can this class assume it's already initialized or do we want to initialize here?

The StorageProvider will be initialized externally. DurableOperationalJobTracker assumes it's already initialized and guards against unavailability with the isAvailable() check added in 464f03e.

The idea is that an operator will choose between the durable or in-memory tracker via configuration. I'm deferring these edits to sidecar.yaml to a follow up PR once the config is actually being used by a cluster-wide operation.

pauloricardomg · 2026-06-12T19:25:51Z

+                                        StorageProvider storageProvider,
+                                        TaskExecutorPool executor)
+    {
+        this.liveJobs = new ConcurrentHashMap<>(serviceConfiguration.operationalJobTrackerSize());


Unlike InMemoryOperationalJobTracker which has removeEldestEntry, the durable tracker has no eviction. If
asyncResult() never completes (e.g., executeBlocking fails to submit due to full pool), the entry stays in the map forever. Is this expected?

Good catch. If executeBlocking fails to submit due to a full pool, the entry will stay in the map forever. As mentioned in the comment below, I will defer a cleanup job for stale entries to a follow up PR, unless you think it should be addressed here.

skoppu22 · 2026-06-18T14:16:10Z

+        {
+            LOGGER.warn("Failed to update terminal status for job {} (attempt {}/{}). error={}",
+                        job.jobId(), attempt, MAX_STATUS_UPDATE_ATTEMPTS, e.getMessage());
+            if (attempt < MAX_STATUS_UPDATE_ATTEMPTS)


This method is called from above method with attempt=1, if 1st attempt fails, we are doing only one more retry. Should we increase value of MAX_STATUS_UPDATE_ATTEMPTS ?

Also, if retries also failed to update, we will have stale entries in CREATED stated. We need a periodic thread running once in a while and removing stale entries (this can be deferred to later as not a blocker).

Increased MAX_STATUS_UPDATE_ATTEMPTS to 3 in d8a6bf9.

Agreed that there should be a periodic thread to remove stale entries in a later patch.

For 3 attempts it's probably fine as-is, but consider adding jitter (see ClusterLeaseClaimTask.initialDelay() for existing precedent in this codebase) so that concurrent sidecar processes retrying against the same transient Cassandra blip don't all retry in lockstep.

…onalJobTracker

…provider unavailable guard

andresbeckruiz · 2026-07-07T20:18:55Z

CI: https://app.circleci.com/pipelines/github/andresbeckruiz/cassandra-sidecar/97/workflows/644be457-446b-475d-be14-1ea9b9953089

skoppu22 · 2026-07-08T17:26:08Z

+                             jobId, e);
+            });
+
+            job.asyncResult().onComplete(ar -> {


I believe we need to chain this code into above executeBlocking's onSuccess to ensure the this runs only after state persisted.

Good catch. Chained the completion handler into the persist's onSuccess so we only register it after the CREATED record is persisted, otherwise a failed persist would still call updateTerminalStatus on a record that was never persisted. Also added a test covering that case in 373e367

skoppu22 · 2026-07-09T11:17:18Z

+
+        assertThat(completed.await(5, TimeUnit.SECONDS)).isTrue();
+        // The tracker registers its completion handler before this test does, so a stale
+        // updateJobStatus would already have run once the job has completed.


[minor] looks like this comment needs to be updated, as we expect updateJobStatus to not run

Addressed in b59c9d9

skoppu22 · 2026-07-09T11:19:27Z

+        job.asyncResult().onComplete(ar -> latch.countDown());
+        assertThat(latch.await(5, TimeUnit.SECONDS)).isTrue();
+
+        loopAssert(2, () -> {


Is this supposed to be 3 as we updated MAX_STATUS_UPDATE_ATTEMPTS to 3?

Addressed in b59c9d9

arjunashok · 2026-07-09T18:55:45Z

Noticed trySetActiveOperation/clearActiveOperation/getActiveOperation on StorageProvider aren't called anywhere yet.

Conflict checks (checkConflict/inflightJobsByOperation) are still scoped to one sidecar's in-memory liveJobs, not the cross-cluster CAS lock the interface describes. Is wiring that up deferred to CASSSIDECAR-378, or expected in this PR?

arjunashok

Seems like only two states reach storage: CREATED state on submit and the terminal status on completion. RUNNING and per-node progress (updateNodeStatus/updateNodeStatuses) are not persisted, and there's no startup/periodic reconciliation (findAllJobs is currently only called from tests).

So a sidecar restart mid-decommission/move/repair leaves the record stuck at CREATED (same symptom as the retry-exhaustion case in my comment), just more common.

Worth calling out explicitly as a known limitation for long-running jobs (RestoreJobDiscoverer already does this for restore jobs, if useful as reference).

arjunashok · 2026-07-09T18:36:35Z

+        if (created[0])
+        {
+            executor.executeBlocking(() -> {
+                storageProvider.persistJob(OperationalJobRecord.fromOperationalJob(job));


Seems like this initial persist gets zero retries and gets removed from liveJobs

Since OperationalJobManager.trySubmitJob already kicked off job.execute() on a separate executor before this callback runs, the job keeps running against Cassandra while the tracker has already forgotten it.

Subsequently, get(jobId) returns null, and inflightJobsByOperation() no longer sees it, and could let a second decommission/move/repair job through while the first is still executing underneath.

Should this keep the job in liveJobs (serving it in a durability-degraded state) rather than dropping it entirely on persist failure?

Good idea, addressed in 696b576 and added a test to ensure job is maintained in liveJobs until completion, regardless of persistJob success.

Thanks. The initial persistJob call still gets exactly one attempt, no retry so a transient storage blip at creation time could downgrade that job to in-memory-only tracking for its lifetime

Worth giving the initial persist the same (3x + jitter)short retry treatment, or is that intentionally out of scope here?

arjunashok · 2026-07-09T18:49:23Z

+        {
+            storageProvider.updateJobStatus(job.jobId(), job.operationType(), job.status(), job.failureReason());
+        }
+        catch (RuntimeException e)


Now that retries are bumped to 3, when all exhaust, the storage record is left at the state it last successfully wrote (typically CREATED). There's no flag distinguishing 'this record's status is stale/unreliable' from 'this job genuinely hasn't started.'

A caller polling GET /job/{id} sees "not started yet" for a job that finished, which can either hang waiting for a terminal state that will never come, or time out and resubmit the same operation (which is not safe for decommission/move). It also blocks the periodic reconciliation sweep we want to add as we would need a way to tell "genuinely still CREATED" from "we know this finished but failed to record it".

Couple of options: (a) a metadata field on OperationalJobRecord marking it durability-degraded, checked only by the reconciliation sweep (which can come later), or (b) a new OperationalJobStatus value (e.g. UNKNOWN) visible through the same field every existing caller already reads.

This is a good point. However, a caller wouldn't see CREATED for an actual running job in the first place: get() serves the live job from liveJobs first, so an in-progress job reports RUNNING, not the storage record. We only fall back to the stored CREATED after the job has truly finished (since updateTerminalStatus runs only on completion), or after a restart, when liveJobs is empty.

Your comment above still holds true if Sidecar restarts-- we could resubmit an operation even if it's running and says "CREATED" in storage. However, this is not a regression, since the current in memory tracker returns null from get() once the job leaves memory, so it's equally capable of making a client think that the job has not started or does not exist and then resubmit.

Given there is no regression, I would probably defer choosing between a) and b) until a PR is made for a reconciliation sweep later on that will actually use it. If this sounds good, I can make a JIRA documenting this follow up. I have also added a Javadoc documenting the limitation and follow up in b238bd4

Agreed on deferring the design choice until the reconciliation sweep PR. The Javadoc covers it well. Could you file that JIRA(unless one already exists) and link it here (and maybe in the class doc) so it's tracked? Thanks.

arjunashok · 2026-07-09T18:50:55Z

+        {
+            LOGGER.warn("Failed to update terminal status for job {} (attempt {}/{}). error={}",
+                        job.jobId(), attempt, MAX_STATUS_UPDATE_ATTEMPTS, e.getMessage());
+            if (attempt < MAX_STATUS_UPDATE_ATTEMPTS)


For 3 attempts it's probably fine as-is, but consider adding jitter (see ClusterLeaseClaimTask.initialDelay() for existing precedent in this codebase) so that concurrent sidecar processes retrying against the same transient Cassandra blip don't all retry in lockstep.

andresbeckruiz force-pushed the CASSSIDECAR-374 branch 2 times, most recently from b43cc8a to 346a446 Compare June 12, 2026 19:14

pauloricardomg reviewed Jun 12, 2026

View reviewed changes

skoppu22 reviewed Jun 18, 2026

View reviewed changes

andresbeckruiz requested review from pauloricardomg and skoppu22 July 1, 2026 20:44

andresbeckruiz added 7 commits July 1, 2026 16:47

CASSSIDECAR-374: Implement durable operational job tracker

c09846d

Address PR review comments

31e787a

Populate nodes list fields in OperationalJobRecord via DurableOperati…

4be1de3

…onalJobTracker

Return lower case name from OperationalJobRecord

3f52509

Remove from live jobs after updateTerminalStatus

6648515

Move job persistence outside of CHM lock and event loop, add storage …

af8364c

…provider unavailable guard

Update MAX_STATUS_UPDATE_ATTEMPTS to 3

c7256ca

andresbeckruiz force-pushed the CASSSIDECAR-374 branch from d8a6bf9 to c7256ca Compare July 1, 2026 20:49

Fix persistence failure test

a0a08ea

andresbeckruiz force-pushed the CASSSIDECAR-374 branch from 998d9f2 to a0a08ea Compare July 1, 2026 21:40

skoppu22 reviewed Jul 8, 2026

View reviewed changes

pauloricardomg approved these changes Jul 8, 2026

View reviewed changes

Update terminal status after successful persist

373e367

skoppu22 reviewed Jul 9, 2026

View reviewed changes

skoppu22 approved these changes Jul 9, 2026

View reviewed changes

Add fixes to DurableOperationalJobTrackerTest

b59c9d9

arjunashok reviewed Jul 9, 2026

View reviewed changes

andresbeckruiz added 3 commits July 9, 2026 15:49

Add retry delay jitter to updateTerminalStatus

8ce2cc5

Maintain job in liveJobs if persistJob fails

696b576

Add Javadoc documenting DurableOperationalJobTracker limitation

b238bd4

	else if (status == OperationalJobStatus.RUNNING && shouldSetStartTime(clusterName, jobId))
	{
	statement = tableSchema.updateStatusWithStartTime()
	.bind(status.name(), lastUpdate, lastUpdate,
	clusterName, jobId, operationType.name());
	}

Uh oh!

Conversation

andresbeckruiz commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Future Work

Uh oh!

pauloricardomg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjunashok commented Jul 9, 2026

Uh oh!

arjunashok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

andresbeckruiz commented Jun 10, 2026 •

edited

Loading

andresbeckruiz Jul 1, 2026 •

edited

Loading

andresbeckruiz Jul 9, 2026 •

edited

Loading