[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6

[filimonov]

What this is

A new opt-in metadata storage type cas for object-storage disks: MergeTree parts are stored
as content-addressed blobs in a shared S3/GCS bucket pool, deduplicated by content hash, with
part manifests and named refs on top — "git for MergeTree". Multiple servers mount the same
pool and share identical data blocks without re-uploading them; same-pool replication fetches
publish a local ref over the shared blobs instead of copying bytes (fetch-by-relink). Garbage
collection runs as a lease-coordinated background round with full audit trail; integrity is
checkable online with SYSTEM CAS FSCK and the clickhouse-disks cas-* applets.

User-facing surface: the cas disk/metadata-storage type, SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tables system.cas_log, system.cas_gc_log,
system.cas_mounts, and the clickhouse-disks commands cas-inspect, cas-fsck,
cas-gc-dryrun, cas-gc-rebuild, cas-drop-member. Full documentation ships in this PR under
docs/en/antalya/cas/ (architecture and operations), plus the system-table pages.

How the series is structured

29 commits, ordered for review; the subsystem lands dark and is switched on by one registration
line.

1. Upstream fixes (11 commits) — standalone correctness fixes to shared code, each
   reviewable on its own: ReadBufferFromFileView position tracking, ReadBufferFromS3
   retry-after-cancel, ThreadGroup parent lifetime, MergeTreeDeduplicationLog fail-closed
   null-writer, S3 conditional writes + 412 no-retry policy, Expect: 100-continue for large
   conditional uploads, GCS conditional-write dialect and GOOG4 signer, LocalObjectStorage
   snapshot listing semantics, the disk-transaction contract (one logical part = one
   transaction, read-your-writes), SYSTEM-on-proxy unwrap, and clickhouse-disks non-zero exit
   code for failed non-interactive commands.
2. The CAS subsystem (8 commits) — a layered library under
   src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/, bottom-up:
   Primitives → Formats → Backend → Pool → Parts → Gc → Tools. No layer is reachable until
   the wiring lands.
3. Integration (4 commits) — the metadata storage and disk transaction, capability
   predicates, the registration line, system logs and tables, SYSTEM commands, registries and
   entry points, fetch-by-relink in DataPartsExchange.
4. Tests (3 commits) — the gtest battery (134 files), stateless and integration suites.
5. CI wiring and docs (3 commits) — CAS-default test lanes, tags for tests that cannot run
   on CAS, and the documentation set.

Behavior changes to shared code (reviewers, look here)

Everything CAS-specific is gated behind the cas disk type; a server without a cas disk gets
identical behavior except for these deliberate fixes:

• clickhouse-disks --query now exits non-zero when a command fails (previously always 0);
  interactive sessions unaffected. This exposed and fixes two latent test defects
  (test_disks_app_func path typo, test_replicated_database reading metadata_path after
  DETACH).
• HTTP 412 on conditional S3 requests is never retried (deterministic failure; also benefits
  pre-existing Iceberg conditional writes).
• The disk-transaction contract commit routes projection sub-parts through the parent
  whole-part transaction and makes staged state readable before commit — behavior-preserving on
  local disks, stricter on object-storage metadata.
• ReadBufferFromFileView position fix also affects the pre-existing packed skip-index reader.

Verification

• Per-layer gtest battery (src/Disks/tests/, 134 files) plus stateless and integration
  suites; CI runs a CAS-default-disk lane over the general stateless set.
• Long-running soak/chaos harness (multi-node shared pool, fault injection, fsck-verified
  invariants) has been green on this tree; the harness itself is intentionally not part of this
  PR.
• The branch is a clean reconstruction of the development tree: the diff between this series
  and the development branch, restricted to shipped paths, is empty by construction.

Developed with AI assistance (Claude); every commit carries Co-Authored-By and
Signed-off-by.

Changelog category (leave one):

• Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added experimental content-addressed storage (CAS) for MergeTree: a new cas metadata storage
type for object-storage disks that stores parts as content-hash-deduplicated blobs in a shared
S3/GCS pool with manifests and refs, allowing multiple servers to share identical data without
re-uploading it, replicate same-pool parts without transferring bytes, and reclaim unreferenced
data with lease-coordinated garbage collection. Includes SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tables system.cas_log, system.cas_gc_log,
system.cas_mounts, and clickhouse-disks commands cas-inspect, cas-fsck, cas-gc-dryrun,
cas-gc-rebuild, cas-drop-member.

Documentation entry for user-facing changes

Documentation is included in this pull request: docs/en/antalya/cas/ (architecture and
operations), docs/en/operations/storing-data.md, docs/en/sql-reference/statements/system.md,
and the three system-table pages.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [ceee42c]

[mkmkme]

@blau-ai

[blau-ai]

CI triage for ceee42c (feature/antalya-26.6/CAS)

Reviewed the praktika reports + TestFlows regression reports for the latest head. There is a lot of red, but almost all of it is noise — I found one clearly PR-caused product bug, plus the CAS-specific regression lanes that need your eyes. Everything else is pre-existing on the (also-red) base, infra/Spark, an image CVE, or CI-confirmed flakiness.

Verdict

• PR-caused (real): 02265_column_ttl — reproducible CAS relink failure. ← the one to fix.
• PR-related (CAS lanes, need your review): the cas_* / cas_s3_cache_* TestFlows regression suites.
• Not PR-caused: all Iceberg integration failures, the non-CAS regression suites, the Grype scan, the arm_tsan stress "Unknown error", and ~6 one-off stateless failures (CI's own reruns pass).

---

1. PR-caused — 02265_column_ttl (Stateless, cas s3 storage lanes)

Fails on two independent lanes (amd_asan_ubsan, cas s3 storage 1/2 and arm_binary, cas s3 storage), and CI's randomized-settings rerun makes it reproducible, not flaky: 6/7 and 13/14 reruns fail. The exception:

Code: 210. DB::Exception: Source localhost did not prove it still holds the manifest
it offered for part 20100101_0_0_1 by relink; the relink is abandoned and the fetch
will be retried later. (NETWORK_ERROR)
  5. DataPartsExchange::Fetcher::relinkPartToDisk (DataPartsExchange.cpp:1549)
  6. DataPartsExchange::Fetcher::fetchSelectedPart
  7. StorageReplicatedMergeTree::fetchPart
  9. StorageReplicatedMergeTree::executeFetch

This is squarely PR code — the PR modifies src/Storages/MergeTree/DataPartsExchange.cpp/.h, and the message is the T2 CONFIRM step of the CAS relink handshake (DataPartsExchange.cpp:1543-1552). What the test exercises: a replicated + TTL table (ttl_02265_r2) drives a merge, r2 fetches the merged part from r1 over the relink path, and the confirm comes back not-proven (source_proved_the_binding == false), so the fetch is thrown into the retry-later class. The retry mechanism itself is by design, but it's firing on essentially every run here, and the resulting exception lands in system.part_log, which 02265 inspects → the test flags "exception in stdout".

Suggested next step: figure out which of the two this is:

• A real confirm regression — the CONFIRM request is systematically not returning CA_CONFIRM_ANSWER_PROVEN (endpoint/cookie routing, or the source drops its binding before the confirm arrives). The catch(...) at :1531 logs the underlying reason at information level — pull clickhouse-server.log for ttl_02265_r2 around the merge and check whether the confirm is refused vs unproven vs threw. If the source is releasing the binding too eagerly under back-to-back TTL merges, that's the fix target.
• Expected churn that's merely noisy — if the retry is legitimately transient here, the part still lands on retry and only the logged exception fails the test. In that case the fix is test-side (the CAS-storage variant of 02265 shouldn't treat a retry-class relink exception as fatal).

My read leans toward the first, given how consistently it reproduces (13/14) — a purely transient race shouldn't hit almost every time.

---

2. PR-related — CAS regression (TestFlows) lanes

CAS-specific suites, so these are yours to confirm, but the failure shape differs:

• cas_lightweight_delete_1, cas_s3_cache_lightweight_delete_1, cas_s3_cache_lightweight_delete_2 (aarch64 + release) — fail in 2–3 s with 7 steps (4 ok, 3 failed). A 3-second "fail" that never reaches the actual delete/merge work looks like an early setup/harness failure in the freshly-moved CAS lanes, not a product bug — consistent with your recent ci: stabilize the CAS test lanes and make the moved tests runnable. Worth checking the setup step in the TestFlows log directly (I couldn't extract the assertion cleanly from the HTML report).
• cas_alter_attach_1, cas_s3_cache_alter_attach_1 (release) — these ran the full 4.5 h and had scenario-level failures, so those are real runs; check the per-scenario diffs.

---

3. NOT PR-caused

Iceberg integration tests — the PR touches zero Iceberg code.

• test_storage_iceberg_multistorage::test_num_rows_cache_no_collision_across_buckets (assert 0 >= 1) fails deterministically across 5 lanes including ones that have nothing to do with CAS storage (amd_msan 8/10, amd_tsan 2/6, arm_binary distributed plan, amd_asan_ubsan db disk 8/8, targeted). A CAS change can't explain failures on non-CAS lanes → pre-existing on antalya-26.6 (the base branch's own CI is red on the last 5 runs).
• test_storage_iceberg_with_spark::test_schema_inference[*] — all fail inside create_iceberg_table / SELECT query calls to the Spark-backed cluster = Spark/infra flakiness, not this PR.
• Same for the iceberg_2 regression suites (aarch64 + release).

Non-CAS regression suites — alter_attach_1, alter_replace, settings, tiered_storage_local, ldap_role_mapping, s3_minio_2, s3_export_part. Standard Antalya suites, unrelated to CAS; base branch is red, so treat as pre-existing/flaky unless a specific one blocks you.

Grype Scan (clickhouse-server:...-alpine) — 1 high/critical CVE in the base image dependencies, not in PR code. Pre-existing/infra.

Stress test (arm_tsan) — Unknown error, 1/5. Infra/flaky, safe to re-run.

Stateless one-offs — CI's own randomized-settings rerun clears them (all passed on 2–86 reruns):
01055_compact_parts_granularity, 01938_joins_identifiers, 02967_parallel_replicas_join_algo_and_analyzer_1, 03164_optimize_read_in_order_nullable, 04075_right_any_join_incorrect, 03047_on_fly_mutations_projections. Safe to re-run. (02967 logged Too many active sessions in group Disk, count 9100 before passing on rerun — probably harmless, but a glance won't hurt since "group Disk" is object-storage territory.)

Borderline / low priority — 03572_export_merge_tree_part_limits_and_table_functions (arm_asan_ubsan, targeted): the in-test "wait for exports" hit its 60 s timeout and the whole test blew the 180 s cap — under arm+asan only; it passed on the amd targeted lane. The PR does modify this test, so it's not purely environmental, but the symptom is slowness, not wrong results. If it keeps flaking, consider bumping the in-test export-wait timeout (60 s is tight under asan) or marting the test long.

---

TL;DR: fix 02265_column_ttl (CAS relink confirm in DataPartsExchange.cpp:1543-1552) and check the fast-failing cas_*_lightweight_delete regression setup. The Iceberg wall, the non-CAS regression suites, Grype, the stress error, and the six stateless one-offs are not caused by this PR — they're pre-existing on the red base branch, Spark/infra, or CI-confirmed flakes.

I didn't push anything — this was a triage request. Happy to open a blau/* PR with a fix for 02265 (or the test-side guard) if you tell me which direction you want.

— automated triage; CI reports are data, correctness is confirmed by re-running CI.

[filimonov]

cas_lightweight_delete_1, cas_s3_cache_lightweight_delete_1, cas_s3_cache_lightweight_delete_2

The failures in cas_lightweight_delete_1 and cas_s3_cache_lightweight_delete_1 were caused by a TestFlows harness bug: --only skipped the binary download, while cluster startup still checked the computed local path. This is unrelated to CAS/S3 functionality or artifact availability.

It was fixed in clickhouse-regression@6948a42:
Altinity/clickhouse-regression@6948a42

This explanation does not cover cas_s3_cache_lightweight_delete_2, which uses --skip rather than --only and therefore needs separate triage.

One harness issue also remains: --cas and --cas-s3-cache currently use the same /cas report suffix, so their reports can overwrite each other. They should use distinct suffixes, for example /cas and /cas-s3-cache.

cas_alter_attach_1, cas_s3_cache_alter_attach_1

This is a very high-contention shard. It executes many feature groups concurrently and creates substantial namespace churn in data/cas/ref_catalog.

Most of the later failures are consistent with ref_catalog saturation and S3 timeouts. The tables are cleaned up, so the pressure comes from rapid namespace lifecycle changes and asynchronous catalog reclamation, rather than simply from leaked tables.

However, not every failure should be classified as a scalability issue yet. The replica sanity failure looks like a test race, and an early replicated ATTACH PARTITION FROM data mismatch occurred before the explicit S3 timeouts. That mismatch needs an isolated reproduction.

This requires follow-up but does not have to block the PR.

Additionally (from PR #2073 run).

Server died in MSan CAS 2/3

This is a real shutdown-liveness defect. The CI job timed out and initiated shutdown, but ClickHouse did not terminate within the harness deadline while a GC round was in flight.

The actionable issue is that shutdown can wait indefinitely, or for too long, for an in-flight GC operation. It can be tracked as a known issue for now.

tiered_storage_cas

For MovePart events, system.part_log.path_on_disk is empty where the test expects an external-disk path. This is reproducible on both x86_64 and aarch64.

This is an observability/compatibility gap. We should clarify the expected path_on_disk semantics for CAS and either populate a meaningful logical path or adjust the test if an empty value is intentional. It is not a blocker, but it needs follow-up.