Does tiered storage offloading change message IDs?

[jeloba]

See the title.

We have issues seeking to message ids, that we stored in an index. We found that this may be caused by offloading into tiered storage.

Is this true? Is offloading creating new ledgers with new message ids?

[lhotari]

We have issues seeking to message ids, that we stored in an index. We found that this may be caused by offloading into tiered storage.

Is this true? Is offloading creating new ledgers with new message ids?

Message IDs don't change for offloaded ledgers. What problem are you experiencing? What Pulsar version do you have and what type of deployment?

[haithir]

@lhotari
We found the following errors in the Pulsar logs after trying to seek to a stored message ID and read from there:

2024-11-22T10:48:14,500+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L53035749 E0-E0, Sent to [pulsar-bookie-6.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181], Heard from [] : bitset = {}, Error = 'No such ledger exists on Bookies'. First unread entry is (-1, rc = null) 2024-11-22T10:48:14,500+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [********/check-archive/persistent/metadata][reader-b24c841fba] read failed from ledger at position:53035749:0 org.apache.bookkeeper.mledger.ManagedLedgerException$LedgerNotExistException: No such ledger exists on Bookies

The data there exists. After throwing away the stored message ID, we could read it again.

We have S3 offloading enabled and thought it may be related to this and maybe happens if the ledger is removed for the offloaded data.

Is this an error or are there expected scenarios where message IDs become invalid?

[lhotari]

Is this an error or are there expected scenarios where message IDs become invalid?

@haithir

There have been some rare bugs in Pulsar and BookKeeper in the past where invididual entries get corrupted already in writing and become unreadable. That's why running on a recent supported and maintained version is important for stability.

Another reason could be a case where bookies fail and the cluster doesn't have sufficient replication level to recover. That's why running with at least 2 replicates (E=2,Qw=2,Qa=2) is recommended and paying attention to the bookie autorecovery state.

It always possible that there are also unresolved bugs that lead to transient errors where reading an entry fails. Which Pulsar version do you use?

[haithir]

We still use 3.1.1 and monitoring the logs, I found now that the "missing ledger" log is not too seldom. We use the default write quorum of 2. It looks also that the ledgers self-repair after some time. I think we need to upgrade as you recommend. Thank you so far