How should producer creation handle backlog-quota rejection?

[jaswanthnaidumalasanimalasani-code]

We publish through an HTTP endpoint that creates producers on demand, one per topic, and caches them with a 30s TTL.

The relevant setup is:

• One PulsarClient, shared across all topics and callers.
• Producers are cached per topic and evicted 30s after the last write/access. The cache is limited to 1000 entries.
• Eviction flushes and closes the producer.
• Producers use Shared access mode, batching and compression are disabled, and maxPendingMessages(500) with blockIfQueueFull(false).
• If a topic is not in the cache, the request creates the producer inline and then publishes.

The 30s TTL means an intermittently used topic can leave the cache and trigger producer creation again on a later request.

When a topic reaches its backlog quota with producer_exception, the broker refuses new producers. That behavior is expected.

What I am trying to understand is how the caller is expected to handle this refusal. The client appears to treat the quota exception as retriable and continues producer creation instead of surfacing the exception immediately.

The behavior below is not specific to an old release. I checked master and the current release branches, and the relevant code is the same on all of them.

Observed behavior

With:

operationTimeout       = 30s
initialBackoffInterval = 100ms
maxBackoffInterval     = 60s
sendTimeout            = 30s

the retry sequence is approximately:

Attempt	Start	Backoff	Elapsed
1	0.0s	100ms	0.1s
5	1.5s	1.6s	3.1s
7	6.3s	6.4s	12.7s
8	12.7s	12.8s	25.5s
9	25.5s	4.4s*	29.9s
10	29.9s	51.2s	81.1s
11	81.1s	—	creation fails

• truncated by mandatoryStop.

Attempt 10 is admitted just before the 30s deadline, but the subsequent backoff is ~51s. As a result, producer creation fails after ~81s even though operationTimeout is 30s.

The exact timings vary slightly because of backoff jitter.

What I found in the client

isRetriableError appears to be a denylist. Neither of these exceptions is explicitly excluded:

• ProducerBlockedQuotaExceededException
• ProducerBlockedQuotaExceededError

ProducerImpl has a specific branch for the quota case: Topic backlog quota exceeded. Throwing Exception on producer.
and calls failPendingMessages, but it does not complete producerCreatedFuture exceptionally.

The TopicTerminatedException and ProducerFencedException cases immediately below it do:

producerCreatedFuture.completeExceptionally(cause);

The quota case instead reaches the generic retriable path and schedules another reconnect.

Also, operationTimeout appears to be checked when deciding whether another attempt can start. Once an attempt has been admitted, the subsequent backoff is scheduled without applying the remaining deadline to that delay. This is what allows the attempt at ~29.9s to result in another ~51s wait.

mandatoryStop only truncates the backoff crossing the stop point; subsequent backoff continues normally.

What we tried

We originally created producers synchronously, so HTTP request threads could stay blocked in ProducerBuilderImpl.create() for the duration of the retry sequence. We have tried two ways to avoid that, and neither feels like the intended answer.

1. Create asynchronously, with our own deadline. We switched to createAsync() and wrapped it in orTimeout(...) so the request thread is never held. This solves the blocking, but not the underlying behavior: creation for a quota-exhausted topic stays pending until our deadline fires, and requests waiting on it accumulate along with their payloads. The request-thread pool had been an implicit bound on how many could be outstanding, and removing the blocking removed that bound too.

2. Keep creation synchronous, but reduce operationTimeout and maxBackoffInterval. With a lower backoff ceiling, every sleep is short enough that the typed quota exception surfaces within our timeout, so we can recognise it and stop attempting that topic for a while. This works. But it only works because one value happens to fit inside the other, across two settings that are otherwise unrelated, on a PulsarClient shared by every topic we publish to. Nothing enforces that relationship, and changing either value silently changes whether the exception is observable.

Questions

1. Is it intentional that ProducerBlockedQuotaExceededException / ProducerBlockedQuotaExceededError are treated as retriable?

   In particular, is this because the same retry path is also used for producer_request_hold, where waiting for the quota to clear is expected?

2. Is there a supported way to bound the total producer creation time?

   operationTimeout appears to control retry admission rather than the complete createAsync() operation. Is wrapping it in an application-level deadline, as in approach 1, the expected way to do this, or is there a client setting for it that I have missed?

3. Is reducing maxBackoffInterval the expected way to make the quota exception observable within a desired timeout?

4. For an HTTP producer service, is sharing one PulsarClient across topics/tenants the recommended approach, or is there a recommended client scope for isolating this kind of retry behavior?

5. Is there a recommended way to determine that a topic is already at its backlog quota before attempting producer creation?

We build the client directly using PulsarClient.builder() rather than through the WebSocket proxy.

Related issues I found are #26373 and #26343, although I don't see either covering this producer-creation/retriability behavior.

I can provide the relevant thread dumps or code paths if useful.