Proposal: a Claude Code skill for migrating from Confluent.Kafka to Dekaf

[zcsizmadia]

@thomhurst

I've been working on making migrations from Confluent.Kafka easier, and I'd like to know whether you'd want this in Dekaf, and if so, how you'd like me to proceed.

The idea

A Claude Code skill that takes an agent through a migration using the existing guide (migrating-from-confluent-kafka.md). The skill holds the procedure and the docs stay the source of facts. It:

• inventories every Kafka client and checks the compatibility gates (Kafka 4.0+, Classic groups, target frameworks, config keys)
• asks the user before any behaviour change (offset semantics, defaults that differ, error handling, startup) instead of guessing
• migrates one file at a time, keeps record bytes and key-to-partition mapping identical, and keeps test harnesses on Confluent.Kafka as an independent check
• ends with a report: decisions, where bytes could differ, and the rollout steps left

It would ship as a plugin from this repo, so users would install it with:

/plugin marketplace add thomhurst/Dekaf
/plugin install dekaf@dekaf

That adds .claude-plugin/marketplace.json and plugins/dekaf/skills/migrate-from-confluent-kafka/SKILL.md (about 200 lines), plus a short "Migrating with Claude Code?" box in the guide.

Validation so far

Each run used a fresh agent with only the skill and the docs, against Kafka 4.1, compared with the original Confluent build:

Project	Result
Confluent examples/Producer	Same partitions, offsets, keys (null vs empty) and values as the Confluent build
Confluent examples/Consumer	Committed offsets carry over from a Confluent group; read-committed, manual assignment and rebalances match
Confluent examples/ExactlyOnce	Exactly-once word counts with map and reduce killed mid-transaction: all 707 counts exact, keys on the same partitions as with Confluent (after a workaround for an issue below)
Eventuous src/Kafka (library)	Existing test plus 5 new contract tests pass: bytes, headers and partition placement checked against Confluent

A way to find problems, too

Running the skill on existing projects turned out to be a good test of Dekaf itself. Each run compares the migrated code against the original Confluent build, so every difference shows up as a gap in the guide, a gap in the skill, or a Dekaf bug. These four runs found two real bugs and several doc errors. Running it on more open-source projects that use Confluent.Kafka would keep sharpening the skill and keep surfacing migration issues and bugs before users hit them.

What it surfaced

Docs (I have fixes ready): partitioning.md says the key hash matches librdkafka (librdkafka defaults to CRC32 consistent_random); the native consumer's ReadUncommitted default versus librdkafka's read_committed; session timeout under KIP-848; Partitions/Positions API names; plus about 20 smaller rows for the migration guide's API map.

Possible Dekaf issues, confirmed unless marked:

1. Committing an empty transaction throws InvalidTxnState (librdkafka and Java complete it locally). This crashes timer-based consume-transform-produce loops. Repro is ten lines.
2. The Confluent config adapter leaves Dekaf's Murmur2 partitioner in place when Partitioner is absent, so keys move partitions after migrating.
3. WithMaxRequestSize, WithMaxPollInterval and WithRebalanceTimeout are internal, so records over 1 MB and long rebalance callbacks need the adapter.
4. SeekToBeginning positions at offset 0, not the log start; with the default Latest reset, a trimmed partition skips to the end. (To reproduce.)
5. The producer always injects traceparent/tracestate headers when tracing listens; there's no opt-out.
6. DI-registered clients initialize at host startup, so an unreachable fallback cluster stops the app starting.
7. Smaller ones: about 20 public builder methods without XML docs; start-kafka.sh runs Kafka 3.5; noisy warning logs during broker restarts.

How would you like to proceed?

1. Are you interested in the skill and plugin living in this repo, or would you prefer it elsewhere?
2. Should I file the issues above separately first, and which ones?
3. For PRs, would docs fixes first, then the skill, then code fixes per issue work for you?

I have a current version of the skill and a detailed list of the issues, each with its cause, source references and a reproduction or how to reproduce it, ready to share or push to my fork. Happy to adjust any of it.

[zcsizmadia]

A follow-up on candidates for further validation runs. If this direction is useful, these are the projects I'd run the skill on next, each compared against its original Confluent build:

With integration tests (they check output with Confluent.Kafka, so a green run proves interoperability):

• Trendyol/kafka-retry-job: consume → produce app with Testcontainers tests
• CoreWCF src/CoreWCF.Kafka: Kafka transport with a large suite (SASL/SSL, dead-letter queue, message properties)
• dotnet/aspire playground/kafka: Aspire Confluent config sections (manual run)

Real-world code without Kafka tests:

• Trendyol/PollingOutboxPublisher: producer-only outbox
• dotnetcore/CAP DotNetCore.CAP.Kafka: library with its own connection pool
• joaofbantunes/DotNetMicroservicesObservabilitySample: consumer with tracing and metrics

Libraries and transports (not yet checked): Topos, Orleans.Streams.Kafka, Akka.Streams.Kafka, Brighter, Wolverine and MassTransit's Kafka rider.

The rest of Confluent's examples: the Schema Registry ones (Avro, Protobuf, JSON Schema), AdminClient, MultiProducer, Web, ExactlyOnceOldBroker, and the OAuth, TLS and encryption examples.

Stop tests: KafkaFlow and Silverback wrap Confluent.Kafka too deeply for a migration; the skill should recognise that and stop.

My suggested order would be kafka-retry-job, the Schema Registry examples, CoreWCF, then CAP. Happy to take requests if there are projects you'd rather see.