Skip to content

docs: pipeline: outputs: kafka: document Protobuf serialization - #2717

Merged
eschabell merged 2 commits into
fluent:masterfrom
eschabell:erics_out_kafka_updates
Sep 24, 2026
Merged

eschabell merged 2 commits into
fluent:masterfrom
eschabell:erics_out_kafka_updates

Conversation

@eschabell

@eschabell eschabell commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator
  • Add protobuf to the format row's list of available formats.
  • Add the protobuf_message row to the Configuration parameters
    table, between otlp_logs_partition_by_resource and
    queue_full_retries.
  • Rescope schema_id and schema_registry_subject from Avro-only to
    Avro or Protobuf.
  • Rewrite the schema_str row: it is the inline Avro alternative to a
    registry and is not supported with format: protobuf.
  • Add a Protobuf support section covering the
    -DFLB_PROTOBUF_ENCODER=On build option and its Protobuf 3.12
    requirement, the mandatory schema_registry_url, recursive
    references resolution and its 64-file / 4 MiB limits,
    protobuf_message selection rules, the JSON-then-encode record
    path, Confluent wire framing, and YAML and classic examples.
  • Promote "Resolve schemas from a registry" from h4 to h3 and rewrite
    its intro: the registry is enabled by either FLB_AVRO_ENCODER or
    FLB_PROTOBUF_ENCODER, not by Avro support alone.
  • Promote "Kafka configuration file with raw format" to h3 so it stays
    a sibling rather than nesting under the registry section, and add
    the missing space in rawformat.
  • Fix the MSK build requirements table delimiter row, which tripped
    markdownlint MD060 four times.

Signed-off-by: Eric D. Schabell eric@schabell.org

Summary by CodeRabbit

  • Documentation
    • Documented Protobuf encoding support for the Kafka output plugin, including build requirements and the protobuf_message option.
    • Clarified how schema settings apply to Protobuf, including that schema_str is unsupported with this format.
    • Added details on schema registry resolution, referenced-schema limits, message selection, and startup failure messages.
    • Updated Avro and Protobuf availability guidance, and refined section organization and formatting.

  - Add `protobuf` to the `format` row's list of available formats.
  - Add the `protobuf_message` row to the Configuration parameters
    table, between `otlp_logs_partition_by_resource` and
    `queue_full_retries`.
  - Rescope `schema_id` and `schema_registry_subject` from Avro-only to
    Avro or Protobuf.
  - Rewrite the `schema_str` row: it is the inline Avro alternative to a
    registry and is not supported with `format: protobuf`.
  - Add a Protobuf support section covering the
    `-DFLB_PROTOBUF_ENCODER=On` build option and its Protobuf 3.12
    requirement, the mandatory `schema_registry_url`, recursive
    `references` resolution and its 64-file / 4 MiB limits,
    `protobuf_message` selection rules, the JSON-then-encode record
    path, Confluent wire framing, and YAML and classic examples.
  - Promote "Resolve schemas from a registry" from h4 to h3 and rewrite
    its intro: the registry is enabled by either FLB_AVRO_ENCODER or
    FLB_PROTOBUF_ENCODER, not by Avro support alone.
  - Promote "Kafka configuration file with raw format" to h3 so it stays
    a sibling rather than nesting under the registry section, and add
    the missing space in `raw`format.
  - Fix the MSK build requirements table delimiter row, which tripped
    markdownlint MD060 four times.

Signed-off-by: Eric D. Schabell <eric@schabell.org>
@eschabell
eschabell requested review from a team and patrick-stephens as code owners September 20, 2026 14:58
@coderabbitai

coderabbitai Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 3fc6b74b-66e9-4408-a639-f58033518e9c

📥 Commits

Reviewing files that changed from the base of the PR and between 8cfc72b and 344ab00.

📒 Files selected for processing (1)
  • pipeline/outputs/kafka.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • pipeline/outputs/kafka.md

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


📝 Walkthrough

Walkthrough

The Kafka output documentation adds Protobuf format and message configuration, build requirements, schema registry behavior, encoding details, and examples. It also updates Avro support descriptions and adjusts section headings and table formatting.

Changes

Kafka Protobuf documentation

Layer / File(s) Summary
Protobuf configuration and encoding
pipeline/outputs/kafka.md
The documentation adds the protobuf format, protobuf_message, build requirements, schema graph limits, message selection rules, Confluent framing, and configuration examples.
Registry resolution and document structure
pipeline/outputs/kafka.md
The documentation updates Avro and Protobuf registry resolution and support descriptions. It also promotes headings, corrects the raw-format heading text, and reformats the platform requirements table separator.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: braydonk

Merge Risk: ⚪ Minimal · up to 344ab

No concrete issue was identified that would block merging the Kafka Protobuf documentation.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: documenting Protobuf serialization for the Kafka output.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@eschabell eschabell self-assigned this Sep 20, 2026
@eschabell eschabell added waiting-on-review Waiting on a review from mainteners 5.1.3 labels Sep 20, 2026
@eschabell

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@eschabell

Copy link
Copy Markdown
Collaborator Author

@patrick-stephens ready for review!

Comment thread pipeline/outputs/kafka.md
…elease builds

  - Avro support: state it isn't in any official release packages or
    container images and needs a source build (was "isn't enabled by
    default for releases")
  - Protobuf support: same rewording
  - Registry schema resolution: "neither of which is enabled by default"
    becomes "neither of which is included in official release builds"

Signed-off-by: Eric D. Schabell <eric@schabell.org>
@eschabell
eschabell merged commit ca2da90 into fluent:master Sep 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5.1.3 waiting-on-review Waiting on a review from mainteners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants