From d44326812698f9fdece2a9323826cf2104b72f0c Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Sat, 26 Sep 2026 15:45:35 +0200 Subject: [PATCH] docs: pipeline: outputs: gcs: gate total_file_size and net.* to v5.1.3 Neither feature documented by PR #2716 is present in v5.1.2 or any earlier tag. Both ship in v5.1.3. - Annotate the total_file_size row and scope the two Buffering paragraphs that describe the size based upload trigger. - Scope the net.* bullet under Networking and TLS configuration. flb_upstream_create is present in v5.1.2, but flb_output_upstream_set is not, so net.* properties set on a GCS output had no effect in released builds. The GCS plugin itself shipped in v5.1 and is unaffected, as is the tls.* bullet. Refs #2727 Signed-off-by: Eric D. Schabell --- pipeline/outputs/gcs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/outputs/gcs.md b/pipeline/outputs/gcs.md index 875d51508..bc9711410 100644 --- a/pipeline/outputs/gcs.md +++ b/pipeline/outputs/gcs.md @@ -134,7 +134,7 @@ pipeline: | `store_dir` | Directory used to locally buffer data before uploading it. | `/tmp/fluent-bit/gcs` | | `store_dir_limit_size` | Limits the amount of data buffered in `store_dir` to limit disk usage. When the limit is reached, data is discarded. Set to `0` for unlimited. | `0` | | `subject_token_type` | `OIDC` subject token type presented to the Google Security Token Service. Only used when `enable_identity_federation` is `true`. See [Workload Identity Federation](#workload-identity-federation). | `urn:ietf:params:oauth:token-type:jwt` | -| `total_file_size` | Maximum size of the file currently being buffered for a tag. When that file reaches this size, Fluent Bit uploads it as an object, even when `upload_timeout` hasn't elapsed. Files already sealed and waiting to upload don't count toward this size, so a tag can hold more than this amount on disk. The minimum accepted value is `1M`. Set to `0` to upload only when `upload_timeout` elapses. See [Buffering](#buffering). | `100M` | +| `total_file_size` | Maximum size of the file currently being buffered for a tag. When that file reaches this size, Fluent Bit uploads it as an object, even when `upload_timeout` hasn't elapsed. Files already sealed and waiting to upload don't count toward this size, so a tag can hold more than this amount on disk. The minimum accepted value is `1M`. Set to `0` to upload only when `upload_timeout` elapses. See [Buffering](#buffering). Supported in v5.1.3 or later. | `100M` | | `unify_tag` | Whether to buffer records from every tag into a single file instead of one file per tag. See [Unified tag buffering](#unified-tag-buffering). | `false` | | `unify_tag_name` | Logical tag that replaces the record tag when `unify_tag` is enabled. It's stored as the buffer chunk metadata and used for `$TAG` in `gcs_key_format`. It doesn't set the name of the local buffer file. See [Unified tag buffering](#unified-tag-buffering). | `fluent-bit-buffer-file-unify-tag.log` | | `upload_timeout` | When this amount of time elapses, Fluent Bit uploads the buffered data and starts a new object. Set to `60m` to upload a new object every hour. | `10m` | @@ -188,9 +188,9 @@ pipeline: ## Buffering -This plugin buffers records as files in `store_dir` and uploads them when either `upload_timeout` elapses or the file currently being buffered for a tag reaches `total_file_size`, so it requires a writeable filesystem. Because the plugin has its own buffering system, the `storage.total_limit_size` parameter isn't meaningful. Use `store_dir_limit_size` and `store_chunk_limit` to limit disk usage instead. +This plugin buffers records as files in `store_dir` and uploads them when `upload_timeout` elapses, so it requires a writeable filesystem. In Fluent Bit version 5.1.3 and greater, the plugin also uploads the file currently being buffered for a tag when it reaches `total_file_size`. Because the plugin has its own buffering system, the `storage.total_limit_size` parameter isn't meaningful. Use `store_dir_limit_size` and `store_chunk_limit` to limit disk usage instead. -When adding a batch of records would push a buffered file past `total_file_size`, Fluent Bit seals that file, schedules it for immediate upload, and starts a new file for the incoming records. A sealed file never receives more data, even if its upload fails and is retried later. Because the check runs against whole batches, an uploaded object can be slightly smaller or larger than `total_file_size`. Set `total_file_size` to `0` to disable the size trigger and upload only on `upload_timeout`. A non-zero value smaller than `1M` fails at startup with `'total_file_size' must be at least 1M (or 0 to disable the size trigger)`. +The `total_file_size` size trigger is available in Fluent Bit version 5.1.3 and greater. When adding a batch of records would push a buffered file past `total_file_size`, Fluent Bit seals that file, schedules it for immediate upload, and starts a new file for the incoming records. A sealed file never receives more data, even if its upload fails and is retried later. Because the check runs against whole batches, an uploaded object can be slightly smaller or larger than `total_file_size`. Set `total_file_size` to `0` to disable the size trigger and upload only on `upload_timeout`. A non-zero value smaller than `1M` fails at startup with `'total_file_size' must be at least 1M (or 0 to disable the size trigger)`. ### Unified tag buffering @@ -202,7 +202,7 @@ When `unify_tag` is enabled, the value of `unify_tag_name` replaces the record t This plugin inherits the core Fluent Bit networking and TLS settings. The upload connection to `storage.googleapis.com` always uses TLS on port 443, so you don't need to turn TLS on. Use these settings to tune how that connection behaves: -- [Networking Setup](../../administration/networking.md): `net.*` properties such as connection timeouts, keepalive behavior, and the source address. Fluent Bit applies these to the GCS upload connection. They don't apply to the separate connections used to obtain credentials from the Google Security Token Service, IAM Credentials, or metadata server endpoints. +- [Networking Setup](../../administration/networking.md): `net.*` properties such as connection timeouts, keepalive behavior, and the source address. Fluent Bit version 5.1.3 and greater applies these to the GCS upload connection. They don't apply to the separate connections used to obtain credentials from the Google Security Token Service, IAM Credentials, or metadata server endpoints. - [Security and TLS](../../administration/transport-security.md): `tls.*` properties such as `tls.ca_file`, `tls.verify`, and `tls.vhost`, which control certificate validation for the upload connection. Set only the properties your environment requires. Most of them take a value or a path rather than an on or off flag, and the defaults are suitable for connecting to Cloud Storage.