Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The following is an example of a `SERVICE` section:
Log_Level debug
```

For scheduler and retry details, see [scheduling and retries](../../scheduling-and-retries.md#Scheduling-and-Retries).
For scheduler and retry details, see [scheduling and retries](../../scheduling-and-retries.md).

## Config input

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ In Fluent Bit versions 5.1.0 and 5.1.1, setting `FLB_LOG_LEVEL` and `log_level`

FIPS mode is available in Fluent Bit version 5.1 and greater. It requires OpenSSL 3.0 or greater with the FIPS provider installed on the host.

When `security.fips_mode` is enabled, Fluent Bit activates the OpenSSL FIPS provider during startup and verifies that a FIPS-approved algorithm can be fetched from it. If the provider isn't available or can't be activated, Fluent Bit logs the OpenSSL errors and exits instead of starting with non-compliant cryptography. You can set the same behavior from the command line with the [`--enable-fips`](../../configuring-fluent-bit.md#require-fips-mode-with---enable-fips) flag.
When `security.fips_mode` is enabled, Fluent Bit activates the OpenSSL FIPS provider during startup and verifies that a FIPS-approved algorithm can be fetched from it. If the provider isn't available or can't be activated, Fluent Bit logs the OpenSSL errors and exits instead of starting with non-compliant cryptography. You can set the same behavior from the command line with the [`--enable-fips`](../../configuring-fluent-bit.md#require-fips-mode-with-enable-fips) flag.

FIPS mode can't be changed by [hot reload](../../../administration/hot-reload.md). If a reloaded configuration changes `security.fips_mode`, Fluent Bit halts the reload and keeps running with the previous configuration.

Expand Down Expand Up @@ -176,7 +176,7 @@ The following storage-related keys can be set as children to the `storage` key:

For storage and buffering details, see [Buffering](../../../pipeline/buffering.md) and [Backpressure](../../backpressure.md).

For scheduler and retry details, see [Scheduling and retries](../../scheduling-and-retries.md#Scheduling-and-Retries).
For scheduler and retry details, see [Scheduling and retries](../../scheduling-and-retries.md).

## Configuration example

Expand Down
2 changes: 1 addition & 1 deletion installation/upgrade-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ If you have existing queries based on the resource's `project_id,` update your q

The migration from v1.4 to v1.5 is pretty straightforward.

- The `keepalive` configuration mode has been renamed to `net.keepalive`. Now, all Network I/O keepalive is enabled by default. To learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. - If you use the Elasticsearch output plugin, the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch tolerate this, but Elasticsearch v5.6 through v6.1 require a `type` without a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#faq-underscore) for more.
- The `keepalive` configuration mode has been renamed to `net.keepalive`. Now, all Network I/O keepalive is enabled by default. To learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#connection-keepalive) section. - If you use the Elasticsearch output plugin, the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch tolerate this, but Elasticsearch v5.6 through v6.1 require a `type` without a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#mapping-type-names-cant-start-with-underscores) for more.

## Fluent Bit v1.4

Expand Down
2 changes: 1 addition & 1 deletion pipeline/filters/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ The plugin supports the following configuration parameters:
| `tls.verify_hostname` | When enabled, turns on hostname validation for certificates. | `Off` |
| `tls.vhost` | Set an optional TLS virtual host for the Kubernetes API server connection. | _none_ |
| `use_journal` | When enabled, the filter reads logs in `Journald` format. | `Off` |
| `use_kubelet` | Optional feature flag to get metadata information from Kubelet instead of calling Kube Server API to enhance the log. This could mitigate the [Kube API heavy traffic issue for large cluster](kubernetes.md#optional-feature-using-kubelet-to-get-metadata). If used when any [Kubernetes Namespace Meta](#kubernetes-namespace-meta) fields are enabled, Kubelet will be used to fetch pod data, but namespace meta will still be fetched using the `kube_url` settings. In v5.1.3 or later, the pod name and namespace are matched exactly against the Kubelet pod list instead of by prefix. | `Off` |
| `use_kubelet` | Optional feature flag to get metadata information from Kubelet instead of calling Kube Server API to enhance the log. This could mitigate the [Kube API heavy traffic issue for large cluster](kubernetes.md#using-kubelet-to-get-metadata). If used when any [Kubernetes Namespace Meta](#kubernetes-namespace-meta) fields are enabled, Kubelet will be used to fetch pod data, but namespace meta will still be fetched using the `kube_url` settings. In v5.1.3 or later, the pod name and namespace are matched exactly against the Kubelet pod list instead of by prefix. | `Off` |
| `use_pod_association` | Deprecated alias for `aws_use_pod_association`. Kept for backward compatibility with AWS Observability users. | `Off` |
| `use_tag_for_meta` | When enabled, Kubernetes metadata (for example, `pod_name`, `container_name`, and `namespace_name`) will be extracted from the tag itself. Connection to Kubernetes API Server won't get established and API calls for metadata won't be made. See [Workflow of Tail and Kubernetes Filter](#workflow-of-tail-and-kubernetes-filter) and [Custom tag For enhanced filtering](#custom-tags-for-enhanced-filtering) to better understand metadata extraction from tags. | `Off` |

Expand Down
2 changes: 1 addition & 1 deletion pipeline/inputs/standard-input.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The plugin supports the following configuration parameters:
If no parser is configured for the `stdin` plugin, it expects valid JSON input data in one of the following formats:

- A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }`
- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which can be:
- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#events-or-records) format, which can be:
- `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds.
- From Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where _`TIMESTAMP`_ has the same meaning as previous and _`METADATA`_ is a JSON object.

Expand Down
2 changes: 1 addition & 1 deletion pipeline/inputs/tail.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The plugin supports the following configuration parameters:
| `skip_empty_lines` | Skips empty lines in the log file from any further processing or output. | `false` |
| `skip_long_lines` | When a monitored file reaches its buffer capacity due to a very long line (`buffer_max_size`), the default behavior is to stop monitoring that file. `skip_long_lines` alters that behavior and instructs Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. | `false` |
| `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` |
| `tag` | Set a tag with `regexextract` fields that will be placed on lines read. For example, `kube.<namespace_name>.<pod_name>.<container_name>.<container_id>`. Tag expansion is supported: if the tag includes an asterisk (`*`), that asterisk will be replaced with the absolute path of the monitored file, with slashes replaced by dots. See [Workflow of Tail + Kubernetes Filter](../filters/kubernetes.md#workflow-of-tail--kubernetes-filter). | _none_ |
| `tag` | Set a tag with `regexextract` fields that will be placed on lines read. For example, `kube.<namespace_name>.<pod_name>.<container_name>.<container_id>`. Tag expansion is supported: if the tag includes an asterisk (`*`), that asterisk will be replaced with the absolute path of the monitored file, with slashes replaced by dots. See [Workflow of Tail + Kubernetes Filter](../filters/kubernetes.md#workflow-of-tail-and-kubernetes-filter). | _none_ |
| `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64})\.log$`. | _none_ |
| `thread.ring_buffer.capacity` | Number of slots in the ring buffer for data entries when running in [threaded](../../administration/multithreading.md) mode. Each slot can hold one data entry. | `1024` |
| `thread.ring_buffer.window` | Percentage threshold (1-100) of the ring buffer capacity at which a flush is triggered when running in [threaded](../../administration/multithreading.md) mode. | `5` |
Expand Down
4 changes: 2 additions & 2 deletions pipeline/outputs/opensearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ This plugin supports the following parameters:
| `aws_profile` | AWS profile name. | `default` |
| `aws_region` | Specify the AWS region for Amazon OpenSearch Service. | _none_ |
| `aws_role_arn` | AWS IAM Role to assume to put records to your Amazon cluster. | _none_ |
| `aws_service_name` | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | `es` |
| `aws_service_name` | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](#fluent-bit-and-amazon-opensearch-serverless) section on Amazon OpenSearch Serverless for more information. | `es` |
| `aws_sts_endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service. | _none_ |
| `buffer_size` | Specify the buffer size used to read the response from the OpenSearch HTTP service. Use for debugging purposes where it's required to read full responses. The response size grows depending of the number of records inserted. Set this value to `False` to set an unlimited amount of memory. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `512k` |
| `compress` | Set payload compression mechanism. Allowed value: `gzip`. Enabling compression reduces network bandwidth usage but might increase CPU usage. | _none_ |
Expand Down Expand Up @@ -54,7 +54,7 @@ This plugin supports the following parameters:
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
| `write_operation` | Operation to use to write in bulk requests. | `create` |

The parameters `index` and `type` can be confusing if you are new to OpenSearch. If you have used a common relational database before, they can be compared to the `database` and `table` concepts. Also see [the FAQ](opensearch.md#faq).
The parameters `index` and `type` can be confusing if you are new to OpenSearch. If you have used a common relational database before, they can be compared to the `database` and `table` concepts. Also see [the FAQ](#frequently-asked-questions).

{% hint style="info" %}

Expand Down
4 changes: 2 additions & 2 deletions pipeline/outputs/stackdriver.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
| `metadata_server` | Prefix for a metadata server. | Value of environment variable `$METADATA_SERVER`, or `http://metadata.google.internal` if unset. |
| `namespace` | A namespace identifier, such as a cluster name or environment. If the resource type is either `generic_node` or `generic_task`, this field is required. | _none_ |
| `node_id` | A unique identifier for the node within the namespace, such as a hostname or IP address. If the resource type is `generic_node`, this field is required. | _none_ |
| `project_id_key` | The value of this field is used by the Stackdriver output plugin to find the GCP `projectID` from `jsonPayload` and then extract the value of it to set the `PROJECT_ID` within LogEntry `logName`, which controls the GCP project that should receive these logs. | `logging.googleapis.com/projectId`. See [Stackdriver Special Fields](https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields) for more info. |
| `project_id_key` | The value of this field is used by the Stackdriver output plugin to find the GCP `projectID` from `jsonPayload` and then extract the value of it to set the `PROJECT_ID` within LogEntry `logName`, which controls the GCP project that should receive these logs. | `logging.googleapis.com/projectId`. See [Stackdriver Special Fields](https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#logentry-fields) for more info. |

Check warning on line 30 in pipeline/outputs/stackdriver.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.SentenceLengthLong] Improve readability by using fewer than 35 words in this sentence. Raw Output: {"message":"[FluentBit.SentenceLengthLong] Improve readability by using fewer than 35 words in this sentence.","location":{"path":"pipeline/outputs/stackdriver.md","range":{"start":{"line":30,"column":22},"end":{"line":30,"column":25}}},"severity":"INFO","code":{"value":"FluentBit.SentenceLengthLong"}}
| `resource` | Set resource type of data. Supported resource types: `k8s_container`, `k8s_node`, `k8s_pod`, `global`, `generic_node`, `generic_task`, and `gce_instance`. | `global` |
| `resource_labels` | An optional list of comma-separated strings specifying resource label plain text assignments (`new=value`) and mappings from an original field in the log entry to a destination field (`destination=$original`). Nested fields and environment variables are also supported using the [record accessor syntax](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/record-accessor). If configured, all resource labels will be assigned using this API only, with the exception of `project_id`. See [Resource Labels](#resource-labels) for more details. | _none_ |
| `service_account_email` | Account email associated with the service. Only available if no credentials file has been provided. | Value of environment variable `$SERVICE_ACCOUNT_EMAIL` |
Expand Down Expand Up @@ -261,7 +261,7 @@
/projects/<project ID>/logs/<log tag>
```

You might be in a scenario where being more specific about the log name is important (for example, [integration with Log Router rules](https://docs.cloud.google.com/logging/docs/routing/overview) or [controlling cardinality of log based metrics](https://cloud.google.com/logging/docs/logs-based-metrics/troubleshooting#too-many-time-series)). You can control the log name directly on a per-log basis by using the [`logging.googleapis.com/logName` special field](https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields). You can configure a `log_name_key` if you'd like to use something different from `logging.googleapis.com/logName`. For example, if the `log_name_key` is set to `mylognamefield` will extract the log name from `mylognamefield` in the log.
You might be in a scenario where being more specific about the log name is important (for example, [integration with Log Router rules](https://docs.cloud.google.com/logging/docs/routing/overview) or [controlling cardinality of log based metrics](https://cloud.google.com/logging/docs/logs-based-metrics/troubleshooting#too-many-time-series)). You can control the log name directly on a per-log basis by using the [`logging.googleapis.com/logName` special field](https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#logentry-fields). You can configure a `log_name_key` if you'd like to use something different from `logging.googleapis.com/logName`. For example, if the `log_name_key` is set to `mylognamefield` will extract the log name from `mylognamefield` in the log.

## Troubleshooting

Expand Down
2 changes: 1 addition & 1 deletion pipeline/parsers/regular-expression.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Use the _regular expression_ parser format to create custom parsers with Ruby regular expressions. These regular expressions use named capture to define which content belongs to which key name.

Use [Tail multiline](../inputs/tail.md#multiline) when you need to support regular expressions across multiple lines from a `tail`. The Tail input plugin treats each line as a separate entity.
Use [Tail multiline](../inputs/tail.md#multiline-support) when you need to support regular expressions across multiple lines from a `tail`. The Tail input plugin treats each line as a separate entity.

{% hint style="warning" %}

Expand Down
Loading