Configuring the logging collector
Logging for Red Hat OpenShift collects operations and application logs from your cluster and enriches the data with Kubernetes pod and project metadata. All supported modifications to the log collector can be performed though the spec.collection
stanza in the ClusterLogging
custom resource (CR).
Configuring the log collector
You can configure which log collector type your logging uses by modifying the ClusterLogging
custom resource (CR).
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead. |
Prerequisites
You have administrator permissions.
You have installed the OpenShift CLI (
oc
).You have installed the Red Hat OpenShift Logging Operator.
You have created a
ClusterLogging
CR.
Procedure
Modify the
ClusterLogging
CRcollection
spec:ClusterLogging
CR exampleapiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
collection:
type: <log_collector_type> (1)
resources: {}
tolerations: {}
# ...
1 The log collector type you want to use for the logging. This can be vector
orfluentd
.Apply the
ClusterLogging
CR by running the following command:$ oc apply -f <filename>.yaml
Configuring resources and scheduling for logging collectors
Administrators can modify the resources or scheduling of the collector by creating a ClusterLogging
custom resource (CR) that is in the same namespace and has the same name as the ClusterLogForwarder
CR that it supports.
The applicable stanzas for the ClusterLogging
CR when using multiple log forwarders in a deployment are managmentState
and collection
. All other stanzas are ignored.
Prerequisites
You have administrator permissions.
You have installed the Red Hat OpenShift Logging Operator version 5.8 or newer.
You have created a
ClusterLogForwarder
CR.
Procedure
Create a
ClusterLogging
CR that supports your existingClusterLogForwarder
CR:Example
ClusterLogging
CR YAMLapiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
name: <name> (1)
namespace: <namespace> (2)
spec:
managementState: "Managed"
collection:
type: "vector"
tolerations:
- key: "logging"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 6000
resources:
limits:
memory: 1Gi
requests:
cpu: 100m
memory: 1Gi
nodeSelector:
collector: needed
# ...
1 The name must be the same name as the ClusterLogForwarder
CR.2 The namespace must be the same namespace as the ClusterLogForwarder
CR.Apply the
ClusterLogging
CR by running the following command:$ oc apply -f <filename>.yaml
Viewing logging collector pods
You can view the logging collector pods and the corresponding nodes that they are running on.
Procedure
Run the following command in a project to view the logging collector pods and their details:
$ oc get pods --selector component=collector -o wide -n <project_name>
Example output
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
collector-8d69v 1/1 Running 0 134m 10.130.2.30 master1.example.com <none> <none>
collector-bd225 1/1 Running 0 134m 10.131.1.11 master2.example.com <none> <none>
collector-cvrzs 1/1 Running 0 134m 10.130.0.21 master3.example.com <none> <none>
collector-gpqg2 1/1 Running 0 134m 10.128.2.27 worker1.example.com <none> <none>
collector-l9j7j 1/1 Running 0 134m 10.129.2.31 worker2.example.com <none> <none>
Configure log collector CPU and memory limits
The log collector allows for adjustments to both the CPU and memory limits.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc -n openshift-logging edit ClusterLogging instance
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
name: instance
namespace: openshift-logging
spec:
collection:
type: fluentd
resources:
limits: (1)
memory: 736Mi
requests:
cpu: 100m
memory: 736Mi
# ...
1 Specify the CPU and memory limits and requests as needed. The values shown are the default values.
Advanced configuration for the Fluentd log forwarder
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead. |
The logging for Red Hat OpenShift includes multiple Fluentd parameters that you can use for tuning the performance of the Fluentd log forwarder. With these parameters, you can change the following Fluentd behaviors:
Chunk and chunk buffer sizes
Chunk flushing behavior
Chunk forwarding retry behavior
Fluentd collects log data in a single blob called a chunk. When Fluentd creates a chunk, the chunk is considered to be in the stage, where the chunk gets filled with data. When the chunk is full, Fluentd moves the chunk to the queue, where chunks are held before being flushed, or written out to their destination. Fluentd can fail to flush a chunk for a number of reasons, such as network issues or capacity issues at the destination. If a chunk cannot be flushed, Fluentd retries flushing as configured.
By default in OKD, Fluentd uses the exponential backoff method to retry flushing, where Fluentd doubles the time it waits between attempts to retry flushing again, which helps reduce connection requests to the destination. You can disable exponential backoff and use the periodic retry method instead, which retries flushing the chunks at a specified interval.
These parameters can help you determine the trade-offs between latency and throughput.
To optimize Fluentd for throughput, you could use these parameters to reduce network packet count by configuring larger buffers and queues, delaying flushes, and setting longer times between retries. Be aware that larger buffers require more space on the node file system.
To optimize for low latency, you could use the parameters to send data as soon as possible, avoid the build-up of batches, have shorter queues and buffers, and use more frequent flush and retries.
You can configure the chunking and flushing behavior using the following parameters in the ClusterLogging
custom resource (CR). The parameters are then automatically added to the Fluentd config map for use by Fluentd.
These parameters are:
|
Parameter | Description | Default |
---|---|---|
| The maximum size of each chunk. Fluentd stops writing data to a chunk when it reaches this size. Then, Fluentd sends the chunk to the queue and opens a new chunk. |
|
| The maximum size of the buffer, which is the total size of the stage and the queue. If the buffer size exceeds this value, Fluentd stops adding data to chunks and fails with an error. All data not in chunks is lost. | Approximately 15% of the node disk distributed across all outputs. |
| The interval between chunk flushes. You can use |
|
| The method to perform flushes:
|
|
| The number of threads that perform chunk flushing. Increasing the number of threads improves the flush throughput, which hides network latency. |
|
| The chunking behavior when the queue is full:
|
|
| The maximum time in seconds for the |
|
| The retry method when flushing fails:
|
|
| The maximum time interval to attempt retries before the record is discarded. |
|
| The time in seconds before the next chunk flush. |
|
For more information on the Fluentd chunk lifecycle, see Buffer Plugins in the Fluentd documentation.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc edit ClusterLogging instance
Add or modify any of the following parameters:
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
name: instance
namespace: openshift-logging
spec:
collection:
fluentd:
buffer:
chunkLimitSize: 8m (1)
flushInterval: 5s (2)
flushMode: interval (3)
flushThreadCount: 3 (4)
overflowAction: throw_exception (5)
retryMaxInterval: "300s" (6)
retryType: periodic (7)
retryWait: 1s (8)
totalLimitSize: 32m (9)
# ...
1 Specify the maximum size of each chunk before it is queued for flushing. 2 Specify the interval between chunk flushes. 3 Specify the method to perform chunk flushes: lazy
,interval
, orimmediate
.4 Specify the number of threads to use for chunk flushes. 5 Specify the chunking behavior when the queue is full: throw_exception
,block
, ordrop_oldest_chunk
.6 Specify the maximum interval in seconds for the exponential_backoff
chunk flushing method.7 Specify the retry type when chunk flushing fails: exponential_backoff
orperiodic
.8 Specify the time in seconds before the next chunk flush. 9 Specify the maximum size of the chunk buffer. Verify that the Fluentd pods are redeployed:
$ oc get pods -l component=collector -n openshift-logging
Check that the new values are in the
fluentd
config map:$ oc extract configmap/collector-config --confirm
Example fluentd.conf
<buffer>
@type file
path '/var/lib/fluentd/default'
flush_mode interval
flush_interval 5s
flush_thread_count 3
retry_type periodic
retry_wait 1s
retry_max_interval 300s
retry_timeout 60m
queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
chunk_limit_size 8m
overflow_action throw_exception
disable_chunk_backup true
</buffer>