How-To: Set up Fluentd, Elastic search and Kibana in Kubernetes

How to install Fluentd, Elastic Search, and Kibana to search logs in Kubernetes

Prerequisites

Install Elastic search and Kibana

  1. Create namespace for monitoring tool and add Helm repo for Elastic Search

    1. kubectl create namespace dapr-monitoring
  2. Add Elastic helm repo

    1. helm repo add elastic https://helm.elastic.co
    2. helm repo update
  3. Install Elastic Search using Helm

By default the chart creates 3 replicas which must be on different nodes. If your cluster has less than 3 nodes, specify a lower number of replicas. For example, this sets it to 1:

  1. helm install elasticsearch elastic/elasticsearch -n dapr-monitoring --set replicas=1

Otherwise:

  1. helm install elasticsearch elastic/elasticsearch -n dapr-monitoring

If you are using minikube or want to disable persistent volumes for development purposes, you can disable it by using the following command:

  1. helm install elasticsearch elastic/elasticsearch -n dapr-monitoring --set persistence.enabled=false,replicas=1
  1. Install Kibana

    1. helm install kibana elastic/kibana -n dapr-monitoring
  2. Validation

    Ensure Elastic Search and Kibana are running in your Kubernetes cluster.

    1. kubectl get pods -n dapr-monitoring
    2. NAME READY STATUS RESTARTS AGE
    3. elasticsearch-master-0 1/1 Running 0 6m58s
    4. kibana-kibana-95bc54b89-zqdrk 1/1 Running 0 4m21s

Install Fluentd

  1. Install config map and Fluentd as a daemonset

Download these config files:

Note: If you already have Fluentd running in your cluster, please enable the nested json parser to parse JSON formatted log from Dapr.

Apply the configurations to your cluster:

  1. kubectl apply -f ./fluentd-config-map.yaml
  2. kubectl apply -f ./fluentd-dapr-with-rbac.yaml
  1. Ensure that Fluentd is running as a daemonset; the number of instances should be the same as the number of cluster nodes. In the example below we only have 1 node.
  1. kubectl get pods -n kube-system -w
  2. NAME READY STATUS RESTARTS AGE
  3. coredns-6955765f44-cxjxk 1/1 Running 0 4m41s
  4. coredns-6955765f44-jlskv 1/1 Running 0 4m41s
  5. etcd-m01 1/1 Running 0 4m48s
  6. fluentd-sdrld 1/1 Running 0 14s

Install Dapr with JSON formatted logs

  1. Install Dapr with enabling JSON-formatted logs

    1. helm repo add dapr https://dapr.github.io/helm-charts/
    2. helm repo update
    3. helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
  2. Enable JSON formatted log in Dapr sidecar

Add dapr.io/log-as-json: "true" annotation to your deployment yaml.

Example:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: pythonapp
  5. namespace: default
  6. labels:
  7. app: python
  8. spec:
  9. replicas: 1
  10. selector:
  11. matchLabels:
  12. app: python
  13. template:
  14. metadata:
  15. labels:
  16. app: python
  17. annotations:
  18. dapr.io/enabled: "true"
  19. dapr.io/app-id: "pythonapp"
  20. dapr.io/log-as-json: "true"
  21. ...

Search logs

Note: Elastic Search takes a time to index the logs that Fluentd sends.

  1. Port-forward to svc/kibana-kibana
  1. $ kubectl port-forward svc/kibana-kibana 5601 -n dapr-monitoring
  2. Forwarding from 127.0.0.1:5601 -> 5601
  3. Forwarding from [::1]:5601 -> 5601
  4. Handling connection for 5601
  5. Handling connection for 5601
  1. Browse http://localhost:5601

  2. Click Management -> Index Management

kibana management

  1. Wait until dapr-* is indexed.

index log

  1. Once dapr-* indexed, click Kibana->Index Patterns and Create Index Pattern

create index pattern

  1. Define index pattern - type dapr* in index pattern

define index pattern

  1. Select time stamp filed: @timestamp

timestamp

  1. Confirm that scope, type, app_id, level, etc are being indexed.

Note: if you cannot find the indexed field, please wait. it depends on the volume of data and resource size where elastic search is running.

indexing

  1. Click discover icon and search scope:*

Note: it would take some time to make log searchable based on the data volume and resource.

discover

References

Last modified August 2, 2021 : Fix Java SDK link (#1695) (2c67fd1)