Viewing the status of the log store

You can view the status of the OpenShift Elasticsearch Operator and for a number of Elasticsearch components.

Viewing the status of the log store

You can view the status of your log store.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Change to the openshift-logging project.

    1. $ oc project openshift-logging

  2. To view the status:

    1. Get the name of the log store instance:

      1. $ oc get Elasticsearch

      Example output

      1. NAME            AGE
      2. elasticsearch   5h9m

    2. Get the log store status:

      1. $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      1. $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      1. status: (1)
      2.   cluster: (2)
      3.     activePrimaryShards: 30
      4.     activeShards: 60
      5.     initializingShards: 0
      6.     numDataNodes: 3
      7.     numNodes: 3
      8.     pendingTasks: 0
      9.     relocatingShards: 0
      10.     status: green
      11.     unassignedShards: 0
      12.   clusterHealth: ""
      13.   conditions: [] (3)
      14.   nodes: (4)
      15.   - deploymentName: elasticsearch-cdm-zjf34ved-1
      16.     upgradeStatus: {}
      17.   - deploymentName: elasticsearch-cdm-zjf34ved-2
      18.     upgradeStatus: {}
      19.   - deploymentName: elasticsearch-cdm-zjf34ved-3
      20.     upgradeStatus: {}
      21.   pods: (5)
      22.     client:
      23.       failed: []
      24.       notReady: []
      25.       ready:
      26.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      27.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      28.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      29.     data:
      30.       failed: []
      31.       notReady: []
      32.       ready:
      33.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      34.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      35.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      36.     master:
      37.       failed: []
      38.       notReady: []
      39.       ready:
      40.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      41.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      42.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      43.   shardAllocationEnabled: all
      1In the output, the cluster status fields appear in the status stanza.
      2The status of the log store:

      • The number of active primary shards.

      • The number of active shards.

      • The number of shards that are initializing.

      • The number of log store data nodes.

      • The total number of log store nodes.

      • The number of pending tasks.

      • The log store status: green, red, yellow.

      • The number of unassigned shards.

      3Any status conditions, if present. The log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the log store and proxy containers.

      • Container Terminated for both the log store and proxy containers.

      • Pod unschedulable. Also, a condition is shown for a number of issues, see Example condition messages.

      4The log store nodes in the cluster, with upgradeStatus.
      5The log store client, data, and master pods in the cluster, listed under ‘failed`, notReady or ready state.

Example condition messages

The following are examples of some condition messages from the Status section of the Elasticsearch instance.

This status message indicates a node has exceeded the configured low watermark and no shard will be allocated to this node.

  1. status:
  2.   nodes:
  3.   - conditions:
  4.     - lastTransitionTime: 2019-03-15T15:57:22Z
  5.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
  6.         be allocated on this node.
  7.       reason: Disk Watermark Low
  8.       status: "True"
  9.       type: NodeStorage
  10.     deploymentName: example-elasticsearch-cdm-0-1
  11.     upgradeStatus: {}

This status message indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes.

  1. status:
  2.   nodes:
  3.   - conditions:
  4.     - lastTransitionTime: 2019-03-15T16:04:45Z
  5.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
  6.         from this node.
  7.       reason: Disk Watermark High
  8.       status: "True"
  9.       type: NodeStorage
  10.     deploymentName: example-elasticsearch-cdm-0-1
  11.     upgradeStatus: {}

This status message indicates the log store node selector in the CR does not match any nodes in the cluster:

  1. status:
  2.     nodes:
  3.     - conditions:
  4.       - lastTransitionTime: 2019-04-10T02:26:24Z
  5.         message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
  6.         reason: Unschedulable
  7.         status: "True"
  8.         type: Unschedulable

This status message indicates that the log store CR uses a non-existent PVC.

  1. status:
  2.    nodes:
  3.    - conditions:
  4.      - last Transition Time:  2019-04-10T05:55:51Z
  5.        message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
  6.        reason:                Unschedulable
  7.        status:                True
  8.        type:                  Unschedulable

This status message indicates that your log store cluster does not have enough nodes to support your log store redundancy policy.

  1. status:
  2.   clusterHealth: ""
  3.   conditions:
  4.   - lastTransitionTime: 2019-04-17T20:01:31Z
  5.     message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
  6.       add more nodes with data roles
  7.     reason: Invalid Settings
  8.     status: "True"
  9.     type: InvalidRedundancy

This status message indicates your cluster has too many control plane nodes (also known as the master nodes):

  1. status:
  2.   clusterHealth: green
  3.   conditions:
  4.     - lastTransitionTime: '2019-04-17T20:12:34Z'
  5.       message: >-
  6.         Invalid master nodes count. Please ensure there are no more than 3 total
  7.         nodes with master roles
  8.       reason: Invalid Settings
  9.       status: 'True'
  10.       type: InvalidMasters

Viewing the status of the log store components

You can view the status for a number of the log store components.

Elasticsearch indices

You can view the status of the Elasticsearch indices.

  1. Get the name of an Elasticsearch pod:

    1. $ oc get pods --selector component=elasticsearch -o name

    Example output

    1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
    2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
    3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

  2. Get the status of the indices:

    1. $ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

    Example output

    1. Defaulting container name to elasticsearch.
    2. Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.
    4. green  open   infra-000002                                                     S4QANnf1QP6NgCegfnrnbQ   3   1     119926            0        157             78
    5. green  open   audit-000001                                                     8_EQx77iQCSTzFOXtxRqFw   3   1          0            0          0              0
    6. green  open   .security                                                        iDjscH7aSUGhIdq0LheLBQ   1   1          5            0          0              0
    7. green  open   .kibana_-377444158_kubeadmin                                     yBywZ9GfSrKebz5gWBZbjw   3   1          1            0          0              0
    8. green  open   infra-000001                                                     z6Dpe__ORgiopEpW6Yl44A   3   1     871000            0        874            436
    9. green  open   app-000001                                                       hIrazQCeSISewG3c2VIvsQ   3   1       2453            0          3              1
    10. green  open   .kibana_1                                                        JCitcBMSQxKOvIq6iQW6wg   1   1          0            0          0              0
    11. green  open   .kibana_-1595131456_user1                                        gIYFIEGRRe-ka0W3okS-mQ   3   1          1            0          0              0

Log store pods

You can view the status of the pods that host the log store.

  1. Get the name of a pod:

    1. $ oc get pods --selector component=elasticsearch -o name

    Example output

    1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
    2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
    3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

  2. Get the status of a pod:

    1. $ oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

    The output includes the following status information:

    Example output

    1. ....
    2. Status:             Running
    4. ....
    6. Containers:
    7.   elasticsearch:
    8.     Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
    9.     State:          Running
    10.       Started:      Mon, 08 Jun 2020 10:17:56 -0400
    11.     Ready:          True
    12.     Restart Count:  0
    13.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    15. ....
    17.   proxy:
    18.     Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
    19.     State:          Running
    20.       Started:      Mon, 08 Jun 2020 10:18:38 -0400
    21.     Ready:          True
    22.     Restart Count:  0
    24. ....
    26. Conditions:
    27.   Type              Status
    28.   Initialized       True
    29.   Ready             True
    30.   ContainersReady   True
    31.   PodScheduled      True
    33. ....
    35. Events:          <none>

Log storage pod deployment configuration

You can view the status of the log store deployment configuration.

  1. Get the name of a deployment configuration:

    1. $ oc get deployment --selector component=elasticsearch -o name

    Example output

    1. deployment.extensions/elasticsearch-cdm-1gon-1
    2. deployment.extensions/elasticsearch-cdm-1gon-2
    3. deployment.extensions/elasticsearch-cdm-1gon-3

  2. Get the deployment configuration status:

    1. $ oc describe deployment elasticsearch-cdm-1gon-1

    The output includes the following status information:

    Example output

    1. ....
    2.   Containers:
    3.    elasticsearch:
    4.     Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.3
    5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    7. ....
    9. Conditions:
    10.   Type           Status   Reason
    11.   ----           ------   ------
    12.   Progressing    Unknown  DeploymentPaused
    13.   Available      True     MinimumReplicasAvailable
    15. ....
    17. Events:          <none>

Log store replica set

You can view the status of the log store replica set.

  1. Get the name of a replica set:

    1. $ oc get replicaSet --selector component=elasticsearch -o name
    3. replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
    4. replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
    5. replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

  2. Get the status of the replica set:

    1. $ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

    The output includes the following status information:

    Example output

    1. ....
    2.   Containers:
    3.    elasticsearch:
    4.     Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch6@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25
    5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    7. ....
    9. Events:          <none>