Viewing the status of the Elasticsearch log store

You can view the status of the OpenShift Elasticsearch Operator and for a number of Elasticsearch components.

Viewing the status of the log store

You can view the status of your log store.

Prerequisites

  • OpenShift Logging and Elasticsearch must be installed.

Procedure

  1. Change to the openshift-logging project.

    1. $ oc project openshift-logging

  2. To view the status:

    1. Get the name of the log store instance:

      1. $ oc get Elasticsearch

      Example output

      1. NAME            AGE
      2. elasticsearch   5h9m

    2. Get the log store status:

      1. $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      1. $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      1. status: (1)
      2.   cluster: (2)
      3.     activePrimaryShards: 30
      4.     activeShards: 60
      5.     initializingShards: 0
      6.     numDataNodes: 3
      7.     numNodes: 3
      8.     pendingTasks: 0
      9.     relocatingShards: 0
      10.     status: green
      11.     unassignedShards: 0
      12.   clusterHealth: ""
      13.   conditions: [] (3)
      14.   nodes: (4)
      15.   - deploymentName: elasticsearch-cdm-zjf34ved-1
      16.     upgradeStatus: {}
      17.   - deploymentName: elasticsearch-cdm-zjf34ved-2
      18.     upgradeStatus: {}
      19.   - deploymentName: elasticsearch-cdm-zjf34ved-3
      20.     upgradeStatus: {}
      21.   pods: (5)
      22.     client:
      23.       failed: []
      24.       notReady: []
      25.       ready:
      26.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      27.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      28.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      29.     data:
      30.       failed: []
      31.       notReady: []
      32.       ready:
      33.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      34.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      35.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      36.     master:
      37.       failed: []
      38.       notReady: []
      39.       ready:
      40.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      41.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      42.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
      43.   shardAllocationEnabled: all
      1In the output, the cluster status fields appear in the status stanza.
      2The status of the log store:

      • The number of active primary shards.

      • The number of active shards.

      • The number of shards that are initializing.

      • The number of log store data nodes.

      • The total number of log store nodes.

      • The number of pending tasks.

      • The log store status: green, red, yellow.

      • The number of unassigned shards.

      3Any status conditions, if present. The log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the log store and proxy containers.

      • Container Terminated for both the log store and proxy containers.

      • Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

      4The log store nodes in the cluster, with upgradeStatus.
      5The log store client, data, and master pods in the cluster, listed under ‘failed`, notReady, or ready state.

Example condition messages

The following are examples of some condition messages from the Status section of the Elasticsearch instance.

The following status message indicates that a node has exceeded the configured low watermark, and no shard will be allocated to this node.

  1. status:
  2.   nodes:
  3.   - conditions:
  4.     - lastTransitionTime: 2019-03-15T15:57:22Z
  5.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
  6.         be allocated on this node.
  7.       reason: Disk Watermark Low
  8.       status: "True"
  9.       type: NodeStorage
  10.     deploymentName: example-elasticsearch-cdm-0-1
  11.     upgradeStatus: {}

The following status message indicates that a node has exceeded the configured high watermark, and shards will be relocated to other nodes.

  1. status:
  2.   nodes:
  3.   - conditions:
  4.     - lastTransitionTime: 2019-03-15T16:04:45Z
  5.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
  6.         from this node.
  7.       reason: Disk Watermark High
  8.       status: "True"
  9.       type: NodeStorage
  10.     deploymentName: example-elasticsearch-cdm-0-1
  11.     upgradeStatus: {}

The following status message indicates that the log store node selector in the CR does not match any nodes in the cluster:

  1. status:
  2.     nodes:
  3.     - conditions:
  4.       - lastTransitionTime: 2019-04-10T02:26:24Z
  5.         message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
  6.         reason: Unschedulable
  7.         status: "True"
  8.         type: Unschedulable

The following status message indicates that the log store CR uses a non-existent persistent volume claim (PVC).

  1. status:
  2.    nodes:
  3.    - conditions:
  4.      - last Transition Time:  2019-04-10T05:55:51Z
  5.        message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
  6.        reason:                Unschedulable
  7.        status:                True
  8.        type:                  Unschedulable

The following status message indicates that your log store cluster does not have enough nodes to support the redundancy policy.

  1. status:
  2.   clusterHealth: ""
  3.   conditions:
  4.   - lastTransitionTime: 2019-04-17T20:01:31Z
  5.     message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
  6.       add more nodes with data roles
  7.     reason: Invalid Settings
  8.     status: "True"
  9.     type: InvalidRedundancy

This status message indicates your cluster has too many control plane nodes:

  1. status:
  2.   clusterHealth: green
  3.   conditions:
  4.     - lastTransitionTime: '2019-04-17T20:12:34Z'
  5.       message: >-
  6.         Invalid master nodes count. Please ensure there are no more than 3 total
  7.         nodes with master roles
  8.       reason: Invalid Settings
  9.       status: 'True'
  10.       type: InvalidMasters

The following status message indicates that Elasticsearch storage does not support the change you tried to make.

For example:

  1. status:
  2.   clusterHealth: green
  3.   conditions:
  4.     - lastTransitionTime: "2021-05-07T01:05:13Z"
  5.       message: Changing the storage structure for a custom resource is not supported
  6.       reason: StorageStructureChangeIgnored
  7.       status: 'True'
  8.       type: StorageStructureChangeIgnored

The reason and type fields specify the type of unsupported change:

StorageClassNameChangeIgnored

Unsupported change to the storage class name.

StorageSizeChangeIgnored

Unsupported change the storage size.

StorageStructureChangeIgnored

Unsupported change between ephemeral and persistent storage structures.

If you try to configure the ClusterLogging custom resource (CR) to switch from ephemeral to persistent storage, the OpenShift Elasticsearch Operator creates a persistent volume claim (PVC) but does not create a persistent volume (PV). To clear the StorageStructureChangeIgnored status, you must revert the change to the ClusterLogging CR and delete the PVC.

Viewing the status of the log store components

You can view the status for a number of the log store components.

Elasticsearch indices

You can view the status of the Elasticsearch indices.

  1. Get the name of an Elasticsearch pod:

    1. $ oc get pods --selector component=elasticsearch -o name

    Example output

    1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
    2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
    3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

  2. Get the status of the indices:

    1. $ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

    Example output

    1. Defaulting container name to elasticsearch.
    2. Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.
    4. green  open   infra-000002                                                     S4QANnf1QP6NgCegfnrnbQ   3   1     119926            0        157             78
    5. green  open   audit-000001                                                     8_EQx77iQCSTzFOXtxRqFw   3   1          0            0          0              0
    6. green  open   .security                                                        iDjscH7aSUGhIdq0LheLBQ   1   1          5            0          0              0
    7. green  open   .kibana_-377444158_kubeadmin                                     yBywZ9GfSrKebz5gWBZbjw   3   1          1            0          0              0
    8. green  open   infra-000001                                                     z6Dpe__ORgiopEpW6Yl44A   3   1     871000            0        874            436
    9. green  open   app-000001                                                       hIrazQCeSISewG3c2VIvsQ   3   1       2453            0          3              1
    10. green  open   .kibana_1                                                        JCitcBMSQxKOvIq6iQW6wg   1   1          0            0          0              0
    11. green  open   .kibana_-1595131456_user1                                        gIYFIEGRRe-ka0W3okS-mQ   3   1          1            0          0              0

Log store pods

You can view the status of the pods that host the log store.

  1. Get the name of a pod:

    1. $ oc get pods --selector component=elasticsearch -o name

    Example output

    1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
    2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
    3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

  2. Get the status of a pod:

    1. $ oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

    The output includes the following status information:

    Example output

    1. ....
    2. Status:             Running
    4. ....
    6. Containers:
    7.   elasticsearch:
    8.     Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
    9.     State:          Running
    10.       Started:      Mon, 08 Jun 2020 10:17:56 -0400
    11.     Ready:          True
    12.     Restart Count:  0
    13.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    15. ....
    17.   proxy:
    18.     Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
    19.     State:          Running
    20.       Started:      Mon, 08 Jun 2020 10:18:38 -0400
    21.     Ready:          True
    22.     Restart Count:  0
    24. ....
    26. Conditions:
    27.   Type              Status
    28.   Initialized       True
    29.   Ready             True
    30.   ContainersReady   True
    31.   PodScheduled      True
    33. ....
    35. Events:          <none>

Log storage pod deployment configuration

You can view the status of the log store deployment configuration.

  1. Get the name of a deployment configuration:

    1. $ oc get deployment --selector component=elasticsearch -o name

    Example output

    1. deployment.extensions/elasticsearch-cdm-1gon-1
    2. deployment.extensions/elasticsearch-cdm-1gon-2
    3. deployment.extensions/elasticsearch-cdm-1gon-3

  2. Get the deployment configuration status:

    1. $ oc describe deployment elasticsearch-cdm-1gon-1

    The output includes the following status information:

    Example output

    1. ....
    2.   Containers:
    3.    elasticsearch:
    4.     Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8
    5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    7. ....
    9. Conditions:
    10.   Type           Status   Reason
    11.   ----           ------   ------
    12.   Progressing    Unknown  DeploymentPaused
    13.   Available      True     MinimumReplicasAvailable
    15. ....
    17. Events:          <none>

Log store replica set

You can view the status of the log store replica set.

  1. Get the name of a replica set:

    1. $ oc get replicaSet --selector component=elasticsearch -o name
    3. replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
    4. replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
    5. replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

  2. Get the status of the replica set:

    1. $ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

    The output includes the following status information:

    Example output

    1. ....
    2.   Containers:
    3.    elasticsearch:
    4.     Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25
    5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    7. ....
    9. Events:          <none>

Elasticsearch cluster status

The Grafana dashboard in the Observe section of the OKD web console displays the status of the Elasticsearch cluster.

To get the status of the OpenShift Elasticsearch cluster, visit the Grafana dashboard in the Observe section of the OKD web console at <cluster_url>/monitoring/dashboards/grafana-dashboard-cluster-logging.

Elasticsearch status fields

eo_elasticsearch_cr_cluster_management_state

Shows whether the Elasticsearch cluster is in a managed or unmanaged state. For example:

  1. eo_elasticsearch_cr_cluster_management_state{state="managed"} 1
  2. eo_elasticsearch_cr_cluster_management_state{state="unmanaged"} 0

eo_elasticsearch_cr_restart_total

Shows the number of times the Elasticsearch nodes have restarted for certificate restarts, rolling restarts, or scheduled restarts. For example:

  1. eo_elasticsearch_cr_restart_total{reason="cert_restart"} 1
  2. eo_elasticsearch_cr_restart_total{reason="rolling_restart"} 1
  3. eo_elasticsearch_cr_restart_total{reason="scheduled_restart"} 3

es_index_namespaces_total

Shows the total number of Elasticsearch index namespaces. For example:

  1. Total number of Namespaces.
  2. es_index_namespaces_total 5

es_index_document_count

Shows the number of records for each namespace. For example:

  1. es_index_document_count{namespace="namespace_1"} 25
  2. es_index_document_count{namespace="namespace_2"} 10
  3. es_index_document_count{namespace="namespace_3"} 5

The “Secret Elasticsearch fields are either missing or empty” message

If Elasticsearch is missing the admin-cert, admin-key, logging-es.crt, or logging-es.key files, the dashboard shows a status message similar to the following example:

  1. message": "Secret \"elasticsearch\" fields are either missing or empty: [admin-cert, admin-key, logging-es.crt, logging-es.key]",
  2. "reason": "Missing Required Secrets",