Recommended infrastructure practices

This topic provides recommended performance and scalability practices for infrastructure in OKD.

Infrastructure node sizing

Infrastructure nodes are nodes that are labeled to run pieces of the OKD environment. The infrastructure node resource requirements depend on the cluster age, nodes, and objects in the cluster, as these factors can lead to an increase in the number of metrics or time series in Prometheus. The following infrastructure node size recommendations are based on the results observed in cluster-density testing detailed in the Control plane node sizing section, where the monitoring stack and the default ingress-controller were moved to these nodes.

Number of worker nodesCluster density, or number of namespacesCPU coresMemory (GB)

27

500

4

24

120

1000

8

48

252

4000

16

128

501

4000

32

128

In general, three infrastructure nodes are recommended per cluster.

These sizing recommendations should be used as a guideline. Prometheus is a highly memory intensive application; the resource usage depends on various factors including the number of nodes, objects, the Prometheus metrics scraping interval, metrics or time series, and the age of the cluster. In addition, the router resource usage can also be affected by the number of routes and the amount/type of inbound requests.

These recommendations apply only to infrastructure nodes hosting Monitoring, Ingress and Registry infrastructure components installed during cluster creation.

In OKD 4, half of a CPU core (500 millicore) is now reserved by the system by default compared to OKD 3.11 and previous versions. This influences the stated sizing recommendations.

Scaling the Cluster Monitoring Operator

OKD exposes metrics that the Cluster Monitoring Operator collects and stores in the Prometheus-based monitoring stack. As an administrator, you can view dashboards for system resources, containers, and components metrics in the OKD web console by navigating to ObserveDashboards.

Prometheus database storage requirements

Red Hat performed various tests for different scale sizes.

The Prometheus storage requirements below are not prescriptive and should be used as a reference. Higher resource consumption might be observed in your cluster depending on workload activity and resource density, including the number of pods, containers, routes, or other resources exposing metrics collected by Prometheus.

Table 1. Prometheus Database storage requirements based on number of nodes/pods in the cluster
Number of NodesNumber of pods (2 containers per pod)Prometheus storage growth per dayPrometheus storage growth per 15 daysNetwork (per tsdb chunk)

50

1800

6.3 GB

94 GB

16 MB

100

3600

13 GB

195 GB

26 MB

150

5400

19 GB

283 GB

36 MB

200

7200

25 GB

375 GB

46 MB

Approximately 20 percent of the expected size was added as overhead to ensure that the storage requirements do not exceed the calculated value.

The above calculation is for the default OKD Cluster Monitoring Operator.

CPU utilization has minor impact. The ratio is approximately 1 core out of 40 per 50 nodes and 1800 pods.

Recommendations for OKD

  • Use at least two infrastructure (infra) nodes.

  • Use at least three openshift-container-storage nodes with non-volatile memory express (SSD or NVMe) drives.

Configuring cluster monitoring

You can increase the storage capacity for the Prometheus component in the cluster monitoring stack.

Procedure

To increase the storage capacity for Prometheus:

  1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. data:
    4. config.yaml: |
    5. prometheusK8s:
    6. retention: {{PROMETHEUS_RETENTION_PERIOD}} (1)
    7. nodeSelector:
    8. node-role.kubernetes.io/infra: ""
    9. volumeClaimTemplate:
    10. spec:
    11. storageClassName: {{STORAGE_CLASS}} (2)
    12. resources:
    13. requests:
    14. storage: {{PROMETHEUS_STORAGE_SIZE}} (3)
    15. alertmanagerMain:
    16. nodeSelector:
    17. node-role.kubernetes.io/infra: ""
    18. volumeClaimTemplate:
    19. spec:
    20. storageClassName: {{STORAGE_CLASS}} (2)
    21. resources:
    22. requests:
    23. storage: {{ALERTMANAGER_STORAGE_SIZE}} (4)
    24. metadata:
    25. name: cluster-monitoring-config
    26. namespace: openshift-monitoring
    1The default value of Prometheus retention is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.
    2The storage class for your cluster.
    3A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
    4A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
  2. Add values for the retention period, storage class, and storage sizes.

  3. Save the file.

  4. Apply the changes by running:

    1. $ oc create -f cluster-monitoring-config.yaml

Additional resources