Managing the maximum number of pods per node

In OKD, you can configure the number of pods that can run on a node based on the number of processor cores on the node, a hard limit or both. If you use both options, the lower of the two limits the number of pods on a node.

Exceeding these values can result in:

  • Increased CPU utilization by OKD.

  • Slow pod scheduling.

  • Potential out-of-memory scenarios, depending on the amount of memory in the node.

  • Exhausting the IP address pool.

  • Resource overcommitting, leading to poor user application performance.

A pod that is holding a single container actually uses two containers. The second container sets up networking prior to the actual container starting. As a result, a node running 10 pods actually has 20 containers running.

The podsPerCore parameter limits the number of pods the node can run based on the number of processor cores on the node. For example, if podsPerCore is set to 10 on a node with 4 processor cores, the maximum number of pods allowed on the node is 40.

The maxPods parameter limits the number of pods the node can run to a fixed value, regardless of the properties of the node.

Configuring the maximum number of pods per node

Two parameters control the maximum number of pods that can be scheduled to a node: podsPerCore and maxPods. If you use both options, the lower of the two limits the number of pods on a node.

For example, if podsPerCore is set to 10 on a node with 4 processor cores, the maximum number of pods allowed on the node will be 40.

Prerequisites

  1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

    1. View the machine config pool:

      1. $ oc describe machineconfigpool <name>

      For example:

      1. $ oc describe machineconfigpool worker

      Example output

      1. apiVersion: machineconfiguration.openshift.io/v1
      2. kind: MachineConfigPool
      3. metadata:
      4. creationTimestamp: 2019-02-08T14:52:39Z
      5. generation: 1
      6. labels:
      7. custom-kubelet: small-pods (1)
      1If a label has been added it appears under labels.
    2. If the label is not present, add a key/value pair:

      1. $ oc label machineconfigpool worker custom-kubelet=small-pods

      You can alternatively apply the following YAML to add the label:

      1. apiVersion: machineconfiguration.openshift.io/v1
      2. kind: MachineConfigPool
      3. metadata:
      4. labels:
      5. custom-kubelet: small-pods
      6. name: worker

Procedure

  1. Create a custom resource (CR) for your configuration change.

    Sample configuration for a max-pods CR

    1. apiVersion: machineconfiguration.openshift.io/v1
    2. kind: KubeletConfig
    3. metadata:
    4. name: set-max-pods (1)
    5. spec:
    6. machineConfigPoolSelector:
    7. matchLabels:
    8. custom-kubelet: small-pods (2)
    9. kubeletConfig:
    10. podsPerCore: 10 (3)
    11. maxPods: 250 (4)
    1Assign a name to CR.
    2Specify the label to apply the configuration change.
    3Specify the number of pods the node can run based on the number of processor cores on the node.
    4Specify the number of pods the node can run to a fixed value, regardless of the properties of the node.

    Setting podsPerCore to 0 disables this limit.

    In the above example, the default value for podsPerCore is 10 and the default value for maxPods is 250. This means that unless the node has 25 cores or more, by default, podsPerCore will be the limiting factor.

  2. List the MachineConfigPool CRDs to see if the change is applied. The UPDATING column reports True if the change is picked up by the Machine Config Controller:

    1. $ oc get machineconfigpools

    Example output

    1. NAME CONFIG UPDATED UPDATING DEGRADED
    2. master master-9cc2c72f205e103bb534 False False False
    3. worker worker-8cecd1236b33ee3f8a5e False True False

    Once the change is complete, the UPDATED column reports True.

    1. $ oc get machineconfigpools

    Example output

    1. NAME CONFIG UPDATED UPDATING DEGRADED
    2. master master-9cc2c72f205e103bb534 False True False
    3. worker worker-8cecd1236b33ee3f8a5e True False False