Expanding the cluster

After deploying an installer-provisioned OKD cluster, you can use the following procedures to expand the number of worker nodes. Ensure that each prospective worker node meets the prerequisites.

Preparing the bare metal node

Preparing the bare metal node requires executing the following procedure from the provisioner node.

Procedure

  1. Get the oc binary, if needed. It should already exist on the provisioner node.

    1. $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc
    1. $ sudo cp oc /usr/local/bin

  2. Install the ipmitool.

    1. $ sudo dnf install -y OpenIPMI ipmitool

  3. Power off the bare metal node and ensure it is off.

    1. $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

    Where <management-server-ip> is the IP address of the bare metal node’s base board management controller.

    1. $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power status
    1. Chassis Power is off

  4. Retrieve the username and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the username and password. In the following example, the username is root and the password is calvin.

    1. $ echo -ne "root" | base64
    1. $ echo -ne "calvin" | base64

  5. Create a configuration file for the bare metal node.

    1. $ vim bmh.yaml
    1. ---
    2. apiVersion: v1
    3. kind: Secret
    4. metadata:
    5.   name: openshift-worker-<num>-bmc-secret
    6. type: Opaque
    7. data:
    8.   username: <base64-of-uid>
    9.   password: <base64-of-pwd>
    10. ---
    11. apiVersion: metal3.io/v1alpha1
    12. kind: BareMetalHost
    13. metadata:
    14.   name: openshift-worker-<num>
    15. spec:
    16.   online: true
    17.   bootMACAddress: <NIC1-mac-address>
    18.   bmc:
    19.     address: ipmi://<bmc-ip>
    20.     credentialsName: openshift-worker-<num>-bmc-secret

    Replace <num> for the worker number of bare metal node in two name fields and credentialsName field. Replace <base64-of-uid> with the base64 string of the username. Replace <base64-of-pwd> with the base64 string of the password. Replace <NIC1-mac-address> with the MAC address of the bare metal node’s first NIC. Replace <bmc-ip> with the IP address of the bare metal node’s baseboard management controller.

  6. Create the bare metal node.

    1. $ oc -n openshift-machine-api create -f bmh.yaml
    1. secret/openshift-worker-<num>-bmc-secret created
    2. baremetalhost.metal3.io/openshift-worker-<num> created

    Where <num> will be the worker number.

  7. Power up and inspect the bare metal node.

    1. $ oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number.

    1. NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
    2. openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

Provisioning the bare metal node

Provisioning the bare metal node requires executing the following procedure from the provisioner node.

Procedure

  1. Ensure the PROVISIONING STATUS is ready before provisioning the bare metal node.

    1. $  oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number.

    1. NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
    2. openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

  2. Get a count of the number of worker nodes.

    1. $ oc get nodes
    1. NAME                                                STATUS   ROLES           AGE     VERSION
    2. provisioner.openshift.example.com            Ready    master          30h     v1.16.2
    3. openshift-master-1.openshift.example.com            Ready    master          30h     v1.16.2
    4. openshift-master-2.openshift.example.com            Ready    master          30h     v1.16.2
    5. openshift-master-3.openshift.example.com            Ready    master          30h     v1.16.2
    6. openshift-worker-0.openshift.example.com            Ready    master          30h     v1.16.2
    7. openshift-worker-1.openshift.example.com            Ready    master          30h     v1.16.2

  3. Get the machine set.

    1. $ oc get machinesets -n openshift-machine-api
    1. NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
    2. ...
    3. openshift-worker-0.example.com      1         1         1       1           55m
    4. openshift-worker-1.example.com      1         1         1       1           55m

  4. Increase the number of worker nodes by one.

    1. $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

    Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the machine set from the previous step.

  5. Check the status of the bare metal node.

    1. $ oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number. The status changes from ready to provisioning.

    1. NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
    2. openshift-worker-<num>   OK       provisioning          openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

    The provisioning status remains until the OKD cluster provisions the node. This can take 30 minutes or more. Once complete, the status will change to provisioned.

    1. NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
    2. openshift-worker-<num>   OK       provisioned           openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

  6. Once provisioned, ensure the bare metal node is ready.

    1. $ oc get nodes
    1. NAME                                          STATUS   ROLES   AGE     VERSION
    2. provisioner.openshift.example.com             Ready    master  30h     v1.16.2
    3. openshift-master-1.openshift.example.com      Ready    master  30h     v1.16.2
    4. openshift-master-2.openshift.example.com      Ready    master  30h     v1.16.2
    5. openshift-master-3.openshift.example.com      Ready    master  30h     v1.16.2
    6. openshift-worker-0.openshift.example.com      Ready    master  30h     v1.16.2
    7. openshift-worker-1.openshift.example.com      Ready    master  30h     v1.16.2
    8. openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.16.2

    You can also check the kubelet.

    1. $ ssh openshift-worker-<num>
    1. [kni@openshift-worker-<num>]$ journalctl -fu kubelet