Replacing a master host

You can replace a failed master host.

First, remove the failed master host from your cluster, and then add a replacement master host. If the failed master host ran etcd, scale up etcd by adding etcd to the new master host.

You must complete all sections of this topic.

Deprecating a master host

Master hosts run important services, such as the OKD API and controllers services. In order to deprecate a master host, these services must be stopped.

The OKD API service is an active/active service, so stopping the service does not affect the environment as long as the requests are sent to a separate master server. However, the OKD controllers service is an active/passive service, where the services use etcd to decide the active master.

Deprecating a master host in a multi-master architecture includes removing the master from the load balancer pool to avoid new connections attempting to use that master. This process depends heavily on the load balancer used. The steps below show the details of removing the master from haproxy. In the event that OKD is running on a cloud provider, or using a F5 appliance, see the specific product documents to remove the master from rotation.

Procedure

  1. Remove the backend section in the /etc/haproxy/haproxy.cfg configuration file. For example, if deprecating a master named master-0.example.com using haproxy, ensure the host name is removed from the following:

    1. backend mgmt8443
    2. balance source
    3. mode tcp
    4. # MASTERS 8443
    5. server master-1.example.com 192.168.55.12:8443 check
    6. server master-2.example.com 192.168.55.13:8443 check
  2. Then, restart the haproxy service.

    1. $ sudo systemctl restart haproxy
  3. Once the master is removed from the load balancer, disable the API and controller services by moving definition files out of the static pods dir /etc/origin/node/pods:

    1. # mkdir -p /etc/origin/node/pods/disabled
    2. # mv /etc/origin/node/pods/controller.yaml /etc/origin/node/pods/disabled/:
    3. +
  4. Because the master host is a schedulable OKD node, follow the steps in the Deprecating a node host section.

  5. Remove the master host from the [masters] and [nodes] groups in the /etc/ansible/hosts Ansible inventory file to avoid issues if running any Ansible tasks using that inventory file.

    Deprecating the first master host listed in the Ansible inventory file requires extra precautions.

    The /etc/origin/master/ca.serial.txt file is generated on only the first master listed in the Ansible host inventory. If you deprecate the first master host, copy the /etc/origin/master/ca.serial.txt file to the rest of master hosts before the process.

    In OKD 3.11 clusters running multiple masters, one of the master nodes includes additional CA certificates in /etc/origin/master, /etc/etcd/ca, and /etc/etcd/generated_certs. These are required for application node and etcd node scale-up operations and must be restored on another master node if the CA host master is being deprecated.

  6. The kubernetes service includes the master host IPs as endpoints. To verify that the master has been properly deprecated, review the kubernetes service output and see if the deprecated master has been removed:

    1. $ oc describe svc kubernetes -n default
    2. Name: kubernetes
    3. Namespace: default
    4. Labels: component=apiserver
    5. provider=kubernetes
    6. Annotations: <none>
    7. Selector: <none>
    8. Type: ClusterIP
    9. IP: 10.111.0.1
    10. Port: https 443/TCP
    11. Endpoints: 192.168.55.12:8443,192.168.55.13:8443
    12. Port: dns 53/UDP
    13. Endpoints: 192.168.55.12:8053,192.168.55.13:8053
    14. Port: dns-tcp 53/TCP
    15. Endpoints: 192.168.55.12:8053,192.168.55.13:8053
    16. Session Affinity: ClientIP
    17. Events: <none>

    After the master has been successfully deprecated, the host where the master was previously running can be safely deleted.

Adding hosts

You can add new hosts to your cluster by running the scaleup.yml playbook. This playbook queries the master, generates and distributes new certificates for the new hosts, and then runs the configuration playbooks on only the new hosts. Before running the scaleup.yml playbook, complete all prerequisite host preparation steps.

The scaleup.yml playbook configures only the new host. It does not update NO_PROXY in master services, and it does not restart master services.

You must have an existing inventory file, for example /etc/ansible/hosts, that is representative of your current cluster configuration in order to run the scaleup.yml playbook.

See the cluster maximums section for the recommended maximum number of nodes.

Procedure

  1. Ensure you have the latest playbooks by updating the openshift-ansible package:

    1. # yum update openshift-ansible
  2. Edit your /etc/ansible/hosts file and add new_ to the [OSEv3:children] section. For example, to add a new node host, add new_nodes:

    1. [OSEv3:children]
    2. masters
    3. nodes
    4. new_nodes

    To add new master hosts, add new_masters.

  3. Create a [new_] section to specify host information for the new hosts. Format this section like an existing section, as shown in the following example of adding a new node:

    1. [nodes]
    2. master[1:3].example.com
    3. node1.example.com openshift_node_group_name='node-config-compute'
    4. node2.example.com openshift_node_group_name='node-config-compute'
    5. infra-node1.example.com openshift_node_group_name='node-config-infra'
    6. infra-node2.example.com openshift_node_group_name='node-config-infra'
    7. [new_nodes]
    8. node3.example.com openshift_node_group_name='node-config-infra'

    See Configuring Host Variables for more options.

    When adding new masters, add hosts to both the [new_masters] section and the [new_nodes] section to ensure that the new master host is part of the OpenShift SDN:

    1. [masters]
    2. master[1:2].example.com
    3. [new_masters]
    4. master3.example.com
    5. [nodes]
    6. master[1:2].example.com
    7. node1.example.com openshift_node_group_name='node-config-compute'
    8. node2.example.com openshift_node_group_name='node-config-compute'
    9. infra-node1.example.com openshift_node_group_name='node-config-infra'
    10. infra-node2.example.com openshift_node_group_name='node-config-infra'
    11. [new_nodes]
    12. master3.example.com

    If you label a master host with the node-role.kubernetes.io/infra=true label and have no other dedicated infrastructure nodes, you must also explicitly mark the host as schedulable by adding openshift_schedulable=true to the entry. Otherwise, the registry and router pods cannot be placed anywhere.

  4. Change to the playbook directory and run the openshift_node_group.yml playbook. If your inventory file is located somewhere other than the default of /etc/ansible/hosts, specify the location with the -i option:

    1. $ cd /usr/share/ansible/openshift-ansible
    2. $ ansible-playbook [-i /path/to/file] \
    3. playbooks/openshift-master/openshift_node_group.yml

    This creates the ConfigMap for the new node groups, and ultimately, the configuration file of the node on the host.

    Running the openshift_node_group.yaml playbook only updates new nodes. It cannot be run to update existing nodes in a cluster.

  5. Run the scaleup.yml playbook. If your inventory file is located somewhere other than the default of /etc/ansible/hosts, specify the location with the -i option.

    • For additional nodes:

      1. $ ansible-playbook [-i /path/to/file] \
      2. playbooks/openshift-node/scaleup.yml
    • For additional masters:

      1. $ ansible-playbook [-i /path/to/file] \
      2. playbooks/openshift-master/scaleup.yml
  1. Set the node label to logging-infra-fluentd=true, if you deployed the EFK stack in your cluster:

    1. # oc label node/new-node.example.com logging-infra-fluentd=true
  2. After the playbook runs, verify the installation.

  3. Move any hosts that you defined in the [new_] section to their appropriate section. By moving these hosts, subsequent playbook runs that use this inventory file treat the nodes correctly. You can keep the empty [new_] section. For example, when adding new nodes:

    1. [nodes]
    2. master[1:3].example.com
    3. node1.example.com openshift_node_group_name='node-config-compute'
    4. node2.example.com openshift_node_group_name='node-config-compute'
    5. node3.example.com openshift_node_group_name='node-config-compute'
    6. infra-node1.example.com openshift_node_group_name='node-config-infra'
    7. infra-node2.example.com openshift_node_group_name='node-config-infra'
    8. [new_nodes]

Scaling etcd

You can scale the etcd cluster vertically by adding more resources to the etcd hosts or horizontally by adding more etcd hosts.

Due to the voting system etcd uses, the cluster must always contain an odd number of members.

Having a cluster with an odd number of etcd hosts can account for fault tolerance. Having an odd number of etcd hosts does not change the number needed for a quorum but increases the tolerance for failure. For example, with a cluster of three members, quorum is two, which leaves a failure tolerance of one. This ensures the cluster continues to operate if two of the members are healthy.

Having an in-production cluster of three etcd hosts is recommended.

The new host requires a fresh Red Hat Enterprise Linux version 7 dedicated host. The etcd storage should be located on an SSD disk to achieve maximum performance and on a dedicated disk mounted in /var/lib/etcd.

Prerequisites

  1. Before you add a new etcd host, perform a backup of both etcd configuration and data to prevent data loss.

  2. Check the current etcd cluster status to avoid adding new hosts to an unhealthy cluster. Run this command:

    1. # ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
    2. --key=/etc/etcd/peer.key \
    3. --cacert="/etc/etcd/ca.crt" \
    4. --endpoints="https://*master-0.example.com*:2379,\
    5. https://*master-1.example.com*:2379,\
    6. https://*master-2.example.com*:2379"
    7. endpoint health
    8. https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
    9. https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
    10. https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms
  3. Before running the scaleup playbook, ensure the new host is registered to the proper Red Hat software channels:

    1. # subscription-manager register \
    2. --username=*<username>* --password=*<password>*
    3. # subscription-manager attach --pool=*<poolid>*
    4. # subscription-manager repos --disable="*"
    5. # subscription-manager repos \
    6. --enable=rhel-7-server-rpms \
    7. --enable=rhel-7-server-extras-rpms

    etcd is hosted in the rhel-7-server-extras-rpms software channel.

  4. Make sure all unused etcd members are removed from the etcd cluster. This must be completed before running the scaleup playbook.

    1. List the etcd members:

      1. # etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
      2. --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URLS member list -w table

      Copy the unused etcd member ID, if applicable.

    2. Remove the unused member by specifying its ID in the following command:

      1. # etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
      2. --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URL member remove UNUSED_ETCD_MEMBER_ID
  1. Upgrade etcd and iptables on the current etcd nodes:

    1. # yum update etcd iptables-services
  2. Back up the /etc/etcd configuration for the etcd hosts.

  3. If the new etcd members will also be OKD nodes, add the desired number of hosts to the cluster.

  4. The rest of this procedure assumes you added one host, but if you add multiple hosts, perform all steps on each host.

Adding a new etcd host using Ansible

Procedure

  1. In the Ansible inventory file, create a new group named [new_etcd] and add the new host. Then, add the new_etcd group as a child of the [OSEv3] group:

    1. [OSEv3:children]
    2. masters
    3. nodes
    4. etcd
    5. new_etcd (1)
    6. ... [OUTPUT ABBREVIATED] ...
    7. [etcd]
    8. master-0.example.com
    9. master-1.example.com
    10. master-2.example.com
    11. [new_etcd] (1)
    12. etcd0.example.com (1)
    1Add these lines.
  2. From the host that installed OKD and hosts the Ansible inventory file, change to the playbook directory and run the etcd scaleup playbook:

    1. $ cd /usr/share/ansible/openshift-ansible
    2. $ ansible-playbook playbooks/openshift-etcd/scaleup.yml
  3. After the playbook runs, modify the inventory file to reflect the current status by moving the new etcd host from the [new_etcd] group to the [etcd] group:

    1. [OSEv3:children]
    2. masters
    3. nodes
    4. etcd
    5. new_etcd
    6. ... [OUTPUT ABBREVIATED] ...
    7. [etcd]
    8. master-0.example.com
    9. master-1.example.com
    10. master-2.example.com
    11. etcd0.example.com
  4. If you use Flannel, modify the flanneld service configuration on every OKD host, located at /etc/sysconfig/flanneld, to include the new etcd host:

    1. FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379
  5. Restart the flanneld service:

    1. # systemctl restart flanneld.service

Manually adding a new etcd host

If you do not run etcd as static pods on master nodes, you might need to add another etcd host.

Procedure

Modify the current etcd cluster

To create the etcd certificates, run the openssl command, replacing the values with those from your environment.

  1. Create some environment variables:

    1. export NEW_ETCD_HOSTNAME="*etcd0.example.com*"
    2. export NEW_ETCD_IP="192.168.55.21"
    3. export CN=$NEW_ETCD_HOSTNAME
    4. export SAN="IP:${NEW_ETCD_IP}, DNS:${NEW_ETCD_HOSTNAME}"
    5. export PREFIX="/etc/etcd/generated_certs/etcd-$CN/"
    6. export OPENSSLCFG="/etc/etcd/ca/openssl.cnf"

    The custom openssl extensions used as etcdv3_ca* include the $SAN environment variable as subjectAltName. See /etc/etcd/ca/openssl.cnf for more information.

  2. Create the directory to store the configuration and certificates:

    1. # mkdir -p ${PREFIX}
  3. Create the server certificate request and sign it: (server.csr and server.crt)

    1. # openssl req -new -config ${OPENSSLCFG} \
    2. -keyout ${PREFIX}server.key \
    3. -out ${PREFIX}server.csr \
    4. -reqexts etcd_v3_req -batch -nodes \
    5. -subj /CN=$CN
    6. # openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    7. -out ${PREFIX}server.crt \
    8. -in ${PREFIX}server.csr \
    9. -extensions etcd_v3_ca_server -batch
  4. Create the peer certificate request and sign it: (peer.csr and peer.crt)

    1. # openssl req -new -config ${OPENSSLCFG} \
    2. -keyout ${PREFIX}peer.key \
    3. -out ${PREFIX}peer.csr \
    4. -reqexts etcd_v3_req -batch -nodes \
    5. -subj /CN=$CN
    6. # openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    7. -out ${PREFIX}peer.crt \
    8. -in ${PREFIX}peer.csr \
    9. -extensions etcd_v3_ca_peer -batch
  5. Copy the current etcd configuration and ca.crt files from the current node as examples to modify later:

    1. # cp /etc/etcd/etcd.conf ${PREFIX}
    2. # cp /etc/etcd/ca.crt ${PREFIX}
  6. While still on the surviving etcd host, add the new host to the cluster. To add additional etcd members to the cluster, you must first adjust the default localhost peer in the **peerURLs** value for the first member:

    1. Get the member ID for the first member using the member list command:

      1. # etcdctl --cert-file=/etc/etcd/peer.crt \
      2. --key-file=/etc/etcd/peer.key \
      3. --ca-file=/etc/etcd/ca.crt \
      4. --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \ (1)
      5. member list
      1Ensure that you specify the URLs of only active etcd members in the —peers parameter value.
    2. Obtain the IP address where etcd listens for cluster peers:

      1. $ ss -l4n | grep 2380
    3. Update the value of **peerURLs** using the etcdctl member update command by passing the member ID and IP address obtained from the previous steps:

      1. # etcdctl --cert-file=/etc/etcd/peer.crt \
      2. --key-file=/etc/etcd/peer.key \
      3. --ca-file=/etc/etcd/ca.crt \
      4. --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
      5. member update 511b7fb6cc0001 https://172.18.1.18:2380
    4. Re-run the member list command and ensure the peer URLs no longer include localhost.

  1. Add the new host to the etcd cluster. Note that the new host is not yet configured, so the status stays as unstarted until the you configure the new host.

    You must add each member and bring it online one at a time. When you add each additional member to the cluster, you must adjust the peerURLs list for the current peers. The peerURLs list grows by one for each member added. The etcdctl member add command outputs the values that you must set in the etcd.conf file as you add each member, as described in the following instructions.

    1. # etcdctl -C https://${CURRENT_ETCD_HOST}:2379 \
    2. --ca-file=/etc/etcd/ca.crt \
    3. --cert-file=/etc/etcd/peer.crt \
    4. --key-file=/etc/etcd/peer.key member add ${NEW_ETCD_HOSTNAME} https://${NEW_ETCD_IP}:2380 (1)
    5. Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster
    6. ETCD_NAME="<NEW_ETCD_HOSTNAME>"
    7. ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<CLUSTERMEMBER1_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER2_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER3_NAME>=https:/<CLUSTERMEMBER3_IP>:2380"
    8. ETCD_INITIAL_CLUSTER_STATE="existing"
    1In this line, 10.3.9.222 is a label for the etcd member. You can specify the host name, IP address, or a simple name.
  2. Update the sample ${PREFIX}/etcd.conf file.

    1. Replace the following values with the values generated in the previous step:

      • ETCD_NAME

      • ETCD_INITIAL_CLUSTER

      • ETCD_INITIAL_CLUSTER_STATE

  1. 2. Modify the following variables with the new host IP from the output of the previous step. You can use `${NEW_ETCD_IP}` as the value.
  2. ```
  3. ETCD_LISTEN_PEER_URLS
  4. ETCD_LISTEN_CLIENT_URLS
  5. ETCD_INITIAL_ADVERTISE_PEER_URLS
  6. ETCD_ADVERTISE_CLIENT_URLS
  7. ```
  8. 3. If you previously used the member system as an etcd node, you must overwrite the current values in the ***/etc/etcd/etcd.conf*** file.
  9. 4. Check the file for syntax errors or missing IP addresses, otherwise the etcd service might fail:
  10. ```
  11. # vi ${PREFIX}/etcd.conf
  12. ```
  1. On the node that hosts the installation files, update the [etcd] hosts group in the /etc/ansible/hosts inventory file. Remove the old etcd hosts and add the new ones.

  2. Create a tgz file that contains the certificates, the sample configuration file, and the ca and copy it to the new host:

    1. # tar -czvf /etc/etcd/generated_certs/${CN}.tgz -C ${PREFIX} .
    2. # scp /etc/etcd/generated_certs/${CN}.tgz ${CN}:/tmp/
Modify the new etcd host
  1. Install iptables-services to provide iptables utilities to open the required ports for etcd:

    1. # yum install -y iptables-services
  2. Create the OS_FIREWALL_ALLOW firewall rules to allow etcd to communicate:

    • Port 2379/tcp for clients

    • Port 2380/tcp for peer communication

      1. # systemctl enable iptables.service --now
      2. # iptables -N OS_FIREWALL_ALLOW
      3. # iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
      4. # iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
      5. # iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
      6. # iptables-save | tee /etc/sysconfig/iptables

      In this example, a new chain OS_FIREWALL_ALLOW is created, which is the standard naming the OKD installer uses for firewall rules.

      If the environment is hosted in an IaaS environment, modify the security groups for the instance to allow incoming traffic to those ports as well.

  1. Install etcd:

    1. # yum install -y etcd

    Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed,

  2. Ensure the etcd service is not running by removing the etcd pod definition:

    1. # mkdir -p /etc/origin/node/pods-stopped
    2. # mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
  3. Remove any etcd configuration and data:

    1. # rm -Rf /etc/etcd/*
    2. # rm -Rf /var/lib/etcd/*
  4. Extract the certificates and configuration files:

    1. # tar xzvf /tmp/etcd0.example.com.tgz -C /etc/etcd/
  5. Start etcd on the new host:

    1. # systemctl enable etcd --now
  6. Verify that the host is part of the cluster and the current cluster health:

    • If you use the v2 etcd api, run the following command:

      1. # etcdctl --cert-file=/etc/etcd/peer.crt \
      2. --key-file=/etc/etcd/peer.key \
      3. --ca-file=/etc/etcd/ca.crt \
      4. --peers="https://*master-0.example.com*:2379,\
      5. https://*master-1.example.com*:2379,\
      6. https://*master-2.example.com*:2379,\
      7. https://*etcd0.example.com*:2379"\
      8. cluster-health
      9. member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
      10. member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
      11. member 8b8904727bf526a5 is healthy: got healthy result from https://192.168.55.21:2379
      12. member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
      13. cluster is healthy
    • If you use the v3 etcd api, run the following command:

      1. # ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
      2. --key=/etc/etcd/peer.key \
      3. --cacert="/etc/etcd/ca.crt" \
      4. --endpoints="https://*master-0.example.com*:2379,\
      5. https://*master-1.example.com*:2379,\
      6. https://*master-2.example.com*:2379,\
      7. https://*etcd0.example.com*:2379"\
      8. endpoint health
      9. https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
      10. https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
      11. https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms
      12. https://etcd0.example.com:2379 is healthy: successfully committed proposal: took = 1.498829ms
Modify each OKD master
  1. Modify the master configuration in the etcClientInfo section of the /etc/origin/master/master-config.yaml file on every master. Add the new etcd host to the list of the etcd servers OKD uses to store the data, and remove any failed etcd hosts:

    1. etcdClientInfo:
    2. ca: master.etcd-ca.crt
    3. certFile: master.etcd-client.crt
    4. keyFile: master.etcd-client.key
    5. urls:
    6. - https://master-0.example.com:2379
    7. - https://master-1.example.com:2379
    8. - https://master-2.example.com:2379
    9. - https://etcd0.example.com:2379
  2. Restart the master API service:

    • On every master:

      1. # master-restart api
      2. # master-restart controllers

      The number of etcd nodes must be odd, so you must add at least two hosts.

  1. If you use Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every OKD host to include the new etcd host:

    1. FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379
  2. Restart the flanneld service:

    1. # systemctl restart flanneld.service