Install an OpenShift 4 cluster on Windows nodes

Big picture

Install an OpenShift 4 cluster on AWS with Calico on Windows nodes.

Value

Run Windows workloads on OpenShift 4 with Calico.

How to

Before you begin

Limitations

Due to an upstream issue, Windows pods can only be run in specific namespaces if you disable SCC. To do this, label the namespace with openshift.io/run-level: "1".

Create a configuration file for the OpenShift installer

First, create a staging directory for the installation. This directory will contain the configuration file, along with cluster state files, that OpenShift installer will create:

  1. mkdir openshift-tigera-install && cd openshift-tigera-install

Now run OpenShift installer to create a default configuration file:

  1. openshift-install create install-config

Install an OpenShift 4 cluster on Windows nodes - 图1note

Refer to the OpenShift installer documentation for more information about the installer and any configuration changes required for your platform.

Once the installer has finished, your staging directory will contain the configuration file install-config.yaml.

Update the configuration file to use Calico

Override the OpenShift networking to use Calico and update the AWS instance types to meet the system requirements:

  1. sed -i 's/OpenShiftSDN/Calico/' install-config.yaml

Generate the install manifests

Now generate the Kubernetes manifests using your configuration file:

  1. openshift-install create manifests

Download the Calico manifests for OpenShift and add them to the generated manifests directory:

  1. mkdir calico
  2. wget -qO- https://github.com/projectcalico/calico/releases/download/v3.24.5/ocp.tgz | tar xvz --strip-components=1 -C calico
  3. cp calico/* manifests/

Configure VXLAN

Edit the Installation custom resource manifest manifests/01-cr-installation.yaml so that it configures an OpenShift Calico cluster with VXLAN enabled and BGP disabled:

  1. apiVersion: operator.tigera.io/v1
  2. kind: Installation
  3. metadata:
  4. name: default
  5. spec:
  6. variant: Calico
  7. calicoNetwork:
  8. bgp: Disabled
  9. ipPools:
  10. - blockSize: 26
  11. cidr: 10.128.0.0/14
  12. encapsulation: VXLAN
  13. natOutgoing: Enabled
  14. nodeSelector: all()

Create the cluster

Start the cluster creation with the following command and wait for it to complete.

  1. openshift-install create cluster

Once the above command is complete, you can verify Calico is installed by verifying the components are available with the following command.

  1. oc get tigerastatus

Install an OpenShift 4 cluster on Windows nodes - 图2note

To get more information, add -o yaml to the above command.

Next, install calicoctl and ensure strict affinity is true:

  1. kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

Add Windows nodes to the cluster

Download the latest Windows Node Installer (WNI) binary wni that matches your OpenShift minor version.

Install an OpenShift 4 cluster on Windows nodes - 图3note

For OpenShift 4.6, use the latest wni for OpenShift 4.5. A wni binary for OpenShift 4.6 is not published yet.

Next, determine the AMI id corresponding to Windows Server 1903 (build 18317) or greater. wni defaults to using Windows Server 2019 (build 10.0.17763) which does not include WinDSR support. One way to do this is by searching for AMI’s matching the string Windows_Server-1903-English-Core-ContainersLatest in the Amazon EC2 console

Next, run wni to add a Windows node to your cluster. Replace AMI_ID, AWS_CREDENTIALS_PATH, AWS_KEY_NAME and AWS_PRIVATE_KEY_PATH with your values:

  1. chmod u+x wni
  2. ./wni aws create \
  3. --image-id AMI_ID \
  4. --kubeconfig openshift-tigera-install/auth/kubeconfig \
  5. --credentials AWS_CREDENTIALS_PATH \
  6. --credential-account default \
  7. --instance-type m5a.large \
  8. --ssh-key AWS_KEY_NAME \
  9. --private-key AWS_PRIVATE_KEY_PATH

An example of running the above steps:

  1. $ chmod u+x wni
  2. $ ./wni aws create \
  3. > --kubeconfig openshift-tigera-install/auth/kubeconfig \
  4. > --credentials ~/.aws/credentials \
  5. > --credential-account default \
  6. > --instance-type m5a.large \
  7. > --ssh-key test-key \
  8. > --private-key /home/user/.ssh/test-key.pem
  9. 2020/10/05 12:52:51 kubeconfig source: /home/user/openshift-tigera-install/auth/kubeconfig
  10. 2020/10/05 12:52:59 Added rule with port 5986 to the security groups of your local IP
  11. 2020/10/05 12:52:59 Added rule with port 22 to the security groups of your local IP
  12. 2020/10/05 12:52:59 Added rule with port 3389 to the security groups of your local IP
  13. 2020/10/05 12:52:59 Using existing Security Group: sg-06d1de22807d5dc48
  14. 2020/10/05 12:57:30 External IP: 52.35.12.231
  15. 2020/10/05 12:57:30 Internal IP: 10.0.90.193

Get the administrator password

The wni binary writes the instance details to the file windows-node-installer.json. An example of the file:

  1. {"InstanceIDs":["i-02e13d4cc76c13c83"],"SecurityGroupIDs":["sg-0a777565d64e1d2ef"]}

Use the instance ID from the file and the path of the private key used to create the instance to get the Administrator user’s password:

  1. aws ec2 get-password-data --instance-id <instance id> --priv-launch-key <aws private key path>

Install Calico for Windows

  1. Remote into the Windows node, open a Powershell window, and prepare the directory for Kubernetes files.

    1. mkdir c:\k
  2. Copy the Kubernetes kubeconfig file (default location: openshift-tigera-install/auth/kubeconfig), to the file c:\k\config.

  3. Download the powershell script, install-calico-windows.ps1.

    1. Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.24.5/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1
  4. Run the installation script, replacing the Kubernetes version with the version corresponding to your version of OpenShift.

    1. c:\install-calico-windows.ps1 -KubeVersion <kube version> -ServiceCidr 172.30.0.0/16 -DNSServerIPs 172.30.0.10

    Install an OpenShift 4 cluster on Windows nodes - 图4note

    Get the Kubernetes version with oc version and use only the major, minor, and patch version numbers. For example from a cluster that returns:

    1. $ oc version
    2. Client Version: 4.5.3
    3. Server Version: 4.5.14
    4. Kubernetes Version: v1.18.3+5302882

    You will use 1.18.3:

  5. Install and start kube-proxy service. Execute following powershell script/commands.

    1. C:\CalicoWindows\kubernetes\install-kube-services.ps1 -service kube-proxy
    2. Start-Service -Name kube-proxy
  6. Verify kube-proxy service is running.

    1. Get-Service -Name kube-proxy

Configure kubelet

From the Windows node, download the Windows Machine Config Bootstrapper wmcb.exe that matches your OpenShift minor version from Windows Machine Config Bootstrapper releases. For example, for OpenShift 4.5.x:

  1. curl https://github.com/openshift/windows-machine-config-bootstrapper/releases/download/v4.5.2-alpha/wmcb.exe -o c:\wmcb.exe

Install an OpenShift 4 cluster on Windows nodes - 图5note

For OpenShift 4.6, use the latest wmcb.exe for OpenShift 4.5. A wmcb.ex binary for OpenShift 4.6 is not published yet.

Next, we will download the the worker.ign file from the API server:

  1. $apiServer = c:\k\kubectl --kubeconfig c:\k\config get po -n openshift-kube-apiserver -l apiserver=true --no-headers -o custom-columns=":metadata.name" | select -first 1
  2. c:\k\kubectl --kubeconfig c:\k\config -n openshift-kube-apiserver exec $apiserver -- curl -ks https://localhost:22623/config/worker > c:\worker.ign
  3. ((Get-Content c:\worker.ign) -join "`n") + "`n" | Set-Content -NoNewline c:\worker.ign

Next, we run wmcb to configure the kubelet:

  1. c:\wmcb.exe initialize-kubelet --ignition-file worker.ign --kubelet-path c:\k\kubelet.exe

Install an OpenShift 4 cluster on Windows nodes - 图6note

The kubelet configuration installed by Windows Machine Config Bootstrapper includes --register-with-taints="os=Windows:NoSchedule" which will require Windows pods to tolerate that taint.

Next, we make a copy of the kubeconfig because wmcb.exe expects the kubeconfig to be the file c:\k\kubeconfig. Then we configure kubelet to use Calico CNI:

  1. cp c:\k\config c:\k\kubeconfig
  2. c:\wmcb.exe configure-cni --cni-dir c:\k\cni --cni-config c:\k\cni\config\10-calico.conf

Finally, clean up the additional files created on the Windows node:

  1. rm c:\k\kubeconfig,c:\wmcb.exe,c:\worker.ign

Exit the remote session to the Windows node and return to a shell to a Linux node.

We need to approve the CSR’s generated by the kubelet’s bootstrapping process. First, view the pending CSR’s:

  1. oc get csr

For example:

  1. $ oc get csr
  2. NAME AGE SIGNERNAME REQUESTOR CONDITION
  3. csr-55brx 4m32s kubernetes.io/kube-apiserver-client-kubelet system:admin Approved,Issued
  4. csr-bmnfd 4m30s kubernetes.io/kubelet-serving system:node:ip-10-0-45-102.us-west-2.compute.internal Pending
  5. csr-hwl89 5m1s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending

To approve the pending CSR’s:

  1. oc get csr -o name | xargs oc adm certificate approve

For example:

  1. $ oc get csr -o name | xargs oc adm certificate approve
  2. certificatesigningrequest.certificates.k8s.io/csr-55brx approved
  3. certificatesigningrequest.certificates.k8s.io/csr-bmnfd approved
  4. certificatesigningrequest.certificates.k8s.io/csr-hwl89 approved

Finally, wait a minute or so and get all nodes:

  1. $ oc get node -owide

If the Windows node registered itself successfully, it should appear in the list with a Ready status, ready to run Windows pods!

Next steps

Recommended - Security