Deploying HA CloudCore

The HA of CloudCore(deployed in k8s cluster)

Note: There are several ways to achieve the HA of cloudcore, for example, ingress, keepalived etc. Here we adopt the keepalived. The HA of cloudcore according to ingress will be achieved later.

Determine the virtual IP of CloudCore

Determine a VIP that the CloudCore service exposed to the edge nodes. Here we recommend keepalived to do that. You had better directly schedule pods to specific number of nodes by nodeSelector when using keepalived. And you have to install keepalived in each of nodes where CloudCore runs. The configuration of keepalived is shown in the end. Here suppose the VIP is 10.10.102.242.

The use of nodeSelector is as follow:

  1. kubectl label nodes [nodename] [key]=[value] # label the nodes where the cloudcore will run

modify the term of nodeselector:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: cloudcore
  5. spec:
  6. template:
  7. spec:
  8. nodeSelector: # configure the nodeSelector here!
  9. [key]: [value]

Create k8s resources

The manifests and scripts in github.com/kubeedge/kubeedge/build/cloud/ha will be used, so place these files to somewhere you can kubectl with (You have to make some modifications to manifests/scrips to suit your environment.)

First, ensure your k8s cluster can pull cloudcore image. If the image not exist. We can make one, and push to your registry.

  1. cd $GOPATH/src/github.com/kubeedge/kubeedge
  2. make image WHAT=cloudcore

We create k8s resources from the manifests in name order. Before creating, check the content of each manifest to make sure it meets your environment.

Note: Now the follow manifests don’t support kubectl logs command yet. If need, you have to make more configuration manually.

02-ha-configmap.yaml

Configure the VIP address of CloudCore which is exposed to the edge nodes in the advertiseAddress, which will be added to SANs in cert of CloudCore. For example:

  1. modules:
  2. cloudHub:
  3. advertiseAddress:
  4. - 10.10.102.242

Note: If you want to reset the CloudCore, run this before creating k8s resources:

  1. kubectl delete namespace kubeedge

Then create k8s resources:

  1. cd build/cloud/ha
  2. for resource in $(ls *.yaml); do kubectl create -f $resource; done

keepalived

The keepalived configuration we recommend is as following. You can adjust it according to your needs.

keepalived.conf:

  • master:
  1. ! Configuration File for keepalived
  2. global_defs {
  3. router_id lb01
  4. vrrp_mcast_group4 224.0.0.19
  5. }
  6. # CloudCore
  7. vrrp_script CloudCore_check {
  8. script "/etc/keepalived/check_cloudcore.sh" # the script for health check
  9. interval 2
  10. weight 2
  11. fall 2
  12. rise 2
  13. }
  14. vrrp_instance CloudCore {
  15. state MASTER
  16. interface eth0 # based on your host
  17. virtual_router_id 167
  18. priority 100
  19. advert_int 1
  20. authentication {
  21. auth_type PASS
  22. auth_pass 1111
  23. }
  24. virtual_ipaddress {
  25. 10.10.102.242/24 # VIP
  26. }
  27. track_script {
  28. CloudCore_check
  29. }
  30. }
  • backup:
  1. ! Configuration File for keepalived
  2. global_defs {
  3. router_id lb02
  4. vrrp_mcast_group4 224.0.0.19
  5. }
  6. # CloudCore
  7. vrrp_script CloudCore_check {
  8. script "/etc/keepalived/check_cloudcore.sh" # the script for health check
  9. interval 2
  10. weight 2
  11. fall 2
  12. rise 2
  13. }
  14. vrrp_instance CloudCore {
  15. state BACKUP
  16. interface eth0 # based on your host
  17. virtual_router_id 167
  18. priority 99
  19. advert_int 1
  20. authentication {
  21. auth_type PASS
  22. auth_pass 1111
  23. }
  24. virtual_ipaddress {
  25. 10.10.102.242/24 # VIP
  26. }
  27. track_script {
  28. CloudCore_check
  29. }
  30. }

check_cloudcore.sh:

  1. #!/usr/bin/env bash
  2. http_code=`curl -k -o /dev/null -s -w %{http_code} https://127.0.0.1:10002/readyz`
  3. if [ $http_code == 200 ]; then
  4. exit 0
  5. else
  6. exit 1
  7. fi