Deploy apiserver-network-proxy (ANP) For Pull mode

Purpose

For a member cluster that joins Karmada in the pull mode, you need to provide a method to connect the network between the Karmada control plane and the member cluster, so that karmada-aggregated-apiserver can access this member cluster.

Deploying ANP to achieve this is one of the methods. This document describes how to deploy ANP for Karmada.

Environment

Karmada can be deployed using the kind tool.

You can directly use hack/local-up-karmada.sh to deploy Karmada.

Actions

Step 1: Download code

To facilitate demonstration, the code is modified based on ANP v0.0.24 to support access to the front server through HTTP. Here is the code repository address: https://github.com/mrlihanbo/apiserver-network-proxy/tree/v0.0.24/dev.

  1. git clone -b v0.0.24/dev https://github.com/mrlihanbo/apiserver-network-proxy.git
  2. cd apiserver-network-proxy/

Step 2: Build images

Build the proxy-server and proxy-agent images.

  1. docker build . --build-arg ARCH=amd64 -f artifacts/images/agent-build.Dockerfile -t swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24
  3. docker build . --build-arg ARCH=amd64 -f artifacts/images/server-build.Dockerfile -t swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24

Step 3: Generate certificates

Run the command to check the IP address of karmada-host:

  1. docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane

Run the make certs command to generate certificates and specify PROXY_SERVER_IP as the IP address obtained in the preceding command.

  1. make certs PROXY_SERVER_IP=x.x.x.x

The certificates are generated in the certs folder.

Step 4: Deploy proxy-server

Save the proxy-server.yaml file in the root directory of the ANP code repository.

unfold me to see the yaml

  1. # proxy-server.yaml
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: proxy-server
  7.   namespace: karmada-system
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: proxy-server
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: proxy-server
  17.     spec:
  18.       containers:
  19.       - command:
  20.         - /proxy-server
  21.         args:
  22.           - --health-port=8092
  23.           - --cluster-ca-cert=/var/certs/server/cluster-ca-cert.crt
  24.           - --cluster-cert=/var/certs/server/cluster-cert.crt 
  25.           - --cluster-key=/var/certs/server/cluster-key.key
  26.           - --mode=http-connect 
  27.           - --proxy-strategies=destHost 
  28.           - --server-ca-cert=/var/certs/server/server-ca-cert.crt
  29.           - --server-cert=/var/certs/server/server-cert.crt 
  30.           - --server-key=/var/certs/server/server-key.key
  31.         image: swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24
  32.         imagePullPolicy: IfNotPresent
  33.         livenessProbe:
  34.           failureThreshold: 3
  35.           httpGet:
  36.             path: /healthz
  37.             port: 8092
  38.             scheme: HTTP
  39.           initialDelaySeconds: 10
  40.           periodSeconds: 10
  41.           successThreshold: 1
  42.           timeoutSeconds: 60
  43.         name: proxy-server
  44.         volumeMounts:
  45.         - mountPath: /var/certs/server
  46.           name: cert
  47.       restartPolicy: Always
  48.       hostNetwork: true
  49.       volumes:
  50.       - name: cert
  51.         secret:
  52.           secretName: proxy-server-cert
  53. ---
  54. apiVersion: v1
  55. kind: Secret
  56. metadata:
  57.   name: proxy-server-cert
  58.   namespace: karmada-system
  59. type: Opaque
  60. data:
  61.   server-ca-cert.crt: |
  62.     {{server_ca_cert}}
  63.   server-cert.crt: |
  64.     {{server_cert}}
  65.   server-key.key: |
  66.     {{server_key}}
  67.   cluster-ca-cert.crt: |
  68.     {{cluster_ca_cert}}
  69.   cluster-cert.crt: |
  70.     {{cluster_cert}}
  71.   cluster-key.key: |
  72.     {{cluster_key}}

Save the replace-proxy-server.sh file in the root directory of the ANP code repository.

unfold me to see the shell

  1. #!/bin/bash
  3. cert_yaml=proxy-server.yaml
  5. SERVER_CA_CERT=$(cat certs/frontend/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  6. sed -i'' -e "s/{{server_ca_cert}}/${SERVER_CA_CERT}/g" ${cert_yaml}
  8. SERVER_CERT=$(cat certs/frontend/issued/proxy-frontend.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  9. sed -i'' -e "s/{{server_cert}}/${SERVER_CERT}/g" ${cert_yaml}
  11. SERVER_KEY=$(cat certs/frontend/private/proxy-frontend.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  12. sed -i'' -e "s/{{server_key}}/${SERVER_KEY}/g" ${cert_yaml}
  14. CLUSTER_CA_CERT=$(cat certs/agent/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  15. sed -i'' -e "s/{{cluster_ca_cert}}/${CLUSTER_CA_CERT}/g" ${cert_yaml}
  17. CLUSTER_CERT=$(cat certs/agent/issued/proxy-frontend.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  18. sed -i'' -e "s/{{cluster_cert}}/${CLUSTER_CERT}/g" ${cert_yaml}
  21. CLUSTER_KEY=$(cat certs/agent/private/proxy-frontend.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  22. sed -i'' -e "s/{{cluster_key}}/${CLUSTER_KEY}/g" ${cert_yaml}

Run the following commands to run the script:

  1. chmod +x replace-proxy-server.sh
  2. bash replace-proxy-server.sh

Deploy the proxy-server on the karmada-host:

  1. kind load docker-image swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24 --name karmada-host
  2. export KUBECONFIG=/root/.kube/karmada.config
  3. kubectl --context=karmada-host apply -f proxy-server.yaml

Step 5: Deploy proxy-agent

Save the proxy-agent.yaml file in the root directory of the ANP code repository.

unfold me to see the yaml

  1. # proxy-agent.yaml
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   labels:
  7.     app: proxy-agent
  8.   name: proxy-agent
  9.   namespace: karmada-system
  10. spec:
  11.   replicas: 1
  12.   selector:
  13.     matchLabels:
  14.       app: proxy-agent
  15.   template:
  16.     metadata:
  17.       labels:
  18.         app: proxy-agent
  19.     spec:
  20.       containers:
  21.         - command:
  22.             - /proxy-agent
  23.           args:
  24.             - '--ca-cert=/var/certs/agent/ca.crt'
  25.             - '--agent-cert=/var/certs/agent/proxy-agent.crt'
  26.             - '--agent-key=/var/certs/agent/proxy-agent.key'
  27.             - '--proxy-server-host={{proxy_server_addr}}'
  28.             - '--proxy-server-port=8091'
  29.             - '--agent-identifiers=host={{identifiers}}'
  30.           image: swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24
  31.           imagePullPolicy: IfNotPresent
  32.           name: proxy-agent
  33.           livenessProbe:
  34.             httpGet:
  35.               scheme: HTTP
  36.               port: 8093
  37.               path: /healthz
  38.             initialDelaySeconds: 15
  39.             timeoutSeconds: 60
  40.           volumeMounts:
  41.             - mountPath: /var/certs/agent
  42.               name: cert
  43.       volumes:
  44.         - name: cert
  45.           secret:
  46.             secretName: proxy-agent-cert
  47. ---
  48. apiVersion: v1
  49. kind: Secret
  50. metadata:
  51.   name: proxy-agent-cert
  52.   namespace: karmada-system
  53. type: Opaque
  54. data:
  55.   ca.crt: |
  56.     {{proxy_agent_ca_crt}}
  57.   proxy-agent.crt: |
  58.     {{proxy_agent_crt}}
  59.   proxy-agent.key: |
  60.     {{proxy_agent_key}}

Save the replace-proxy-agent.sh file in the root directory of the ANP code repository.

unfold me to see the shell

  1. #!/bin/bash
  3. cert_yaml=proxy-agent.yaml
  5. karmada_control_plane_addr=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane)
  6. member3_cluster_addr=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' member3-control-plane)
  7. sed -i'' -e "s/{{proxy_server_addr}}/${karmada_control_plane_addr}/g" ${cert_yaml}
  8. sed -i'' -e "s/{{identifiers}}/${member3_cluster_addr}/g" ${cert_yaml}
  10. PROXY_AGENT_CA_CRT=$(cat certs/agent/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  11. sed -i'' -e "s/{{proxy_agent_ca_crt}}/${PROXY_AGENT_CA_CRT}/g" ${cert_yaml}
  13. PROXY_AGENT_CRT=$(cat certs/agent/issued/proxy-agent.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  14. sed -i'' -e "s/{{proxy_agent_crt}}/${PROXY_AGENT_CRT}/g" ${cert_yaml}
  16. PROXY_AGENT_KEY=$(cat certs/agent/private/proxy-agent.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  17. sed -i'' -e "s/{{proxy_agent_key}}/${PROXY_AGENT_KEY}/g" ${cert_yaml}

Run the following commands to run the script:

  1. chmod +x replace-proxy-agent.sh
  2. bash replace-proxy-agent.sh

Deploy the proxy-agent in the pull mode for a member cluster (in this example, the member3 cluster is in the pull mode.):

  1. kind load docker-image swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24 --name member3
  2. kubectl --kubeconfig=/root/.kube/members.config --context=member3 apply -f proxy-agent.yaml

The ANP deployment is completed now.

Step 6: Add command flags for the karmada-agent deployment

After deploying the ANP deployment, you need to add extra command flags --cluster-api-endpoint and --proxy-server-address for the karmada-agent deployment in the member3 cluster.

Where --cluster-api-endpoint is the APIEndpoint of the cluster. You can obtain it from the KubeConfig file of the member3 cluster.

Where --proxy-server-address is the address of the proxy server that is used to proxy the cluster. In current case, you can set --proxy-server-address to http://<karmada_control_plane_addr>:8088. Get karmada_control_plane_addr value through the following command:

  1. docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane

Set port 8088 by modifying the code in ANP: https://github.com/mrlihanbo/apiserver-network-proxy/blob/v0.0.24/dev/cmd/server/app/server.go#L267. You can also modify it to a different value.