Air-gapped Installation On Linux

The air-gapped installation is almost the same as the online installation except that you must create a local registry to host Docker images. This tutorial demonstrates how to install KubeSphere and Kubernetes in an air-gapped environment.

Step 1: Prepare Linux Hosts

Please see the requirements for hardware and operating system shown below. To get started with multi-node installation, you need to prepare at least three hosts according to the following requirements.

System requirements

SystemsMinimum Requirements (Each node)
Ubuntu 16.04, 18.04CPU: 2 Cores, Memory: 4 G, Disk Space: 100 G
Debian Buster, StretchCPU: 2 Cores, Memory: 4 G, Disk Space: 100 G
CentOS 7.xCPU: 2 Cores, Memory: 4 G, Disk Space: 100 G
Red Hat Enterprise Linux 7CPU: 2 Cores, Memory: 4 G, Disk Space: 100 G
SUSE Linux Enterprise Server 15/openSUSE Leap 15.2CPU: 2 Cores, Memory: 4 G, Disk Space: 100 G

Note

KubeKey uses /var/lib/docker as the default directory where all Docker related files, including images, are stored. It is recommended you add additional storage volumes with at least 100G mounted to /var/lib/docker and /mnt/registry respectively. See fdisk command for reference.

Node requirements

  • It’s recommended that your OS be clean (without any other software installed). Otherwise, there may be conflicts.
  • Ensure your disk of each node is at least 100G.
  • All nodes must be accessible through SSH.
  • Time synchronization for all nodes.
  • sudo/curl/openssl should be used in all nodes.
  • docker must be installed by yourself in an offline environment.

KubeKey can install Kubernetes and KubeSphere together. The dependency that needs to be installed may be different based on the Kubernetes version to be installed. You can refer to the list below to see if you need to install relevant dependencies on your node in advance.

DependencyKubernetes Version ≥ 1.18Kubernetes Version < 1.18
socatRequiredOptional but recommended
conntrackRequiredOptional but recommended
ebtablesOptional but recommendedOptional but recommended
ipsetOptional but recommendedOptional but recommended

Note

  • In an air-gapped environment, you can install these dependencies using a private package, a RPM package (for CentOS) or a Deb package (for Debian).
  • It is recommended you create an OS image file with all relevant dependencies installed in advance. In this way, you can use the image file directly for the installation of OS on each machine, improving deployment efficiency while not worrying about any dependency issues.

Network and DNS requirements

  • Make sure the DNS address in /etc/resolv.conf is available. Otherwise, it may cause some issues of DNS in clusters.
  • If your network configuration uses Firewall or Security Group, you must ensure infrastructure components can communicate with each other through specific ports. It’s recommended that you turn off the firewall. For more information, refer to Port Requirements.

Example machines

This example includes three hosts as below with the master node serving as the taskbox.

Host IPHost NameRole
192.168.0.2mastermaster, etcd
192.168.0.3node1worker
192.168.0.4node2worker

Step 2: Prepare a Private Image Registry

You can use Harbor or any other private image registries. This tutorial uses Docker registry as an example with self-signed certificates (If you have your own private image registry, you can skip this step).

Use self-signed certificates

  1. Generate your own certificate by executing the following commands:

    1. mkdir -p certs
    1. openssl req \
    2. -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
    3. -x509 -days 36500 -out certs/domain.crt

  2. Make sure you specify a domain name in the field Common Name when you are generating your own certificate. For instance, the field is set to dockerhub.kubekey.local in this example.

    self-signed-cert

Start the Docker registry

Run the following commands to start the Docker registry:

  1. docker run -d \
  2.   --restart=always \
  3.   --name registry \
  4.   -v "$(pwd)"/certs:/certs \
  5.   -v /mnt/registry:/var/lib/registry \
  6.   -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
  7.   -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  8.   -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  9.   -p 443:443 \
  10.   registry:2

Configure the registry

  1. Add an entry to /etc/hosts to map the hostname (i.e. the registry domain name; in this case, it is dockerhub.kubekey.local) to the private IP address of your machine as below.

    1. # docker registry
    2. 192.168.0.2 dockerhub.kubekey.local

  2. Execute the following commands to copy the certificate to a specified directory and make Docker trust it.

    1. mkdir -p  /etc/docker/certs.d/dockerhub.kubekey.local
    1. cp certs/domain.crt  /etc/docker/certs.d/dockerhub.kubekey.local/ca.crt

    Note

    The path of the certificate is related to the domain name. When you copy the path, use your actual domain name if it is different from the one set above.

  3. To verify whether the private registry is effective, you can copy an image to your local machine first, and use docker push and docker pull to test it.

Step 3: Download KubeKey

Similar to installing KubeSphere on Linux in an online environment, you also need to download KubeKey v1.1.0 first. Download the tar.gz file, and transfer it to your local machine which serves as the taskbox for installation. After you uncompress the file, execute the following command to make kk executable:

  1. chmod +x kk

Step 4: Prepare Installation Images

As you install KubeSphere and Kubernetes on Linux, you need to prepare an image package containing all the necessary images and download the Kubernetes binary file in advance.

  1. Download the image list file images-list.txt from a machine that has access to the Internet through the following command:

    1. curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/images-list.txt

    Note

    This file lists images under ##+modulename based on different modules. You can add your own images to this file following the same rule. To view the complete file, see Appendix.

  2. Download offline-installation-tool.sh.

    1. curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/offline-installation-tool.sh

  3. Make the .sh file executable.

    1. chmod +x offline-installation-tool.sh

  4. You can execute the command ./offline-installation-tool.sh -h to see how to use the script:

    1. [email protected]:/home/ubuntu# ./offline-installation-tool.sh -h
    2. Usage:
    4.   ./offline-installation-tool.sh [-l IMAGES-LIST] [-d IMAGES-DIR] [-r PRIVATE-REGISTRY] [-v KUBERNETES-VERSION ]
    6. Description:
    7.   -b                     : save kubernetes' binaries.
    8.   -d IMAGES-DIR          : the dir of files (tar.gz) which generated by `docker save`. default: ./kubesphere-images
    9.   -l IMAGES-LIST         : text file with list of images.
    10.   -r PRIVATE-REGISTRY    : target private registry:port.
    11.   -s                     : save model will be applied. Pull the images in the IMAGES-LIST and save images as a tar.gz file.
    12.   -v KUBERNETES-VERSION  : download kubernetes' binaries. default: v1.17.9
    13.   -h                     : usage message

  5. Download the Kubernetes binary file.

    1. ./offline-installation-tool.sh -b -v v1.17.9

    If you cannot access the object storage service of Google, run the following command instead to add the environment variable to change the source.

    1. export KKZONE=cn;./offline-installation-tool.sh -b -v v1.17.9

    Note

    • You can change the Kubernetes version downloaded based on your needs. Recommended Kubernetes versions for KubeSphere v3.1.0: v1.17.9, v1.18.8, v1.19.8 and v1.20.4. If you do not specify a Kubernetes version, KubeKey will install Kubernetes v1.19.8 by default. For more information about supported Kubernetes versions, see Support Matrix.

    • After you run the script, a folder kubekey is automatically created. Note that this file and kk must be placed in the same directory when you create the cluster later.

  6. Pull images in offline-installation-tool.sh.

    1. ./offline-installation-tool.sh -s -l images-list.txt -d ./kubesphere-images

    Note

    You can choose to pull images as needed. For example, you can delete ##k8s-images and related images under it in images-list.text if you already have a Kubernetes cluster.

Step 5: Push Images to Your Private Registry

Transfer your packaged image file to your local machine and execute the following command to push it to the registry.

  1. ./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r dockerhub.kubekey.local

Note

The domain name is dockerhub.kubekey.local in the command. Make sure you use your own registry address.

Step 6: Create a Cluster

In this tutorial, KubeSphere is installed on multiple nodes, so you need to specify a configuration file to add host information. Besides, for air-gapped installation, pay special attention to .spec.registry.privateRegistry, which must be set to your own registry address. See the complete YAML file below for more information.

Create an example configuration file

Execute the following command to generate an example configuration file for installation:

  1. ./kk create config [--with-kubernetes version] [--with-kubesphere version] [(-f | --file) path]

For example:

  1. ./kk create config --with-kubernetes v1.17.9 --with-kubesphere v3.1.0 -f config-sample.yaml

Note

  • Make sure the Kubernetes version is the one you downloaded.

  • If you do not add the flag --with-kubesphere in the command in this step, KubeSphere will not be deployed unless you install it using the addons field in the configuration file or add this flag again when you use ./kk create cluster later.

Edit the configuration file

Edit the generated configuration file config-sample.yaml. Here is an example for your reference:

Warning

For air-gapped installation, you must specify privateRegistry, which is dockerhub.kubekey.local in this example.

  1. apiVersion: kubekey.kubesphere.io/v1alpha1
  2. kind: Cluster
  3. metadata:
  4.   name: sample
  5. spec:
  6.   hosts:
  7.   - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, password: [email protected]}
  8.   - {name: node1, address: 192.168.0.3, internalAddress: 192.168.0.3, password: [email protected]}
  9.   - {name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, password: [email protected]}
  10.   roleGroups:
  11.     etcd:
  12.     - master
  13.     master:
  14.     - master
  15.     worker:
  16.     - master
  17.     - node1
  18.     - node2
  19.   controlPlaneEndpoint:
  20.     domain: lb.kubesphere.local
  21.     address: ""
  22.     port: 6443
  23.   kubernetes:
  24.     version: v1.17.9
  25.     imageRepo: kubesphere
  26.     clusterName: cluster.local
  27.   network:
  28.     plugin: calico
  29.     kubePodsCIDR: 10.233.64.0/18
  30.     kubeServiceCIDR: 10.233.0.0/18
  31.   registry:
  32.     registryMirrors: []
  33.     insecureRegistries: []
  34.     privateRegistry: dockerhub.kubekey.local  # Add the private image registry address here. 
  35.   addons: []
  38. ---
  39. apiVersion: installer.kubesphere.io/v1alpha1
  40. kind: ClusterConfiguration
  41. metadata:
  42.   name: ks-installer
  43.   namespace: kubesphere-system
  44.   labels:
  45.     version: v3.1.0
  46. spec:
  47.   persistence:
  48.     storageClass: ""
  49.   authentication:
  50.     jwtSecret: ""
  51.   zone: ""
  52.   local_registry: ""
  53.   etcd:
  54.     monitoring: false
  55.     endpointIps: localhost
  56.     port: 2379
  57.     tlsEnable: true
  58.   common:
  59.     redis:
  60.       enabled: false
  61.     redisVolumSize: 2Gi
  62.     openldap:
  63.       enabled: false
  64.     openldapVolumeSize: 2Gi
  65.     minioVolumeSize: 20Gi
  66.     monitoring:
  67.       endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
  68.     es:
  69.       elasticsearchMasterVolumeSize: 4Gi
  70.       elasticsearchDataVolumeSize: 20Gi
  71.       logMaxAge: 7
  72.       elkPrefix: logstash
  73.       basicAuth:
  74.         enabled: false
  75.         username: ""
  76.         password: ""
  77.       externalElasticsearchUrl: ""
  78.       externalElasticsearchPort: ""
  79.   console:
  80.     enableMultiLogin: true
  81.     port: 30880
  82.   alerting:
  83.     enabled: false
  84.     # thanosruler:
  85.     #   replicas: 1
  86.     #   resources: {}
  87.   auditing:
  88.     enabled: false
  89.   devops:
  90.     enabled: false
  91.     jenkinsMemoryLim: 2Gi
  92.     jenkinsMemoryReq: 1500Mi
  93.     jenkinsVolumeSize: 8Gi
  94.     jenkinsJavaOpts_Xms: 512m
  95.     jenkinsJavaOpts_Xmx: 512m
  96.     jenkinsJavaOpts_MaxRAM: 2g
  97.   events:
  98.     enabled: false
  99.     ruler:
  100.       enabled: true
  101.       replicas: 2
  102.   logging:
  103.     enabled: false
  104.     logsidecar:
  105.       enabled: true
  106.       replicas: 2
  107.   metrics_server:
  108.     enabled: false
  109.   monitoring:
  110.     storageClass: ""
  111.     prometheusMemoryRequest: 400Mi
  112.     prometheusVolumeSize: 20Gi
  113.   multicluster:
  114.     clusterRole: none
  115.   network:
  116.     networkpolicy:
  117.       enabled: false
  118.     ippool:
  119.       type: none
  120.     topology:
  121.       type: none
  122.   notification:
  123.     enabled: false
  124.   openpitrix:
  125.     store:
  126.       enabled: false
  127.   servicemesh:
  128.     enabled: false
  129.   kubeedge:
  130.     enabled: false
  131.     cloudCore:
  132.       nodeSelector: {"node-role.kubernetes.io/worker": ""}
  133.       tolerations: []
  134.       cloudhubPort: "10000"
  135.       cloudhubQuicPort: "10001"
  136.       cloudhubHttpsPort: "10002"
  137.       cloudstreamPort: "10003"
  138.       tunnelPort: "10004"
  139.       cloudHub:
  140.         advertiseAddress:
  141.           - ""
  142.         nodeLimit: "100"
  143.       service:
  144.         cloudhubNodePort: "30000"
  145.         cloudhubQuicNodePort: "30001"
  146.         cloudhubHttpsNodePort: "30002"
  147.         cloudstreamNodePort: "30003"
  148.         tunnelNodePort: "30004"
  149.     edgeWatcher:
  150.       nodeSelector: {"node-role.kubernetes.io/worker": ""}
  151.       tolerations: []
  152.       edgeWatcherAgent:
  153.         nodeSelector: {"node-role.kubernetes.io/worker": ""}
  154.         tolerations: []

Info

For more information about these parameters, see Multi-node Installation and Kubernetes Cluster Configuration. To enable pluggable components in config-sample.yaml, refer to Enable Pluggle Components for more details.

Step 7: Start Installation

You can execute the following command after you make sure that all steps above are completed.

  1. ./kk create cluster -f config-sample.yaml

Warning

After you transfer the executable file kk and the folder kubekey that contains the Kubernetes binary file to the taskbox machine for installation, they must be placed in the same directory before you execute the command above.

Step 8: Verify Installation

When the installation finishes, you can see the content as follows:

  1. #####################################################
  2. ###              Welcome to KubeSphere!           ###
  3. #####################################################
  5. Console: http://192.168.0.2:30880
  6. Account: admin
  7. Password: [email protected]
  9. NOTES：
  10.   1. After logging into the console, please check the
  11.      monitoring status of service components in
  12.      the "Cluster Management". If any service is not
  13.      ready, please wait patiently until all components
  14.      are ready.
  15.   2. Please modify the default password after login.
  17. #####################################################
  18. https://kubesphere.io             20xx-xx-xx xx:xx:xx
  19. #####################################################

Now, you will be able to access the web console of KubeSphere through http://{IP}:30880 with the default account and password admin/[[email protected]](https://kubesphere.io/cdn-cgi/l/email-protection).

Note

To access the console, make sure port 30880 is opened in your security group.

kubesphere-login

Appendix

Image list of KubeSphere v3.1.0

  1. ##k8s-images
  2. kubesphere/kube-apiserver:v1.20.4
  3. kubesphere/kube-scheduler:v1.20.4
  4. kubesphere/kube-proxy:v1.20.4
  5. kubesphere/kube-controller-manager:v1.20.4
  6. kubesphere/kube-apiserver:v1.19.8
  7. kubesphere/kube-scheduler:v1.19.8
  8. kubesphere/kube-proxy:v1.19.8
  9. kubesphere/kube-controller-manager:v1.19.8
  10. kubesphere/kube-apiserver:v1.18.6
  11. kubesphere/kube-scheduler:v1.18.6
  12. kubesphere/kube-proxy:v1.18.6
  13. kubesphere/kube-controller-manager:v1.18.6
  14. kubesphere/kube-apiserver:v1.17.9
  15. kubesphere/kube-scheduler:v1.17.9
  16. kubesphere/kube-proxy:v1.17.9
  17. kubesphere/kube-controller-manager:v1.17.9
  18. kubesphere/pause:3.1
  19. kubesphere/pause:3.2
  20. kubesphere/etcd:v3.4.13
  21. calico/cni:v3.16.3
  22. calico/kube-controllers:v3.16.3
  23. calico/node:v3.16.3
  24. calico/pod2daemon-flexvol:v3.16.3
  25. coredns/coredns:1.6.9
  26. kubesphere/k8s-dns-node-cache:1.15.12
  27. openebs/provisioner-localpv:2.3.0
  28. openebs/linux-utils:2.3.0
  29. kubesphere/nfs-client-provisioner:v3.1.0-k8s1.11
  30. ##csi-images
  31. csiplugin/csi-neonsan:v1.2.0
  32. csiplugin/csi-neonsan-ubuntu:v1.2.0
  33. csiplugin/csi-neonsan-centos:v1.2.0
  34. csiplugin/csi-provisioner:v1.5.0
  35. csiplugin/csi-attacher:v2.1.1
  36. csiplugin/csi-resizer:v0.4.0
  37. csiplugin/csi-snapshotter:v2.0.1
  38. csiplugin/csi-node-driver-registrar:v1.2.0
  39. csiplugin/csi-qingcloud:v1.2.0
  40. ##kubesphere-images
  41. kubesphere/ks-apiserver:v3.1.0
  42. kubesphere/ks-console:v3.1.0
  43. kubesphere/ks-controller-manager:v3.1.0
  44. kubesphere/ks-installer:v3.1.0
  45. kubesphere/kubectl:v1.17.0
  46. kubesphere/kubectl:v1.18.0
  47. kubesphere/kubectl:v1.19.0
  48. kubesphere/kubectl:v1.20.0
  49. redis:5.0.5-alpine
  50. alpine:3.10.4
  51. haproxy:2.0.4
  52. nginx:1.14-alpine
  53. minio/minio:RELEASE.2019-08-07T01-59-21Z
  54. minio/mc:RELEASE.2019-08-07T23-14-43Z
  55. mirrorgooglecontainers/defaultbackend-amd64:1.4
  56. kubesphere/nginx-ingress-controller:v0.35.0
  57. osixia/openldap:1.3.0
  58. csiplugin/snapshot-controller:v2.0.1
  59. kubesphere/kubefed:v0.7.0
  60. kubesphere/tower:v0.2.0
  61. kubesphere/prometheus-config-reloader:v0.42.1
  62. kubesphere/prometheus-operator:v0.42.1
  63. prom/alertmanager:v0.21.0
  64. prom/prometheus:v2.26.0
  65. prom/node-exporter:v0.18.1
  66. kubesphere/ks-alerting-migration:v3.1.0
  67. jimmidyson/configmap-reload:v0.3.0
  68. kubesphere/notification-manager-operator:v1.0.0
  69. kubesphere/notification-manager:v1.0.0
  70. kubesphere/metrics-server:v0.4.2
  71. kubesphere/kube-rbac-proxy:v0.8.0
  72. kubesphere/kube-state-metrics:v1.9.7
  73. openebs/provisioner-localpv:2.3.0
  74. thanosio/thanos:v0.18.0
  75. grafana/grafana:7.4.3
  76. ##kubesphere-logging-images
  77. kubesphere/elasticsearch-oss:6.7.0-1
  78. kubesphere/elasticsearch-curator:v5.7.6
  79. kubesphere/fluentbit-operator:v0.5.0
  80. kubesphere/fluentbit-operator:migrator
  81. kubesphere/fluent-bit:v1.6.9
  82. elastic/filebeat:6.7.0
  83. kubesphere/kube-auditing-operator:v0.1.2
  84. kubesphere/kube-auditing-webhook:v0.1.2
  85. kubesphere/kube-events-exporter:v0.1.0
  86. kubesphere/kube-events-operator:v0.1.0
  87. kubesphere/kube-events-ruler:v0.2.0
  88. kubesphere/log-sidecar-injector:1.1
  89. docker:19.03
  90. ##istio-images
  91. istio/pilot:1.6.10
  92. istio/proxyv2:1.6.10
  93. jaegertracing/jaeger-agent:1.17
  94. jaegertracing/jaeger-collector:1.17
  95. jaegertracing/jaeger-es-index-cleaner:1.17
  96. jaegertracing/jaeger-operator:1.17.1
  97. jaegertracing/jaeger-query:1.17
  98. kubesphere/kiali:v1.26.1
  99. kubesphere/kiali-operator:v1.26.1
  100. ##kubesphere-devops-images
  101. kubesphere/ks-jenkins:2.249.1
  102. jenkins/jnlp-slave:3.27-1
  103. kubesphere/s2ioperator:v3.1.0
  104. kubesphere/s2irun:v2.1.1
  105. kubesphere/builder-base:v2.1.0
  106. kubesphere/builder-nodejs:v2.1.0
  107. kubesphere/builder-maven:v2.1.0
  108. kubesphere/builder-go:v2.1.0
  109. kubesphere/s2i-binary:v2.1.0
  110. kubesphere/tomcat85-java11-centos7:v2.1.0
  111. kubesphere/tomcat85-java11-runtime:v2.1.0
  112. kubesphere/tomcat85-java8-centos7:v2.1.0
  113. kubesphere/tomcat85-java8-runtime:v2.1.0
  114. kubesphere/java-11-centos7:v2.1.0
  115. kubesphere/java-8-centos7:v2.1.0
  116. kubesphere/java-8-runtime:v2.1.0
  117. kubesphere/java-11-runtime:v2.1.0
  118. kubesphere/nodejs-8-centos7:v2.1.0
  119. kubesphere/nodejs-6-centos7:v2.1.0
  120. kubesphere/nodejs-4-centos7:v2.1.0
  121. kubesphere/python-36-centos7:v2.1.0
  122. kubesphere/python-35-centos7:v2.1.0
  123. kubesphere/python-34-centos7:v2.1.0
  124. kubesphere/python-27-centos7:v2.1.0
  125. kubesphere/notification:flyway_v2.1.2
  126. kubesphere/notification:v2.1.2
  127. ##openpitrix-images
  128. kubesphere/openpitrix-jobs:v3.1.0
  129. ##weave-scope-images
  130. weaveworks/scope:1.13.0
  131. ##kubeedge-images
  132. kubeedge/cloudcore:v1.6.1
  133. kubesphere/edge-watcher:v0.1.0
  134. kubesphere/kube-rbac-proxy:v0.5.0
  135. kubesphere/edge-watcher-agent:v0.1.0
  136. ##example-images-images
  137. kubesphere/examples-bookinfo-productpage-v1:1.16.2
  138. kubesphere/examples-bookinfo-reviews-v1:1.16.2
  139. kubesphere/examples-bookinfo-reviews-v2:1.16.2
  140. kubesphere/examples-bookinfo-reviews-v3:1.16.2
  141. kubesphere/examples-bookinfo-details-v1:1.16.2
  142. kubesphere/examples-bookinfo-ratings-v1:1.16.3
  143. busybox:1.31.1
  144. joosthofman/wget:1.0
  145. kubesphere/netshoot:v1.0
  146. nginxdemos/hello:plain-text
  147. wordpress:4.8-apache
  148. mirrorgooglecontainers/hpa-example:latest
  149. java:openjdk-8-jre-alpine
  150. fluent/fluentd:v1.4.2-2.0
  151. perl:latest