Deploy KubeSphere on Air-Gapped Kubernetes Cluster

Edit

KubeSphere+K8s

This document explains how to install KubeSphere on an air-gapped Kubernetes cluster.

Note: first of all, please read the prerequisites.

Image Registry

You need an image registry set up for the cluster to use. If not yet, please see Configure Harbor for the instructions.

Step 1: Download Images

We provide two ways to download KubeSphere images, the image list and the zipped image package.

Docker uses /var/lib/docker as the default directory where all Docker related files including the images are stored. Before loading the images into Docker, please check if the disk is big enough. Usually we recommend you to add additional storage to the disk mounted at /var/lib/docker since it requires at least 100G more storage. You can use the fdisk command to expand the disk.

Option 1: Image List

KubeSphere provides the list of images for you to download one by one to your target machine.

Note:

  • These image files are about 30G, please make sure there are enough space in your machine.
  • You can use the script download-image-list.sh to pull them in batch.

Image lists (Click here to expand)

  1. ks_minimal_images:
  2.   - kubesphere/ks-console:v2.1.1
  3.   - kubesphere/kubectl:v1.0.0
  4.   - kubesphere/ks-account:v2.1.1
  5.   - kubesphere/ks-devops:flyway-v2.1.0
  6.   - kubesphere/ks-apigateway:v2.1.1
  7.   - kubesphere/ks-apiserver:v2.1.1
  8.   - kubesphere/ks-controller-manager:v2.1.1
  9.   - kubesphere/cloud-controller-manager:v1.4.0
  10.   - kubesphere/ks-installer:v2.1.1
  11.   - quay.azk8s.cn/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
  12.   - mirrorgooglecontainers/defaultbackend-amd64:1.4
  13.   - gcr.azk8s.cn/google_containers/metrics-server-amd64:v0.3.1
  14.   - kubesphere/configmap-reload:v0.3.0
  15.   - kubesphere/prometheus:v2.5.0
  16.   - kubesphere/prometheus-config-reloader:v0.34.0
  17.   - kubesphere/prometheus-operator:v0.34.0
  18.   - kubesphere/kube-rbac-proxy:v0.4.1
  19.   - kubesphere/kube-state-metrics:v1.7.2
  20.   - kubesphere/node-exporter:ks-v0.16.0
  21.   - kubesphere/addon-resizer:1.8.4
  22.   - kubesphere/k8s-prometheus-adapter-amd64:v0.4.1
  23.   - grafana/grafana:5.2.4
  24.   - redis:5.0.5-alpine
  25.   - haproxy:2.0.4
  26.   - alpine:3.10.4
  27.   - quay.azk8s.cn/coreos/etcd:v3.2.18
  28.   - mysql:8.0.11
  29.   - nginx:1.14-alpine
  30.   - postgres:9.6.8
  31.   - osixia/openldap:1.3.0
  32.   - minio/minio:RELEASE.2019-08-07T01-59-21Z
  33.   - minio/mc:RELEASE.2019-08-07T23-14-43Z
  35. ks_notification_images:
  36.   - kubesphere/notification:v2.1.0
  37.   - kubesphere/notification:flyway_v2.1.0
  38.   - kubesphere/alerting-dbinit:v2.1.0
  39.   - kubesphere/alerting:v2.1.0
  40.   - kubesphere/alert_adapter:v2.1.0
  42. openpitrix_images:
  43.   - openpitrix/release-app:v0.4.3
  44.   - openpitrix/openpitrix:flyway-v0.4.8
  45.   - openpitrix/openpitrix:v0.4.8
  46.   - openpitrix/runtime-provider-kubernetes:v0.1.3
  48. ks_devops_images:
  49.   - kubesphere/jenkins-uc:v2.1.1
  50.   - jenkins/jenkins:2.176.2
  51.   - jenkins/jnlp-slave:3.27-1
  52.   - kubesphere/builder-base:v2.1.0
  53.   - kubesphere/builder-nodejs:v2.1.0
  54.   - kubesphere/builder-maven:v2.1.0
  55.   - kubesphere/builder-go:v2.1.0
  56.   - sonarqube:7.4-community
  57.   - kubesphere/s2ioperator:v2.1.1
  58.   - kubesphere/s2irun:v2.1.1
  59.   - kubesphere/s2i-binary:v2.1.0
  60.   - kubesphere/tomcat85-java11-centos7:v2.1.0
  61.   - kubesphere/tomcat85-java11-runtime:v2.1.0
  62.   - kubesphere/tomcat85-java8-centos7:v2.1.0
  63.   - kubesphere/tomcat85-java8-runtime:v2.1.0
  64.   - kubesphere/java-11-centos7:v2.1.0
  65.   - kubesphere/java-8-centos7:v2.1.0
  66.   - kubesphere/java-8-runtime:v2.1.0
  67.   - kubesphere/java-11-runtime:v2.1.0
  68.   - kubesphere/nodejs-8-centos7:v2.1.0
  69.   - kubesphere/nodejs-6-centos7:v2.1.0
  70.   - kubesphere/nodejs-4-centos7:v2.1.0
  71.   - kubesphere/python-36-centos7:v2.1.0
  72.   - kubesphere/python-35-centos7:v2.1.0
  73.   - kubesphere/python-34-centos7:v2.1.0
  74.   - kubesphere/python-27-centos7:v2.1.0
  76. ks_logging_images:
  77.   - kubesphere/elasticsearch-curator:v5.7.6
  78.   - kubesphere/elasticsearch-oss:6.7.0-1
  79.   - kubesphere/fluent-bit:v1.3.2-reload
  80.   - docker.elastic.co/kibana/kibana-oss:6.7.0
  81.   - dduportal/bats:0.4.0
  82.   - docker:19.03
  83.   - kubesphere/fluentbit-operator:v0.1.0
  84.   - kubesphere/fluent-bit:v1.3.5-reload
  85.   - kubesphere/configmap-reload:v0.0.1
  86.   - kubesphere/log-sidecar-injector:1.0
  88. istio_images:
  89.   - istio/kubectl:1.3.3
  90.   - istio/proxy_init:1.3.3
  91.   - istio/proxyv2:1.3.3
  92.   - istio/citadel:1.3.3
  93.   - istio/pilot:1.3.3
  94.   - istio/mixer:1.3.3
  95.   - istio/galley:1.3.3
  96.   - istio/sidecar_injector:1.3.3
  97.   - istio/node-agent-k8s:1.3.3
  98.   - jaegertracing/jaeger-operator:1.13.1
  99.   - jaegertracing/jaeger-agent:1.13
  100.   - jaegertracing/jaeger-collector:1.13
  101.   - jaegertracing/jaeger-query:1.13
  102.   - kubesphere/examples-bookinfo-productpage-v1:1.13.0
  103.   - kubesphere/examples-bookinfo-reviews-v1:1.13.0
  104.   - kubesphere/examples-bookinfo-reviews-v2:1.13.0
  105.   - kubesphere/examples-bookinfo-reviews-v3:1.13.0
  106.   - kubesphere/examples-bookinfo-details-v1:1.13.0
  107.   - kubesphere/examples-bookinfo-ratings-v1:1.13.0
  109. example_images:
  110.   - busybox:1.31.1
  111.   - joosthofman/wget:1.0
  112.   - kubesphere/netshoot:v1.0
  113.   - nginxdemos/hello:plain-text
  114.   - wordpress:4.8-apache
  115.   - mirrorgooglecontainers/hpa-example:latest
  116.   - java:openjdk-8-jre-alpine
  117.   - fluent/fluentd:v1.4.2-2.0
  118.   - perl:latest

Option 2: Zipped Image Package

Execute the following command to download the 7.2G image package.

  1. curl -L https://kubesphere.io/download/images/latest > kubesphere-all-images-v2.1.1.tar.gz \
  2. && tar -zxf kubesphere-all-images-v2.1.1.tar.gz && cd kubesphere-images-v2.1.1

List all images in this folder.

  1. $ tree
  2. .
  3. tree
  4. .
  5. ├── example_images.tar
  6. ├── istio_images.tar
  7. ├── ks_devops_images.tar
  8. ├── ks_logging_images.tar
  9. ├── ks_minimal_images.tar
  10. ├── ks_notification_images.tar
  11. └── openpitrix_images.tar

Load the image packages into docker. If you only want to set up the default minimal installation, just load the ks_minimal_images. An installation with all optional components and examples requires to load all images as follows.

  1. docker load < ks_minimal_images.tar
  2. docker load < openpitrix_images.tar
  3. docker load < ks_logging_images.tar
  4. docker load < ks_devops_images.tar
  5. docker load < istio_images.tar
  6. docker load < ks_notification_images.tar
  7. docker load < example_images.tar

Step 2: Push Images to Harbor

Clone the project ks-installer to your local, then enter the scripts folder.

  1. git clone https://github.com/kubesphere/ks-installer.git
  2. cd ks-installer/scripts

Since we have to push a batch of images to different projects of Harbor, we can use the following script to create the corresponding projects.

  1. vi create_project_harbor.sh

Please replace the image registry information with yours in the script.

  1. ···
  2. url="http://192.168.0.31"
  3. user="admin"
  4. passwd="Harbor12345"

Execute the script to create the corresponding projects in Harbor.

  1. ./create_project_harbor.sh

Execute the following script to push the images that we have loaded above to the Harbor registry in batch.

  1. ./push-image-list.sh 192.168.0.31:80

Step 3: Installation

Back to root folder of this repository, and add a new field of Harbor address. Please edit the kubesphere-minimal.yaml or kubesphere-complete-setup.yaml according to your needs.

  1. ···
  2.     alerting:
  3.       enabled: true
  5.     local_registry: 192.168.0.31:80   # Add a new field of Harbor address to this line.
  7. kind: ConfigMap
  8. ···

Install KubeSphere using kubectl.

  • If there are 1 Core and 2 GB RAM available in your cluster, use the command below to start a default minimal installation only:
  1. kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-minimal.yaml

You can refer to enable pluggable components to install optional components at your will.

  • If there are 8 Cores and 16 GB RAM available in your cluster, use the command below to install a complete KubeSphere, i.e. with all components enabled:
  1. kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-complete-setup.yaml

Step 4: Verify Installation

Verify the real-time logs use the command as follows.

  1. $ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
  3. #####################################################
  4. ###              Welcome to KubeSphere!           ###
  5. #####################################################
  6. Console: http://10.128.0.34:30880
  7. Account: admin
  8. Password: P@88w0rd
  10. NOTE：Please modify the default password after login.
  11. #####################################################

When you see the outputs above, congratulation! You can access KubeSphere console in your browser now.

Step 5: Enable Pluggable Components (Optional)

If you start with a default minimal installation, execute the following command to open the configmap in order to enable pluggable components. Make sure your cluster has enough CPU and memory, and the corresponding images need to be loaded. Please see Configuration Table for the requirements and Enable Pluggable Components for the instructions.

  1. kubectl edit cm -n kubesphere-system ks-installer