Deploy - Air Gap Installation - 《The Longhorn v1.0.0 Documentation》

Requirements:
- Note:
Using manifest file.
Using Helm Chart
Using Rancher App
Troubleshooting

Requirements:

Deploy Longhorn Components images to your own registry.
Deploy Kuberntes CSI driver components images to your own registry.

Note:

CSI driver components images names and tags can be found here.
We recommend using a short registry URL due to a Kubernetes limitation on the length of pod metadata labels. For more information, refer to this section

Using manifest file.

Get Longhorn Deployment manifest file

wget https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
Create Longhorn namespace

kubectl create namespace longhorn-system

If private registry require authentication, Create docker-registry secret in longhorn-system namespace:

kubectl -n longhorn-system create secret docker-registry <SECRET_NAME> --docker-server=<REGISTRY_URL> --docker-username=<REGISTRY_USER> --docker-password=<REGISTRY_PASSWORD>

Add your secret name to longhorn-default-setting ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-default-setting
  namespace: longhorn-system
data:
  default-setting.yaml: |-
    backup-target:
    backup-target-credential-secret:
    create-default-disk-labeled-nodes:
    default-data-path:
    replica-soft-anti-affinity:
    storage-over-provisioning-percentage:
    storage-minimal-available-percentage:
    upgrade-checker:
    default-replica-count:
    guaranteed-engine-cpu:
    default-longhorn-static-storage-class:
    backupstore-poll-interval:
    taint-toleration:
    registry-secret:  <SECRET_NAME>

Add your secret name SECRET_NAME to imagePullSecrets.name in the following resources

longhorn-driver-deployer Deployment
longhorn-manager DaemonSet
longhorn-ui Deployment

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: longhorn-ui
  name: longhorn-ui
  namespace: longhorn-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: longhorn-ui
  template:
    metadata:
      labels:
        app: longhorn-ui
    spec:
      containers:
      - name: longhorn-ui
        image: longhornio/longhorn-ui:v0.8.0
        ports:
        - containerPort: 8000
        env:
          - name: LONGHORN_MANAGER_IP
            value: "http://longhorn-backend:9500"
      imagePullSecrets:
      - name: <SECRET_NAME>                          ## Add SECRET_NAME here
      serviceAccountName: longhorn-service-account

Apply the following modifications to the manifest file
- Modify Kubernetes CSI driver components environment variables in longhorn-driver-deployer Deployment point to your private registry images
  - CSI_ATTACHER_IMAGE
  - CSI_PROVISIONER_IMAGE
  - CSI_NODE_DRIVER_REGISTRAR_IMAGE
  - CSI_RESIZER_IMAGE
```
- name: CSI_ATTACHER_IMAGE
  value: <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
- name: CSI_PROVISIONER_IMAGE
  value: <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
- name: CSI_NODE_DRIVER_REGISTRAR_IMAGE
  value: <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
- name: CSI_RESIZER_IMAGE
  value: <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>
```
- Modify Longhorn images to point to your private registry images
  - longhornio/longhorn-manager
    
    image: <REGISTRY_URL>/longhorn-manager:<LONGHORN_MANAGER_IMAGE_TAG>
  - longhornio/longhorn-engine
    
    image: <REGISTRY_URL>/longhorn-engine:<LONGHORN_ENGINE_IMAGE_TAG>
  - longhornio/longhorn-instance-manager
    
    image: <REGISTRY_URL>/longhorn-instance-manager:<LONGHORN_INSTANCE_MANAGER_IMAGE_TAG>
  - longhornio/longhorn-ui
    
    image: <REGISTRY_URL>/longhorn-ui:<LONGHORN_UI_IMAGE_TAG>

Example:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: longhorn-ui
  name: longhorn-ui
  namespace: longhorn-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: longhorn-ui
  template:
    metadata:
      labels:
        app: longhorn-ui
    spec:
      containers:
      - name: longhorn-ui
        image: <REGISTRY_URL>/longhorn-ui:<LONGHORN_UI_IMAGE_TAG>   ## Add image name and tag here
        ports:
        - containerPort: 8000
        env:
          - name: LONGHORN_MANAGER_IP
            value: "http://longhorn-backend:9500"
      imagePullSecrets:
      - name: <SECRET_NAME>
      serviceAccountName: longhorn-service-account
```

Deploy Longhorn using modified manifest file kubectl apply -f longhorn.yaml

Using Helm Chart

Clone longhorn repo

git clone https://github.com/longhorn/longhorn.git
In chart/values.yaml

Specify Longhorn images:

image:
  longhorn:
    engine: <REGISTRY_URL>/longhorn-engine
    engineTag: <LONGHORN_ENGINE_IMAGE_TAG>
    manager: <REGISTRY_URL>/longhorn-manager
    managerTag: LONGHORN_MANAGER_IMAGE_TAG<>
    ui: <REGISTRY_URL>/longhorn-ui
    uiTag: <LONGHORN_UI_IMAGE_TAG>
    instanceManager: <REGISTRY_URL>/longhorn-instance-manager
    instanceManagerTag: <LONGHORN_INSTANCE_MANAGER_IMAGE_TAG>

Specify CSI Driver components images:

csi:
  attacherImage: <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
  provisionerImage: <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
  driverRegistrarImage: <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
  resizerImage: <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>

Specify registry secret Name, URL, and Credentials:

defaultSettings:
  registrySecret: <SECRET_NAME>
privateRegistry: 
    registryUrl: <REGISTRY_URL>
    registryUser: <REGISTRY_USER>
    registryPasswd: <REGISTRY_PASSWORD>

Install Longhorn
- Helm2
  
  helm install ./chart --name longhorn --namespace longhorn-system
- Helm3
  
  kubectl create namespace longhorn-system
  
  helm install longhorn ./chart --namespace longhorn-system

Using Rancher App

In Longhorn Images Settings section specify
- Longhorn Manager Image Name e.g. <REGISTRY_URL>/longhorn-manager
- Longhorn Manager Image Tag
- Longhorn Engine Image Name e.g. <REGISTRY_URL>/longhorn-engine
- Longhorn Engine Image Tag
- Longhorn UI Image Name e.g. <REGISTRY_URL>/longhorn-ui
- Longhorn UI Image Tag
- Longhorn Instance Manager Image Name e.g. <REGISTRY_URL>/longhorn-instance-manager
- Longhorn Instance Manager Image Tag
In Longhorn CSI Driver Setting section specify
- Longhorn CSI Attacher Image e.g. <REGISTRY_URL>/csi-attacher:<CSI_ATTACHER_IMAGE_TAG>
- Longhorn CSI Provisioner Image e.g. <REGISTRY_URL>/csi-provisioner:<CSI_PROVISIONER_IMAGE_TAG>
- Longhorn CSI Driver Registrar Image e.g. <REGISTRY_URL>/csi-node-driver-registrar:<CSI_NODE_DRIVER_REGISTRAR_IMAGE_TAG>
- Longhorn CSI Driver Resizer Image e.g. <REGISTRY_URL>/csi-resizer:<CSI_RESIZER_IMAGE_TAG>
In Longhorn Default Settings secton specify
- Private registry secret
In Private Registry Settings secton specify
- Private registry URL
- Private registry user
- Private registry password

Troubleshooting

For Helm/Rancher installation, if user forgot to submit a secret to authenticate to private registry, longhorn-manager DaemonSet will fail to create.

Create the Kubernetes secret

kubectl -n longhorn-system create secret docker-registry <SECRET_NAME> --docker-server=<REGISTRY_URL> --docker-username=<REGISTRY_USER> --docker-password=<REGISTRY_PASSWORD>

Create registry-secret setting object manually.

apiVersion: longhorn.io/v1beta1
kind: Setting
metadata:
  name: registry-secret
  namespace: longhorn-system
value: <SECRET_NAME>

kubectl apply -f registry-secret.yml

Delete Longhorn and re-install it again.
- Helm2
  
  helm uninstall ./chart --name longhorn --namespace longhorn-system
  
  helm install ./chart --name longhorn --namespace longhorn-system
- Helm3
  
  helm uninstall longhorn ./chart --namespace longhorn-system
  
  helm install longhorn ./chart --namespace longhorn-system

longhorn-driver-deployer error: Node is not support mount propagation

If longhorn-instance-manager image name is more than 63 characters long, it will fail to deploy, and longhorn-driver-deployer pod will be in CrashLoopBackOff.

Checking Longhorn driver deployer logs will report the following:

time="2020-03-13T22:49:22Z" level=warning msg="Got an error when checking MountPropagation with node status, Node XXX is not support mount propagation"
time="2020-03-13T22:49:22Z" level=fatal msg="Error deploying driver: CSI cannot be deployed because MountPropagation is not set: Node <NODE_NAME> is not support mount propagation"

Issue can be conformed by checking Longhorn manager log, you should be able to see the following logs:

“Dropping Longhorn node longhorn-system/NODE_NAME out of the queue: fail to sync node for longhorn-system/NODE_NAME: InstanceManager.longhorn.io “instance-manager-e-605e9473” is invalid: metadata.labels: Invalid value: “PRIVATE_REGISTRY_URL-PREFIX-longhorn-instance-manager-v1_20200301”: must be no more than 63 characters”

Longhorn instance manager: metadata.labels must be no more than 63 characters

Using a long registry URL may cause Longhorn installation error

Longhorn manager would report errors in the log when this happened:

"instance-manager-e-xxxxxxxx" is invalid: metadata.labels: Invalid value: "<PRIVATE_REGISTRY_URL>-longhornio-longhorn-instance-manager-v1_20200301": must be no more than 63 characters

Longhorn instance manager pods have labels with key longhorn.io/instance-manager-image and value REGISTRY_URL-USER-IMAGE_NAME-TAG

e.g

metadata:
  labels:
    longhorn.io/component: instance-manager
    longhorn.io/instance-manager-image: <PRIVATE_REGISTRY_URL>-longhornio-longhorn-instance-manager-v1_20200301
    longhorn.io/instance-manager-type: engine
    longhorn.io/node: <NODE_NAME>
  name: instance-manager-e-XXXXXXXX

it’s known Kubernetes limitaion that label value should be no more than 63 characters here

Recommendation:

It’s highly recommended not to manipulate image tags, especially instance manager image tags such as v1_20200301, because we intentionally use the date to avoid associating it with a Longhorn version.

e.g

Longhorn components images
- longhorn-instance-manager: hub.example.com/lh/ins-mgr:v1_20200301
- longhorn-manager: hub.example.com/lh/mgr:v0.8.1
- longhorn-engine: hub.example.com/lh/eng:v0.8.1
- longhorn-ui: hub.examples.com/lh/ui:v0.8.1
Kubernetes CSI images
- CSI Attacher: hub.example.com/csi/attacher:v2.0.0
- CSI Provisioner: hub.example.com/csi/provisioner:v1.4.0
- CSI Node Driver Registrar: hub.example.com/csi/node-driver-reg:v1.2.0
- CSI Resizer: hub.example.com/csi/resizer:v0.3.0