Apache Pulsar Installation on IBM Kubernetes Cluster through Helm chart

IBM Cloud Services - 图1tip

This tutorial uses Apache Pulsar 2.9.3 as an example. If you want to upgrade Pulsar version, follow the instructions in Helm Upgrade Guide.

Deploying a Pulsar cluster on IBM cloud consists of the following steps:

  1. Create VM on IBM Cloud.
  2. Create Kubernete cluster on IBM Cloud.
  3. Prepare VM for connecting to Kubernetes cluster and deploy Pulsar Helm chart on Kubernetes cluster.
  4. Verify the deployment.

1. Create VM on IBM Cloud

  1. Go to IBM Cloud and login with your credentials.
  2. Search for Virtual Server.
  3. Select Virtual Server for Classic.

VM Creation Image 1

  1. Select the type of virtual server as selected “Public” in the image. Type the host name, quantity of the machine and billing method.

VM Creation Image 2

  1. Select location value according to your region. For example: In below image we have selected Chennai in Asia region.

VM Creation Image 3

  1. Select the profile of virtual machine.

VM Creation Image 4

  1. Select the operating system and version.

VM Creation Image 5

  1. Select network interface according to use.

VM Creation Image 6

  1. Select the security group.

VM Creation Image 7

  1. Leave rest of the things could be default. Click on the “Create” button.

VM Creation Image 8

  1. Check created VM in “Navigation Menu” -> “Resource list” Devices

VM Creation Image 9

VM Creation Image 10

  1. Check the detail of VM in overview

VM Creation Image 11

  1. Check the devices list, and click on the menu option on the same page.

VM Creation Image 12

2. Create Kubernetes Cluster on IBM

  1. Search for the Kubernetes services

K8S Creation Image 1

  1. Select the plan details to “Standard”. Please note “Free” plan has the limited resources, which can not fulfill the Pulsar cluster requirement.

K8S Creation Image 2

  1. Select the infrastructure type we are going with the classic and also select the Kubernetes version .

K8S Creation Image 3

  1. Select location and resource group. Select single single zone or multi zone as per your need.

K8S Creation Image 4

K8S Creation Image 5

  1. Select worker pool size and flavor (vCPU, Memory) of the worker.

K8S Creation Image 6

K8S Creation Image 7

  1. Set cluster name as you want.

K8S Creation Image 8 7. Leave rest of the things as we are selecting as default. You can disable below options. Click on create and wait for provisioning of the cluster.

K8S Creation Image 9

  1. After the cluster is successfully provisioned, connect to the cluster. In order to connect click on the “Action” button then click on “Connect via CLI”, it will give you commands, copy that and run in your VM so that we can communicate to cluster through VM. We have to configure VM for communicating to cluster in the next steps will describe how to configure VM for that.

K8S Creation Image 10

  1. Check the created cluster list by clicking on the clusters options.

K8S Creation Image 11

  1. It will show you the list of all the created clusters.

K8S Creation Image 12

3. Prepare VM for connecting to Kubernetes cluster and deploy Pulsar Helm chart on Kubernetes cluster.

Prerequisites

  1. Install IBM Cloud CLI and connect to Kubernetes master node.

  2. Install IBM Cloud CLI Plugins to connect to IKS (IBM Kubernetes services). This is a required step.

  3. Install kubectl 1.23 or later versions.

  4. Install Helm.

IBM Cloud Services - 图26note

Please install all the above things before running the below commands

  1. First take SSH of your VM with your private key. Run all the below commands one by one.
  1. $ ibmcloud login
  2. $ ibmcloud plugin list

Output Output of Command img IBMLogin

  1. For connection to the Kubernetes cluster, you will get a command in the Kubernetes cluster section (Kubernetes console) as below.
  1. $ibmcloud ks cluster config --cluster ccql163t064kpvg5gg10

IBM Cloud Services - 图28note

This step is showing during creation of Kubernetes Cluster.

K8S Creation Image 11

  1. After running the second and third commands shown in the above images, you will get an output as below. You do not need to run the first command because you’ve already logged in to IBM Cloud.

Output Output of Command IKSConnect

Now we are able to run commands of kubectl.

4. Verify the deployment

Make sure all the pods of Pulsar are running. Get the service URL and broker URL for publishing and consuming the messages.

Installation of Pulsar helm in minikube cluster this document is for minikube cluster we are taking reference from that.

  1. Add Pulsar chart repo
  1. $ helm repo add apache https://pulsar.apache.org/charts
  2. $ helm repo update
  3. $ helm repo list

Output Output of Command Helmrepo

  1. Clone the Pulsar Helm chart repository, go inside the pulsar-helm-chart directory
  1. git clone https://github.com/apache/pulsar-helm-chart
  2. cd pulsar-helm-chart
  1. Run the scriptprepare_helm_release.shto create secrets required for installing the Apache Pulsar Helm chart. The username pulsar and password pulsar are used for logging into the Grafana dashboard and Pulsar Manager.

IBM Cloud Services - 图32note

When running the script, you can use -n to specify the Kubernetes namespace where the Pulsar Helm chart is installed,-kto define the Pulsar Helm release name, and -c to create the Kubernetes namespace. For more information about the script, run ./scripts/pulsar/prepare_helm_release.sh --help.

  1. ./scripts/pulsar/prepare_helm_release.sh -n default -k asia -c

Output Output of Command HelmOutPut

  1. Use the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.
  1. helm install --values examples/values-minikube.yaml --set initialize=true asia apache/pulsar

IBM Cloud Services - 图34note

You need to specify --set initialize=true when installing Pulsar the first time. This command installs and starts Apache Pulsar.

Output Output of Command HelmInstall

  1. Check the status of all pods.
  1. kubectl get pods

If all pods start up successfully, you can see that STATUS is changed to Running or Completed.

Output

Output of Command PodStatus

  1. Check the status of all services.
  1. kubectl get services

Output

Output of Command Services

The output shows both services URL and broker URL. Proxy external IPs are the ports changed just now:

  • Service URL port is 80.
  • Broker URL port is 6650. This is default. If you execute all the commands successfully, you can use the Pulsar client to connect to clusters and produce and consume messages through proxy external IPs.