Creating a NATS Super Cluster in Digital Ocean with Helm

Let’s create a super cluster using NATS Gateways. First let’s create 3 different clusters in NYC, Amsterdam, and San Francisco:

  1. doctl kubernetes cluster create nats-k8s-nyc1 --count 3 --region nyc1
  2. doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
  3. doctl kubernetes cluster create nats-k8s-ams3 --count 3 --region ams3

Next, open up the firewall across the 3 regions to be able to access the client, leafnode and gateways ports:

  1. for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
  2. doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
  3. doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7422,address:0.0.0.0/0
  4. doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7522,address:0.0.0.0/0
  5. done

For this setup, we will create a super cluster using the external IPs from the nodes of the 3 clusters. For a production type of setup, it is recommended to use a DNS entry and an A record for each one of the servers.

  1. for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  2. echo "name: $ctx"
  3. for externalIP in `kubectl --context $ctx get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'`; do
  4. echo "- nats://$externalIP:7522";
  5. done
  6. echo
  7. done

The Helm definition would look as follows for the 3 clusters:

  1. # super-cluster.yaml
  2. nats:
  3. externalAccess: true
  4. logging:
  5. debug: false
  6. trace: false
  7. cluster:
  8. enabled: true
  9. gateway:
  10. enabled: true
  11. # NOTE: defined via --set gateway.name="$ctx"
  12. # name: $ctx
  13. gateways:
  14. - name: do-ams3-nats-k8s-ams3
  15. urls:
  16. - nats://142.93.251.181:7522
  17. - nats://161.35.12.245:7522
  18. - nats://161.35.2.153:7522
  19. - name: do-nyc1-nats-k8s-nyc1
  20. urls:
  21. - nats://142.93.251.181:7522
  22. - nats://161.35.12.245:7522
  23. - nats://161.35.2.153:7522
  24. - name: do-sfo2-nats-k8s-sfo2
  25. urls:
  26. - nats://142.93.251.181:7522
  27. - nats://161.35.12.245:7522
  28. - nats://161.35.2.153:7522
  29. natsbox:
  30. enabled: true

Let’s deploy the super cluster with Helm using the name of cluster as the name of the gateway:

  1. for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  2. helm --kube-context $ctx install nats nats/nats -f super-cluster.yaml --set gateway.name=$ctx
  3. done

That’s it! It should now be possible to send some messages across regions:

  1. # Start subscription in Amsterdam
  2. nats-box:~# kubectl --context do-ams3-nats-k8s-ams3 exec -it nats-box -- /bin/sh -l
  3. nats-box:~# nats sub -s nats hello
  4. # Send messages from San Francisco region
  5. nats-box:~# kubectl --context do-sfo2-nats-k8s-sfo2 exec -it nats-box -- /bin/sh -l
  6. nats-box:~# nats pub -s nats hello 'Hello World!'
  7. # From outside of k8s can use the external IPs
  8. nats sub -s 142.93.251.181 hello
  9. nats pub -s 161.35.2.153 hello 'Hello World!'