集群部署

jetlinks以及所需中间件的集群部署

注意：

仅专业版以上支持。
本文档以172.16.4.13、172.16.4.12、172.16.4.11三台服务器来搭建集群。

集群架构图

集群架构图

redis集群安装配置

这里准备在一台服务器上搭建一个三主三从的redis集群。

  1. 安装redis 先下载redis集群部署 - 图2,本文档以redis5.0.5为例。
    下载完成后进行解压、编译、安装。

注意：

redis是由C语言编写的，它的运行需要C环境，所以编译前需安装 gcc。

  1. 新建一个cluster文件夹，用来存放集群节点目录。
    分别在172.16.4.11上创建7000、7003、7001、7004、7002、7005六文件夹，，这些节点分别使用7000、7003、7001、7004、7002、7005端口，以7000节点为例配置如下：
  1. port 7000
  2. bind 172.16.4.11
  3. cluster-enabled yes
  4. cluster-config-file nodes_7000.conf
  5. cluster-node-timeout 5000
  6. appendonly no
  7. daemonize  yes

其他节点只需修改端口和文件名，依次按此进行配置即可，配置完成后启动节点。

注意：

建议appendonly在从节点开启，主节点关闭。

  1. 主从节点分配
  1. redis-cli --cluster create 172.16.4.11:7000 172.16.4.11:7001 172.16.4.11:7002 172.16.4.11:7003 172.16.4.11:7004 172.16.4.11:7005 --cluster -replicas 1

注意：

其中-replicas 1表示每个主节点1个从节点

  1. 查看集群节点信息
  1. ops@jetlinks-server-3:~/data/redis-5.0.5$ redis-cli -h 172.16.4.11  -p 7000 cluster nodes
  2. 562558e5afa0575d1059c47b9531a37cd75a9190 172.16.4.11:7003@17003 slave 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 0 1594368796612 4 connected
  3. bd974dbdd2f8447b27375c832c4c9c99328f4487 172.16.4.11:7005@17005 slave 3ab893f4cdcfaa52254a8cece2b54b561de29990 0 1594368797514 6 connected
  4. 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 172.16.4.11:7002@17002 master - 0 1594368797615 3 connected 10923-16383
  5. 3ab893f4cdcfaa52254a8cece2b54b561de29990 172.16.4.11:7001@17001 master - 0 1594368797000 2 connected 5461-10922
  6. 238585a079196b0ab15ebb47ac681d42c083cdaf 172.16.4.11:7000@17000 myself,master - 0 1594368796000 1 connected 0-5460
  7. d1ae6c623694c5f1fafc64d7abbeac4a9926ff49 172.16.4.11:7004@17004 slave 238585a079196b0ab15ebb47ac681d42c083cdaf 0 1594368796000 5 connected

redis集群搭建请参考redis官方文档集群部署 - 图3

elasticsearch集群安装配置

  1. 分别在三台服务器上安装elasticsearch 前往官网下载集群部署 - 图4。
    下载完成后安装es。
  2. 配置elasticsearch.yml文件
  1. ops@jetlinks-server-3:~/elasticsearch/config$ sudo vi elasticsearch.yml 
  2. # ======================== Elasticsearch Configuration =========================
  3. #
  4. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  5. #       Before you set out to tweak and tune the configuration, make sure you
  6. #       understand what are you trying to accomplish and the consequences.
  7. #
  8. # The primary way of configuring a node is via this file. This template lists
  9. # the most important settings you may want to configure for a production cluster.
  10. #
  11. # Please consult the documentation for further information on configuration options:
  12. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  13. #
  14. # ---------------------------------- Cluster -----------------------------------
  15. #
  16. # Use a descriptive name for your cluster:
  17. #
  18. cluster.name: es-cluster
  19. #
  20. # ------------------------------------ Node ------------------------------------
  21. #
  22. # Use a descriptive name for the node:
  23. #
  24. node.name: node-3
  25. #
  26. # Add custom attributes to the node:
  27. #
  28. #node.attr.rack: r1
  29. #
  30. # ----------------------------------- Paths ------------------------------------
  31. #
  32. # Path to directory where to store the data (separate multiple locations by comma):
  33. #
  34. #path.data: /path/to/data
  35. #
  36. # Path to log files:
  37. #
  38. #path.logs: /path/to/logs
  39. #
  40. # ----------------------------------- Memory -----------------------------------
  41. #
  42. # Lock the memory on startup:
  43. #
  44. #bootstrap.memory_lock: true
  45. #
  46. # Make sure that the heap size is set to about half the memory available
  47. # on the system and that the owner of the process is allowed to use this
  48. # limit.
  49. #
  50. # Elasticsearch performs poorly when the system is swapping the memory.
  51. #
  52. # ---------------------------------- Network -----------------------------------
  53. #
  54. # Set the bind address to a specific IP (IPv4 or IPv6):
  55. #
  56. network.host: 172.16.4.11
  58. transport.tcp.port: 9300
  59. #
  60. # Set a custom port for HTTP:
  61. #
  62. http.port: 9200
  63. #
  64. # For more information, consult the network module documentation.
  65. #
  66. # --------------------------------- Discovery ----------------------------------
  67. #
  68. # Pass an initial list of hosts to perform discovery when new node is started:
  69. # The default list of hosts is ["127.0.0.1", "[::1]"]
  70. #
  71. discovery.zen.ping.unicast.hosts: ["172.16.4.13:9300", "172.16.4.12:19300","172.16.4.11:9300"]
  72. #
  73. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
  74. #
  75. discovery.zen.minimum_master_nodes: 1
  76. #
  77. # For more information, consult the zen discovery module documentation.
  78. #
  79. # ---------------------------------- Gateway -----------------------------------
  80. #
  81. # Block initial recovery after a full cluster restart until N nodes are started:
  82. #
  83. #gateway.recover_after_nodes: 3
  84. #
  85. # For more information, consult the gateway module documentation.
  86. #
  87. # ---------------------------------- Various -----------------------------------
  88. #
  89. # Require explicit names when deleting indices:
  90. #
  91. #action.destructive_requires_name: true

三台服务器不同的配置：

  1. #172.16.4.11
  2. node.name: node-3  
  3. network.host:172.16.4.11
  1. #172.16.4.12
  2. node.name: node-2  
  3. network.host:172.16.4.12
  1. #172.16.4.13
  2. node.name: node-1  
  3. network.host:172.16.4.13

3 在三台服务器上分别启动es

  1. ./bin/elasticsearch

数据库启动

本文档案例使用docker启动postgresql，可参考docker-compose.yml集群部署 - 图5文件中的postgresql配置。

jetlinks集群安装配置

1.修改application.yml,并将jetlinks打成jar包

application.yml文件：

  1. server:
  2.   port: 8844
  3.   max-http-header-size: 64KB
  4. spring:
  5.   profiles:
  6.     active: dev
  7.   application:
  8.     name: jetlinks-platform
  9.   jackson:
  10.     date-format: yyyy-MM-dd HH:mm:ss
  11.     time-zone: Asia/Shanghai
  12.     serialization:
  13.       WRITE_DATES_AS_TIMESTAMPS: true
  14.     default-property-inclusion: non_null
  15.   resources:
  16.     static-locations: file:./static,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/, classpath:/public/
  17.   redis:   
  18.     # redis集群配置
  19.     cluster: 
  20.       nodes:
  21.         - 172.16.4.11:7000
  22.         - 172.16.4.11:7001
  23.         - 172.16.4.11:7002
  24.         - 172.16.4.11:7003
  25.         - 172.16.4.11:7004
  26.         - 172.16.4.11:7005
  27.       max-redirects: 3
  28.     lettuce:
  29.       pool:
  30.         max-active: 1024
  31.     timeout: 20s
  32.     serializer: jdk # 设置fst时,redis key使用string序列化,value使用 fst序列化.
  33.     database: 0
  34.   r2dbc:
  35.     url: r2dbc:postgresql://172.16.4.12:5432/jetlinks    #数据库地址 本文档数据库为docker启动
  36.     username: postgres
  37.     password: jetlinks
  38.     pool:
  39.       max-size: 32
  40.   codec:
  41.     max-in-memory-size: 50MB
  42.   rabbitmq: # 开启了device.message.writer.rabbitmq.enabled=true时生效
  43.     host: localhost
  44.     port: 5672
  45.     username: admin
  46.     password: jetlinks
  47.   kafka:    # 开启了device.message.writer.kafka.enabled=true时生效
  48.     consumer:
  49.       client-id: ${spring.application.name}-consumer:${server.port}
  50.       group-id: ${spring.application.name}
  51.       max-poll-records: 1000
  52.     producer:
  53.       client-id: ${spring.application.name}-producer:${server.port}
  54.       acks: 1
  55.       retries: 3
  56.     bootstrap-servers: ["127.0.0.1:9092"]
  57.   reactor:
  58.     debug-agent:
  59.       enabled: true # 开启调试代理,在打印异常时将会生成调用踪栈信息
  60. easyorm:
  61.   default-schema: public # 数据库默认的schema
  62.   dialect: postgres #数据库方言
  63. elasticsearch:
  64.   client:
  65.     max-conn-total: 128
  66.     connect-timeout: 5000
  67.     socket-timeout: 5000
  68.     connection-request-timeout: 8000
  69.     hosts:                      # es集群配置
  70.       - http://172.16.4.13:9200
  71.       - http://172.16.4.12:19200
  72.       - http://172.16.4.11:9200
  73.   index:
  74.     default-strategy: time-by-month #默认es的索引按月进行分表
  75.     settings:
  76.       number-of-shards: 1 # es 分片数量
  77.       number-of-replicas: 0 # 副本数量
  78. device:
  79.   message:
  80.     writer:
  81.       time-series:
  82.         enabled: true # 直接写出设备消息数据到时序数据库
  83.       kafka:
  84.         enabled: false  # 推送设备消息到kafka
  85.         consumer: true  # 从kafka订阅消息并写入到时序数据库
  86.         topic-name: device.message
  87.       rabbitmq:
  88.         enabled: false  # 推送设备消息到rabbitMQ
  89.         consumer: true  # 从rabbitMQ订阅消息并写入到时序数据库
  90.         thread-size: 4 # 消费线程数
  91.         auto-ack: true  # 自定应答,为true可能导致数据丢失,但是性能最高。
  92.         topic-name: device.message  # exchange名称
  93.         ignore-message-types:
  94.         include-message-types:
  95. hsweb:
  96.   cors:
  97.     enable: true
  98.     configs:
  99.       - path: /**
  100.         allowed-headers: "*"
  101.         allowed-methods: ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]
  102.         allowed-origins: ["*"]
  103.         allow-credentials: true
  104.         max-age: 1800
  105.   dict:
  106.     enum-packages: org.jetlinks
  107.   file:
  108.     upload:
  109.       static-file-path: ./static/upload
  110.       static-location: http://demo2.jetlinks.org:8080/upload   # 此地址为服务器域名，将通过nginx代理到固定服务器上
  111.   webflux:
  112.     response-wrapper: enabled #开启响应包装器(将返回值包装为ResponseMessage)
  113.   #  auth:   #默认的用户配置
  114.   #    users:
  115.   #      admin:
  116.   #        username: admin
  117.   #        password: admin
  118.   #        name: 超级管理员
  119.   authorize:
  120.     auto-parse: true
  121.   cache:
  122.     type: redis
  123.     redis:
  124.       local-cache-type: guava
  125. jetlinks:
  126.   server-id: ${spring.application.name}:${server.port} #设备服务网关服务ID,不同服务请设置不同的ID
  127.   logging:
  128.     system:
  129.       context:
  130.         server: ${spring.application.name}
  131.   protocol:
  132.     spi:
  133.       enabled: false # 为true时开启自动加载通过依赖引入的协议包
  134. rule:
  135.   engine:
  136.     server-id: ${jetlinks.server-id}
  137.     server-name: ${spring.application.name}
  138. logging:
  139.   level:
  140.     org.jetlinks: info
  141.     rule.engine: info
  142.     org.hswebframework: debug
  143.     org.springframework.transaction: debug
  144.     org.springframework.data.r2dbc.connectionfactory: debug
  145.     io.micrometer: warn
  146.     org.hswebframework.expands: error
  147.     system: warn
  148.     org.jetlinks.rule.engine: info
  149.     org.jetlinks.gateway: info
  150.     org.springframework: warn
  151.     org.apache.kafka: warn
  152.     org.jetlinks.pro.device.message.writer: debug
  153.   #    org.elasticsearch: error
  154.   config: classpath:logback-spring.xml
  155. vertx:
  156.   max-event-loop-execute-time-unit: seconds
  157.   max-event-loop-execute-time: 30
  158.   max-worker-execute-time-unit: seconds
  159.   max-worker-execute-time: 30
  160.   prefer-native-transport: true
  162. micrometer:
  163.   time-series:
  164.     tags:
  165.       server: ${spring.application.name}
  166.     metrics:
  167.       default:
  168.         step: 30s

注意：

在多台服务器上启动jetlinks，配置中的jetlinks.server-id必须互不相同。

在项目根目录执行：

  1. mvn clean package -DskipTests
  1. 分别在三台服务器上启动jetlinks 文档以脚本方式启动，脚本如下：
  1. #!/bin/bash
  2. nohup java -jar -Dspring.application.name=jetlinks-cluster-test-3   jetlinks-standalone.jar >jetlinks-pro.log 2>&1 &

注意：

三台服务启动的jetlinks.server-id应该互不相同，在application.yml中jetlinks.server-id引用了spring.application.name，所以此处的name应不相同。

nginx配置

通过nginx来代理前后端。

  1. 安装nginx
    前往官网下载集群部署 - 图6 下载完成后解压、编译、安装。
  2. 修改配置文件

配置文件如下：

  1. upstream iotserver {
  2.   server 172.16.4.11:8844;
  3.   server 172.16.4.12:8844;
  4.   server 172.16.4.13:8844;
  5. }
  7. upstream webserver {
  8.   server 172.16.4.11:9000;
  9. }
  11. upstream fileserver {
  12.   server 172.16.4.11:8844; #此处指定文件上传到该服务器上
  13. }
  15. server {
  17.   listen       8080;
  18.   server_name  demo2.jetlinks.cn;
  20.   location ^~/upload/ {
  21.     proxy_pass http://fileserver;
  22.     proxy_set_header Host $host:$server_port;
  23.     proxy_set_header X-Real-IP  $remote_addr;
  24.     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  25.   }
  27.   location ^~/jetlinks/file/static {
  28.     proxy_pass http://fileserver/file/static;
  29.     proxy_set_header X-Forwarded-Proto $scheme;
  30.     proxy_set_header Host $host:$server_port;
  31.     proxy_set_header X-Real-IP  $remote_addr;
  32.     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  33.     proxy_send_timeout      30m;
  34.     proxy_read_timeout      30m;
  35.     client_max_body_size    100m;
  36.   }
  38.   location ^~/jetlinks/ {
  39.     proxy_pass http://iotserver/;
  40.     proxy_set_header X-Forwarded-Proto $scheme;
  41.     proxy_set_header Host $host:$server_port;
  42.     proxy_set_header X-Real-IP  $remote_addr;
  43.     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  44.     proxy_http_version 1.1;
  45.     proxy_set_header Upgrade $http_upgrade;
  46.     proxy_set_header Connection "upgrade";
  47.     proxy_connect_timeout   1;
  48.     proxy_buffering off;
  49.     chunked_transfer_encoding off;
  50.     proxy_cache off;
  51.     proxy_send_timeout      30m;
  52.     proxy_read_timeout      30m;
  53.     client_max_body_size    100m;
  54.   }
  56.   location / {
  57.     proxy_pass http://webserver/;
  58.   }
  59. }
  1. 启动
  1. systemctl status nginx.service

tcp负载均衡配置方式

用于代理设备连接。此处使用nginx作为演示,也可以使用LVS,haProxy等方式。

配置文件：

  1. load_module /usr/lib/nginx/modules/ngx_stream_module.so;
  2. user  root;
  3. worker_processes  1;
  5. error_log  /etc/nginx/log/error.log warn;
  6. pid        /var/run/nginx.pid;
  9. events {
  10.     worker_connections  1024;
  11. }
  13. stream {
  15.    upstream tcp-test {
  16.        hash $remote_addr consistent;
  17.        server 172.16.4.13:1889 max_fails=3 fail_timeout=10s;
  18.        #server 172.16.4.12:1889 max_fails=3 fail_timeout=10s;
  19.        server 172.16.4.11:1889 max_fails=3 fail_timeout=10s;
  20.    }
  22.    server {
  24.        listen 1884;
  26.        proxy_pass tcp-test;
  28.        proxy_connect_timeout 30s;
  30.        proxy_timeout 30s;
  31.    }
  32. }

注意：

此配置监听的MQTT的1884端口，代理到服务器上的1889。如需代理自定义端口需保持代理端口与jetlinks平台中网络组件开启的端口一致。 如此处平台中MQTT服务网络组件设置的端口应为1889。