集群部署

jetlinks以及所需中间件的集群部署

注意:

仅专业版以上支持。
本文档以172.16.4.13、172.16.4.12、172.16.4.11三台服务器来搭建集群。

集群架构图

集群架构图

redis集群安装配置

这里准备在一台服务器上搭建一个三主三从的redis集群。

  1. 安装redis 先下载redis集群部署 - 图2 (opens new window),本文档以redis5.0.5为例。
    下载完成后进行解压、编译、安装。

注意:

redis是由C语言编写的,它的运行需要C环境,所以编译前需安装 gcc。

  1. 新建一个cluster文件夹,用来存放集群节点目录。
    分别在172.16.4.11上创建7000、7003、7001、7004、7002、7005六文件夹,,这些节点分别使用7000、7003、7001、7004、7002、7005端口,以7000节点为例配置如下:
  1. port 7000
  2. bind 172.16.4.11
  3. cluster-enabled yes
  4. cluster-config-file nodes_7000.conf
  5. cluster-node-timeout 5000
  6. appendonly no
  7. daemonize yes

其他节点只需修改端口和文件名,依次按此进行配置即可,配置完成后启动节点。

注意:

建议appendonly在从节点开启,主节点关闭。

  1. 主从节点分配
  1. redis-cli --cluster create 172.16.4.11:7000 172.16.4.11:7001 172.16.4.11:7002 172.16.4.11:7003 172.16.4.11:7004 172.16.4.11:7005 --cluster -replicas 1

注意:

其中-replicas 1表示每个主节点1个从节点

  1. 查看集群节点信息
  1. ops@jetlinks-server-3:~/data/redis-5.0.5$ redis-cli -h 172.16.4.11 -p 7000 cluster nodes
  2. 562558e5afa0575d1059c47b9531a37cd75a9190 172.16.4.11:7003@17003 slave 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 0 1594368796612 4 connected
  3. bd974dbdd2f8447b27375c832c4c9c99328f4487 172.16.4.11:7005@17005 slave 3ab893f4cdcfaa52254a8cece2b54b561de29990 0 1594368797514 6 connected
  4. 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 172.16.4.11:7002@17002 master - 0 1594368797615 3 connected 10923-16383
  5. 3ab893f4cdcfaa52254a8cece2b54b561de29990 172.16.4.11:7001@17001 master - 0 1594368797000 2 connected 5461-10922
  6. 238585a079196b0ab15ebb47ac681d42c083cdaf 172.16.4.11:7000@17000 myself,master - 0 1594368796000 1 connected 0-5460
  7. d1ae6c623694c5f1fafc64d7abbeac4a9926ff49 172.16.4.11:7004@17004 slave 238585a079196b0ab15ebb47ac681d42c083cdaf 0 1594368796000 5 connected

redis集群搭建请参考redis官方文档集群部署 - 图3 (opens new window)

elasticsearch集群安装配置

  1. 分别在三台服务器上安装elasticsearch 前往官网下载集群部署 - 图4 (opens new window)
    下载完成后安装es。
  2. 配置elasticsearch.yml文件
  1. ops@jetlinks-server-3:~/elasticsearch/config$ sudo vi elasticsearch.yml
  2. # ======================== Elasticsearch Configuration =========================
  3. #
  4. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  5. # Before you set out to tweak and tune the configuration, make sure you
  6. # understand what are you trying to accomplish and the consequences.
  7. #
  8. # The primary way of configuring a node is via this file. This template lists
  9. # the most important settings you may want to configure for a production cluster.
  10. #
  11. # Please consult the documentation for further information on configuration options:
  12. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  13. #
  14. # ---------------------------------- Cluster -----------------------------------
  15. #
  16. # Use a descriptive name for your cluster:
  17. #
  18. cluster.name: es-cluster
  19. #
  20. # ------------------------------------ Node ------------------------------------
  21. #
  22. # Use a descriptive name for the node:
  23. #
  24. node.name: node-3
  25. #
  26. # Add custom attributes to the node:
  27. #
  28. #node.attr.rack: r1
  29. #
  30. # ----------------------------------- Paths ------------------------------------
  31. #
  32. # Path to directory where to store the data (separate multiple locations by comma):
  33. #
  34. #path.data: /path/to/data
  35. #
  36. # Path to log files:
  37. #
  38. #path.logs: /path/to/logs
  39. #
  40. # ----------------------------------- Memory -----------------------------------
  41. #
  42. # Lock the memory on startup:
  43. #
  44. #bootstrap.memory_lock: true
  45. #
  46. # Make sure that the heap size is set to about half the memory available
  47. # on the system and that the owner of the process is allowed to use this
  48. # limit.
  49. #
  50. # Elasticsearch performs poorly when the system is swapping the memory.
  51. #
  52. # ---------------------------------- Network -----------------------------------
  53. #
  54. # Set the bind address to a specific IP (IPv4 or IPv6):
  55. #
  56. network.host: 172.16.4.11
  57. transport.tcp.port: 9300
  58. #
  59. # Set a custom port for HTTP:
  60. #
  61. http.port: 9200
  62. #
  63. # For more information, consult the network module documentation.
  64. #
  65. # --------------------------------- Discovery ----------------------------------
  66. #
  67. # Pass an initial list of hosts to perform discovery when new node is started:
  68. # The default list of hosts is ["127.0.0.1", "[::1]"]
  69. #
  70. discovery.zen.ping.unicast.hosts: ["172.16.4.13:9300", "172.16.4.12:19300","172.16.4.11:9300"]
  71. #
  72. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
  73. #
  74. discovery.zen.minimum_master_nodes: 1
  75. #
  76. # For more information, consult the zen discovery module documentation.
  77. #
  78. # ---------------------------------- Gateway -----------------------------------
  79. #
  80. # Block initial recovery after a full cluster restart until N nodes are started:
  81. #
  82. #gateway.recover_after_nodes: 3
  83. #
  84. # For more information, consult the gateway module documentation.
  85. #
  86. # ---------------------------------- Various -----------------------------------
  87. #
  88. # Require explicit names when deleting indices:
  89. #
  90. #action.destructive_requires_name: true

三台服务器不同的配置:

  1. #172.16.4.11
  2. node.name: node-3
  3. network.host:172.16.4.11
  1. #172.16.4.12
  2. node.name: node-2
  3. network.host:172.16.4.12
  1. #172.16.4.13
  2. node.name: node-1
  3. network.host:172.16.4.13

3 在三台服务器上分别启动es

  1. ./bin/elasticsearch

数据库启动

本文档案例使用docker启动postgresql,可参考docker-compose.yml集群部署 - 图5 (opens new window)文件中的postgresql配置。

jetlinks集群安装配置

1.修改application.yml,并将jetlinks打成jar包

application.yml文件:

  1. server:
  2. port: 8844
  3. max-http-header-size: 64KB
  4. spring:
  5. profiles:
  6. active: dev
  7. application:
  8. name: jetlinks-platform
  9. jackson:
  10. date-format: yyyy-MM-dd HH:mm:ss
  11. time-zone: Asia/Shanghai
  12. serialization:
  13. WRITE_DATES_AS_TIMESTAMPS: true
  14. default-property-inclusion: non_null
  15. resources:
  16. static-locations: file:./static,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/, classpath:/public/
  17. redis:
  18. # redis集群配置
  19. cluster:
  20. nodes:
  21. - 172.16.4.11:7000
  22. - 172.16.4.11:7001
  23. - 172.16.4.11:7002
  24. - 172.16.4.11:7003
  25. - 172.16.4.11:7004
  26. - 172.16.4.11:7005
  27. max-redirects: 3
  28. lettuce:
  29. pool:
  30. max-active: 1024
  31. timeout: 20s
  32. serializer: jdk # 设置fst时,redis key使用string序列化,value使用 fst序列化.
  33. database: 0
  34. r2dbc:
  35. url: r2dbc:postgresql://172.16.4.12:5432/jetlinks #数据库地址 本文档数据库为docker启动
  36. username: postgres
  37. password: jetlinks
  38. pool:
  39. max-size: 32
  40. codec:
  41. max-in-memory-size: 50MB
  42. rabbitmq: # 开启了device.message.writer.rabbitmq.enabled=true时生效
  43. host: localhost
  44. port: 5672
  45. username: admin
  46. password: jetlinks
  47. kafka: # 开启了device.message.writer.kafka.enabled=true时生效
  48. consumer:
  49. client-id: ${spring.application.name}-consumer:${server.port}
  50. group-id: ${spring.application.name}
  51. max-poll-records: 1000
  52. producer:
  53. client-id: ${spring.application.name}-producer:${server.port}
  54. acks: 1
  55. retries: 3
  56. bootstrap-servers: ["127.0.0.1:9092"]
  57. reactor:
  58. debug-agent:
  59. enabled: true # 开启调试代理,在打印异常时将会生成调用踪栈信息
  60. easyorm:
  61. default-schema: public # 数据库默认的schema
  62. dialect: postgres #数据库方言
  63. elasticsearch:
  64. client:
  65. max-conn-total: 128
  66. connect-timeout: 5000
  67. socket-timeout: 5000
  68. connection-request-timeout: 8000
  69. hosts: # es集群配置
  70. - http://172.16.4.13:9200
  71. - http://172.16.4.12:19200
  72. - http://172.16.4.11:9200
  73. index:
  74. default-strategy: time-by-month #默认es的索引按月进行分表
  75. settings:
  76. number-of-shards: 1 # es 分片数量
  77. number-of-replicas: 0 # 副本数量
  78. device:
  79. message:
  80. writer:
  81. time-series:
  82. enabled: true # 直接写出设备消息数据到时序数据库
  83. kafka:
  84. enabled: false # 推送设备消息到kafka
  85. consumer: true # 从kafka订阅消息并写入到时序数据库
  86. topic-name: device.message
  87. rabbitmq:
  88. enabled: false # 推送设备消息到rabbitMQ
  89. consumer: true # 从rabbitMQ订阅消息并写入到时序数据库
  90. thread-size: 4 # 消费线程数
  91. auto-ack: true # 自定应答,为true可能导致数据丢失,但是性能最高。
  92. topic-name: device.message # exchange名称
  93. ignore-message-types:
  94. include-message-types:
  95. hsweb:
  96. cors:
  97. enable: true
  98. configs:
  99. - path: /**
  100. allowed-headers: "*"
  101. allowed-methods: ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]
  102. allowed-origins: ["*"]
  103. allow-credentials: true
  104. max-age: 1800
  105. dict:
  106. enum-packages: org.jetlinks
  107. file:
  108. upload:
  109. static-file-path: ./static/upload
  110. static-location: http://demo2.jetlinks.org:8080/upload # 此地址为服务器域名,将通过nginx代理到固定服务器上
  111. webflux:
  112. response-wrapper: enabled #开启响应包装器(将返回值包装为ResponseMessage)
  113. # auth: #默认的用户配置
  114. # users:
  115. # admin:
  116. # username: admin
  117. # password: admin
  118. # name: 超级管理员
  119. authorize:
  120. auto-parse: true
  121. cache:
  122. type: redis
  123. redis:
  124. local-cache-type: guava
  125. jetlinks:
  126. server-id: ${spring.application.name}:${server.port} #设备服务网关服务ID,不同服务请设置不同的ID
  127. logging:
  128. system:
  129. context:
  130. server: ${spring.application.name}
  131. protocol:
  132. spi:
  133. enabled: false # 为true时开启自动加载通过依赖引入的协议包
  134. rule:
  135. engine:
  136. server-id: ${jetlinks.server-id}
  137. server-name: ${spring.application.name}
  138. logging:
  139. level:
  140. org.jetlinks: info
  141. rule.engine: info
  142. org.hswebframework: debug
  143. org.springframework.transaction: debug
  144. org.springframework.data.r2dbc.connectionfactory: debug
  145. io.micrometer: warn
  146. org.hswebframework.expands: error
  147. system: warn
  148. org.jetlinks.rule.engine: info
  149. org.jetlinks.gateway: info
  150. org.springframework: warn
  151. org.apache.kafka: warn
  152. org.jetlinks.pro.device.message.writer: debug
  153. # org.elasticsearch: error
  154. config: classpath:logback-spring.xml
  155. vertx:
  156. max-event-loop-execute-time-unit: seconds
  157. max-event-loop-execute-time: 30
  158. max-worker-execute-time-unit: seconds
  159. max-worker-execute-time: 30
  160. prefer-native-transport: true
  161. micrometer:
  162. time-series:
  163. tags:
  164. server: ${spring.application.name}
  165. metrics:
  166. default:
  167. step: 30s

注意:

在多台服务器上启动jetlinks,配置中的jetlinks.server-id必须互不相同。

在项目根目录执行:

  1. mvn clean package -DskipTests
  1. 分别在三台服务器上启动jetlinks 文档以脚本方式启动,脚本如下:
  1. #!/bin/bash
  2. nohup java -jar -Dspring.application.name=jetlinks-cluster-test-3 jetlinks-standalone.jar >jetlinks-pro.log 2>&1 &

注意:

三台服务启动的jetlinks.server-id应该互不相同,在application.yml中jetlinks.server-id引用了spring.application.name,所以此处的name应不相同。

nginx配置

通过nginx来代理前后端。

  1. 安装nginx
    前往官网下载集群部署 - 图6 (opens new window) 下载完成后解压、编译、安装。
  2. 修改配置文件

配置文件如下:

  1. upstream iotserver {
  2. server 172.16.4.11:8844;
  3. server 172.16.4.12:8844;
  4. server 172.16.4.13:8844;
  5. }
  6. upstream webserver {
  7. server 172.16.4.11:9000;
  8. }
  9. upstream fileserver {
  10. server 172.16.4.11:8844; #此处指定文件上传到该服务器上
  11. }
  12. server {
  13. listen 8080;
  14. server_name demo2.jetlinks.cn;
  15. location ^~/upload/ {
  16. proxy_pass http://fileserver;
  17. proxy_set_header Host $host:$server_port;
  18. proxy_set_header X-Real-IP $remote_addr;
  19. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  20. }
  21. location ^~/jetlinks/file/static {
  22. proxy_pass http://fileserver/file/static;
  23. proxy_set_header X-Forwarded-Proto $scheme;
  24. proxy_set_header Host $host:$server_port;
  25. proxy_set_header X-Real-IP $remote_addr;
  26. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  27. proxy_send_timeout 30m;
  28. proxy_read_timeout 30m;
  29. client_max_body_size 100m;
  30. }
  31. location ^~/jetlinks/ {
  32. proxy_pass http://iotserver/;
  33. proxy_set_header X-Forwarded-Proto $scheme;
  34. proxy_set_header Host $host:$server_port;
  35. proxy_set_header X-Real-IP $remote_addr;
  36. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  37. proxy_http_version 1.1;
  38. proxy_set_header Upgrade $http_upgrade;
  39. proxy_set_header Connection "upgrade";
  40. proxy_connect_timeout 1;
  41. proxy_buffering off;
  42. chunked_transfer_encoding off;
  43. proxy_cache off;
  44. proxy_send_timeout 30m;
  45. proxy_read_timeout 30m;
  46. client_max_body_size 100m;
  47. }
  48. location / {
  49. proxy_pass http://webserver/;
  50. }
  51. }
  1. 启动
  1. systemctl status nginx.service

tcp负载均衡配置方式

用于代理设备连接。此处使用nginx作为演示,也可以使用LVS,haProxy等方式。

配置文件:

  1. load_module /usr/lib/nginx/modules/ngx_stream_module.so;
  2. user root;
  3. worker_processes 1;
  4. error_log /etc/nginx/log/error.log warn;
  5. pid /var/run/nginx.pid;
  6. events {
  7. worker_connections 1024;
  8. }
  9. stream {
  10. upstream tcp-test {
  11. hash $remote_addr consistent;
  12. server 172.16.4.13:1889 max_fails=3 fail_timeout=10s;
  13. #server 172.16.4.12:1889 max_fails=3 fail_timeout=10s;
  14. server 172.16.4.11:1889 max_fails=3 fail_timeout=10s;
  15. }
  16. server {
  17. listen 1884;
  18. proxy_pass tcp-test;
  19. proxy_connect_timeout 30s;
  20. proxy_timeout 30s;
  21. }
  22. }

注意:

此配置监听的MQTT的1884端口,代理到服务器上的1889。如需代理自定义端口需保持代理端口与jetlinks平台中网络组件开启的端口一致。 如此处平台中MQTT服务网络组件设置的端口应为1889。