Storage负载均衡

用户可以使用BALANCE语句平衡分片和Raft leader的分布,或者删除冗余的Storage服务器。

前提条件

为了平衡分片和Raft leader的分布,Nebula Graph中图空间的副本数都必须大于1。

均衡分片分布

BALANCE DATA语句会开始一个任务,将Nebula Graph集群中的分片平均分配到所有Storage服务器。通过创建和执行一组子任务来迁移数据和均衡分片分布。

示例

以横向扩容Nebula Graph为例,集群中增加新的Storage服务器后,新服务器上没有分片。

  1. 执行命令SHOW HOSTS检查分片的分布。

    1. nebual> SHOW HOSTS;
    2. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    3. | Host | Port | Status | Leader count | Leader distribution | Partition distribution |
    4. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    5. | "storaged0" | 9779 | "ONLINE" | 4 | "basketballplayer:4" | "basketballplayer:15" |
    6. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    7. | "storaged1" | 9779 | "ONLINE" | 8 | "basketballplayer:8" | "basketballplayer:15" |
    8. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    9. | "storaged2" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:15" |
    10. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    11. | "storaged3" | 9779 | "ONLINE" | 0 | "No valid partition" | "No valid partition" |
    12. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    13. | "storaged4" | 9779 | "ONLINE" | 0 | "No valid partition" | "No valid partition" |
    14. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    15. | "Total" | | | 15 | "basketballplayer:15" | "basketballplayer:45" |
    16. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  2. 执行命令BALANCE DATA将所有分片均衡分布。

    1. nebula> BALANCE DATA;
    2. +------------+
    3. | ID |
    4. +------------+
    5. | 1614237867 |
    6. +------------+
  3. 根据返回的任务ID,执行命令BALANCE DATA <balance_id>检查任务状态。

    1. nebula> BALANCE DATA 1614237867;
    2. +--------------------------------------------------------------+-------------------+
    3. | balanceId, spaceId:partId, src->dst | status |
    4. +--------------------------------------------------------------+-------------------+
    5. | "[1614237867, 11:1, storaged1:9779->storaged3:9779]" | "SUCCEEDED" |
    6. +--------------------------------------------------------------+-------------------+
    7. | "[1614237867, 11:1, storaged2:9779->storaged4:9779]" | "SUCCEEDED" |
    8. +--------------------------------------------------------------+-------------------+
    9. | "[1614237867, 11:2, storaged1:9779->storaged3:9779]" | "SUCCEEDED" |
    10. +--------------------------------------------------------------+-------------------+
    11. ...
    12. +--------------------------------------------------------------+-------------------+
    13. | "Total:22, Succeeded:22, Failed:0, In Progress:0, Invalid:0" | 100 |
    14. +--------------------------------------------------------------+-------------------+
  4. 等待所有子任务完成,负载均衡进程结束,执行命令SHOW HOSTS确认分片已经均衡分布。

    1. nebula> SHOW HOSTS;
    2. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    3. | Host | Port | Status | Leader count | Leader distribution | Partition distribution |
    4. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    5. | "storaged0" | 9779 | "ONLINE" | 4 | "basketballplayer:4" | "basketballplayer:9" |
    6. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    7. | "storaged1" | 9779 | "ONLINE" | 8 | "basketballplayer:8" | "basketballplayer:9" |
    8. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    9. | "storaged2" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
    10. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    11. | "storaged3" | 9779 | "ONLINE" | 0 | "No valid partition" | "basketballplayer:9" |
    12. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    13. | "storaged4" | 9779 | "ONLINE" | 0 | "No valid partition" | "basketballplayer:9" |
    14. +-------------+------+----------+--------------+-----------------------------------+------------------------+
    15. | "Total" | | | 15 | "basketballplayer:15" | "basketballplayer:45" |
    16. +-------------+------+----------+--------------+-----------------------------------+------------------------+

如果有子任务失败,请重新执行BALANCE DATA。如果重做负载均衡仍然不能解决问题,请到Nebula Graph社区寻求帮助。

停止负载均衡任务

停止负载均衡任务,请执行命令BALANCE DATA STOP

  • 如果没有正在执行的负载均衡任务,会返回错误。

  • 如果有正在执行的负载均衡任务,会返回停止的任务ID(balance_id)。

BALANCE DATA STOP不会停止正在执行的子任务,而是取消所有后续子任务。用户可以执行命令BALANCE DATA <balance_id>检查停止的任务状态。

一旦所有子任务都完成或停止,用户可以再次执行命令BALANCE DATA

  • 如果前一个负载均衡任务的任何一个子任务失败,Nebula Graph会重新启动之前的负载均衡任务。

  • 如果前一个负载均衡任务的任何一个子任务都没有失败,Nebula Graph会启动一个新的的负载均衡任务。

重置负载均衡任务

如果停止负载均衡任务后重新执行仍然失败,可以尝试用命令BALANCE DATA RESET PLAN重置负载均衡任务,该操作会清空旧的任务。之后再使用BALANCE DATA命令,会新建负载均衡任务,而不是执行旧的任务。

移除Storage服务器

移除指定的Storage服务器来缩小集群规模,可以使用命令BALANCE DATA REMOVE <host_list>

示例

如果需要移除以下两台Storage服务器。

服务器名称IP地址端口
storage3192.168.0.89779
storage4192.168.0.99779

请执行如下命令:

  1. BALANCE DATA REMOVE 192.168.0.8:9779,192.168.0.9:9779;

Nebula Graph将启动一个负载均衡任务,迁移storage3和storage4中的分片,然后将服务器从集群中移除。

均衡leader分布

BALANCE DATA只能均衡分片分布,不能均衡Raft leader分布。用户可以使用命令BALANCE LEADER均衡leader分布。

示例

  1. nebula> BALANCE LEADER;

用户可以执行SHOW HOSTS检查结果。

  1. nebula> SHOW HOSTS;
  2. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  3. | Host | Port | Status | Leader count | Leader distribution | Partition distribution |
  4. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  5. | "storaged0" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
  6. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  7. | "storaged1" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
  8. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  9. | "storaged2" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
  10. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  11. | "storaged3" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
  12. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  13. | "storaged4" | 9779 | "ONLINE" | 3 | "basketballplayer:3" | "basketballplayer:9" |
  14. +-------------+------+----------+--------------+-----------------------------------+------------------------+
  15. | "Total" | | | 15 | "basketballplayer:15" | "basketballplayer:45" |
  16. +-------------+------+----------+--------------+-----------------------------------+------------------------+