测试代码

测试环境&条件

  • 3 台 16C 20G 内存的 docker 容器作为 server node (3 副本)
  • 2 ~ 8 台 8C docker 容器 作为 client
  • 24 个 raft 复制组,平均每台 server node 上各自有 8 个 leader 负责读写请求,不开启 follower 读
  • 压测目标为 JRaft 中的 RheaKV 模块,只压测 put、get 两个接口,其中 get 是保证线性一致读的,key 和 value 大小均为 16 字节
  • 读比例 10%,写比例 90%

目前的测试场景比较简单,以后会增加更多测试场景

测试场景1

场景1: 测试条件

Client 数量Client-BatchingStorage-Type读写比例Replicator-Pipelinekey 大小value 大小
8开启MemoryDB1:9开启16 字节16字节

场景1: 结果汇总:

  • 8 个 client 一共达到 40w+ ops,p95 RT 在 8ms 以内
  • 3 个 server 节点负载没达到极限 load 15 左右,cpu 40% 左右

场景1: 3 个 server 机器负载:

场景1: server1

  1. top - 20:11:14 up 10 days, 23:09, 1 user, load average: 12.29, 6.92, 4.00
  2. Tasks: 36 total, 1 running, 35 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 24.3 us, 17.7 sy, 0.0 ni, 50.0 id, 2.0 wa, 0.0 hi, 0.0 si, 6.0 st
  4. %Cpu1 : 21.9 us, 18.5 sy, 0.0 ni, 49.5 id, 2.0 wa, 0.0 hi, 0.0 si, 8.1 st
  5. %Cpu2 : 20.6 us, 18.6 sy, 0.0 ni, 53.2 id, 2.0 wa, 0.0 hi, 0.0 si, 5.6 st
  6. %Cpu3 : 23.3 us, 20.0 sy, 0.0 ni, 50.3 id, 1.3 wa, 0.0 hi, 0.0 si, 5.0 st
  7. %Cpu4 : 24.1 us, 19.1 sy, 0.0 ni, 49.8 id, 2.3 wa, 0.0 hi, 0.0 si, 4.7 st
  8. %Cpu5 : 21.3 us, 18.9 sy, 0.0 ni, 53.2 id, 2.0 wa, 0.0 hi, 0.0 si, 4.7 st
  9. %Cpu6 : 24.7 us, 18.4 sy, 0.0 ni, 50.2 id, 2.0 wa, 0.0 hi, 0.0 si, 4.7 st
  10. %Cpu7 : 24.8 us, 17.8 sy, 0.0 ni, 50.0 id, 1.7 wa, 0.0 hi, 0.0 si, 5.7 st
  11. %Cpu8 : 26.0 us, 18.3 sy, 0.0 ni, 51.3 id, 2.3 wa, 0.0 hi, 0.0 si, 2.0 st
  12. %Cpu9 : 26.6 us, 16.9 sy, 0.0 ni, 52.2 id, 2.0 wa, 0.0 hi, 0.0 si, 2.3 st
  13. %Cpu10 : 31.7 us, 17.7 sy, 0.0 ni, 46.3 id, 2.3 wa, 0.0 hi, 0.0 si, 2.0 st
  14. %Cpu11 : 23.2 us, 18.9 sy, 0.0 ni, 53.3 id, 2.3 wa, 0.0 hi, 0.0 si, 2.3 st
  15. %Cpu12 : 25.6 us, 18.3 sy, 0.0 ni, 51.5 id, 2.3 wa, 0.0 hi, 0.0 si, 2.3 st
  16. %Cpu13 : 22.6 us, 18.3 sy, 0.0 ni, 54.5 id, 2.3 wa, 0.0 hi, 0.0 si, 2.3 st
  17. %Cpu14 : 24.7 us, 17.3 sy, 0.0 ni, 54.0 id, 1.7 wa, 0.0 hi, 0.0 si, 2.3 st
  18. %Cpu15 : 61.8 us, 8.3 sy, 0.0 ni, 28.2 id, 0.3 wa, 0.0 hi, 0.0 si, 1.3 st
  19. KiB Mem : 62914560 total, 6854596 free, 39128016 used, 16931948 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 6854596 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 15682 root 20 0 12.853g 8.859g 24064 S 708.7 14.8 26:49.38 java

场景1: server2

  1. top - 20:11:47 up 10 days, 23:03, 1 user, load average: 17.68, 8.50, 4.56
  2. Tasks: 33 total, 1 running, 31 sleeping, 0 stopped, 1 zombie
  3. %Cpu0 : 22.7 us, 17.3 sy, 0.0 ni, 35.0 id, 8.3 wa, 0.0 hi, 0.0 si, 16.7 st
  4. %Cpu1 : 20.1 us, 19.4 sy, 0.0 ni, 43.8 id, 9.4 wa, 0.0 hi, 0.0 si, 7.4 st
  5. %Cpu2 : 23.3 us, 20.0 sy, 0.0 ni, 39.7 id, 10.3 wa, 0.0 hi, 0.0 si, 6.7 st
  6. %Cpu3 : 24.1 us, 20.1 sy, 0.0 ni, 40.8 id, 9.4 wa, 0.0 hi, 0.0 si, 5.7 st
  7. %Cpu4 : 21.4 us, 17.7 sy, 0.0 ni, 37.1 id, 9.0 wa, 0.0 hi, 0.0 si, 14.7 st
  8. %Cpu5 : 22.6 us, 19.6 sy, 0.0 ni, 40.5 id, 10.6 wa, 0.0 hi, 0.0 si, 6.6 st
  9. %Cpu6 : 23.6 us, 19.9 sy, 0.0 ni, 40.2 id, 10.3 wa, 0.0 hi, 0.0 si, 6.0 st
  10. %Cpu7 : 20.5 us, 19.9 sy, 0.0 ni, 44.4 id, 9.9 wa, 0.0 hi, 0.0 si, 5.3 st
  11. %Cpu8 : 40.7 us, 13.3 sy, 0.0 ni, 34.3 id, 9.0 wa, 0.0 hi, 0.0 si, 2.7 st
  12. %Cpu9 : 39.9 us, 14.0 sy, 0.0 ni, 35.2 id, 8.6 wa, 0.0 hi, 0.0 si, 2.3 st
  13. %Cpu10 : 39.7 us, 14.2 sy, 0.0 ni, 34.8 id, 8.6 wa, 0.0 hi, 0.0 si, 2.6 st
  14. %Cpu11 : 34.8 us, 16.7 sy, 0.0 ni, 37.5 id, 8.7 wa, 0.0 hi, 0.0 si, 2.3 st
  15. %Cpu12 : 22.3 us, 18.9 sy, 0.0 ni, 44.2 id, 11.0 wa, 0.0 hi, 0.0 si, 3.7 st
  16. %Cpu13 : 38.5 us, 14.6 sy, 0.0 ni, 36.2 id, 8.3 wa, 0.0 hi, 0.0 si, 2.3 st
  17. %Cpu14 : 26.0 us, 18.3 sy, 0.0 ni, 43.0 id, 10.0 wa, 0.0 hi, 0.0 si, 2.7 st
  18. %Cpu15 : 26.9 us, 18.9 sy, 0.0 ni, 40.9 id, 10.3 wa, 0.0 hi, 0.0 si, 3.0 st
  19. KiB Mem : 62914560 total, 6522696 free, 36904808 used, 19487056 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 6522696 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 118164 root 20 0 12.987g 6.889g 24140 S 729.9 11.5 30:59.19 java

场景1: server3

  1. top - 20:12:14 up 6 days, 4:11, 1 user, load average: 13.60, 7.80, 4.29
  2. Tasks: 31 total, 1 running, 30 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 18.8 us, 16.8 sy, 0.0 ni, 53.0 id, 1.3 wa, 0.0 hi, 0.0 si, 10.1 st
  4. %Cpu1 : 90.0 us, 3.3 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 4.7 st
  5. %Cpu2 : 18.1 us, 18.7 sy, 0.0 ni, 54.2 id, 1.7 wa, 0.0 hi, 0.0 si, 7.4 st
  6. %Cpu3 : 19.0 us, 17.0 sy, 0.0 ni, 55.3 id, 1.3 wa, 0.0 hi, 0.0 si, 7.3 st
  7. %Cpu4 : 20.7 us, 16.7 sy, 0.0 ni, 55.0 id, 1.7 wa, 0.0 hi, 0.0 si, 6.0 st
  8. %Cpu5 : 17.2 us, 16.2 sy, 0.0 ni, 57.3 id, 1.6 wa, 0.0 hi, 0.0 si, 7.6 st
  9. %Cpu6 : 15.7 us, 17.1 sy, 0.0 ni, 56.5 id, 1.7 wa, 0.0 hi, 0.0 si, 9.0 st
  10. %Cpu7 : 17.8 us, 17.1 sy, 0.0 ni, 54.4 id, 1.3 wa, 0.0 hi, 0.0 si, 9.4 st
  11. %Cpu8 : 21.9 us, 15.3 sy, 0.0 ni, 56.1 id, 1.3 wa, 0.0 hi, 0.0 si, 5.3 st
  12. %Cpu9 : 21.7 us, 15.3 sy, 0.0 ni, 59.0 id, 1.7 wa, 0.0 hi, 0.0 si, 2.3 st
  13. %Cpu10 : 22.0 us, 17.0 sy, 0.0 ni, 56.7 id, 1.7 wa, 0.0 hi, 0.0 si, 2.7 st
  14. %Cpu11 : 21.6 us, 16.6 sy, 0.0 ni, 57.1 id, 1.7 wa, 0.0 hi, 0.0 si, 3.0 st
  15. %Cpu12 : 22.2 us, 16.2 sy, 0.0 ni, 56.6 id, 2.0 wa, 0.0 hi, 0.0 si, 3.0 st
  16. %Cpu13 : 20.3 us, 17.3 sy, 0.0 ni, 58.5 id, 1.3 wa, 0.0 hi, 0.0 si, 2.7 st
  17. %Cpu14 : 22.7 us, 17.7 sy, 0.0 ni, 55.3 id, 1.7 wa, 0.0 hi, 0.0 si, 2.7 st
  18. %Cpu15 : 22.3 us, 16.9 sy, 0.0 ni, 56.8 id, 1.3 wa, 0.0 hi, 0.0 si, 2.7 st
  19. KiB Mem : 62914560 total, 34784900 free, 12135352 used, 15994308 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 34784900 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 14040 root 20 0 12.893g 7.562g 24064 S 651.5 12.6 34:19.05 java

场景1: 8 个 client 的 ops 数据:

client1:6.6w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 19926351
  4. mean rate = 66044.67 calls/second
  5. 1-minute rate = 51776.67 calls/second
  6. 5-minute rate = 50421.44 calls/second
  7. 15-minute rate = 39736.69 calls/second
  8. min = 0.24 milliseconds
  9. max = 553.79 milliseconds
  10. mean = 6.30 milliseconds
  11. stddev = 19.93 milliseconds
  12. median = 5.70 milliseconds
  13. 75% <= 6.24 milliseconds
  14. 95% <= 7.34 milliseconds
  15. 98% <= 7.89 milliseconds
  16. 99% <= 8.41 milliseconds
  17. 99.9% <= 550.21 milliseconds
  18. get_benchmark_timer
  19. count = 1992665
  20. mean rate = 6604.50 calls/second
  21. 1-minute rate = 5177.87 calls/second
  22. 5-minute rate = 5043.64 calls/second
  23. 15-minute rate = 3976.49 calls/second
  24. min = 0.22 milliseconds
  25. max = 2.37 milliseconds
  26. mean = 0.39 milliseconds
  27. stddev = 0.10 milliseconds
  28. median = 0.37 milliseconds
  29. 75% <= 0.44 milliseconds
  30. 95% <= 0.58 milliseconds
  31. 98% <= 0.65 milliseconds
  32. 99% <= 0.70 milliseconds
  33. 99.9% <= 0.84 milliseconds
  34. put_benchmark_timer
  35. count = 17933686
  36. mean rate = 59438.79 calls/second
  37. 1-minute rate = 46598.96 calls/second
  38. 5-minute rate = 45377.76 calls/second
  39. 15-minute rate = 35760.06 calls/second
  40. min = 3.61 milliseconds
  41. max = 522.33 milliseconds
  42. mean = 9.34 milliseconds
  43. stddev = 38.55 milliseconds
  44. median = 5.87 milliseconds
  45. 75% <= 6.33 milliseconds
  46. 95% <= 7.64 milliseconds
  47. 98% <= 8.76 milliseconds
  48. 99% <= 15.94 milliseconds
  49. 99.9% <= 495.93 milliseconds

client2:5.6w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 20482649
  4. mean rate = 56629.36 calls/second
  5. min = 0.23 milliseconds
  6. max = 560.56 milliseconds
  7. mean = 10.17 milliseconds
  8. stddev = 42.23 milliseconds
  9. median = 6.11 milliseconds
  10. 75% <= 6.74 milliseconds
  11. 95% <= 9.30 milliseconds
  12. 98% <= 14.34 milliseconds
  13. 99% <= 134.99 milliseconds
  14. 99.9% <= 560.56 milliseconds
  15. get_benchmark_timer
  16. count = 1920555
  17. mean rate = 5790.12 calls/second
  18. 1-minute rate = 4400.25 calls/second
  19. 5-minute rate = 4886.09 calls/second
  20. 15-minute rate = 4431.03 calls/second
  21. min = 0.20 milliseconds
  22. max = 483.05 milliseconds
  23. mean = 0.90 milliseconds
  24. stddev = 15.05 milliseconds
  25. median = 0.39 milliseconds
  26. 75% <= 0.47 milliseconds
  27. 95% <= 0.70 milliseconds
  28. 98% <= 0.81 milliseconds
  29. 99% <= 0.90 milliseconds
  30. 99.9% <= 469.68 milliseconds
  31. put_benchmark_timer
  32. count = 17284705
  33. mean rate = 52109.92 calls/second
  34. 1-minute rate = 39601.12 calls/second
  35. 5-minute rate = 43955.99 calls/second
  36. 15-minute rate = 39840.74 calls/second
  37. min = 4.00 milliseconds
  38. max = 566.42 milliseconds
  39. mean = 9.85 milliseconds
  40. stddev = 39.49 milliseconds
  41. median = 6.20 milliseconds
  42. 75% <= 6.75 milliseconds
  43. 95% <= 8.96 milliseconds
  44. 98% <= 11.83 milliseconds
  45. 99% <= 18.95 milliseconds
  46. 99.9% <= 566.42 milliseconds

client3:5.7w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 17381123
  4. mean rate = 57759.74 calls/second
  5. 1-minute rate = 46705.16 calls/second
  6. 5-minute rate = 54031.23 calls/second
  7. 15-minute rate = 54070.53 calls/second
  8. min = 0.18 milliseconds
  9. max = 503.15 milliseconds
  10. mean = 9.05 milliseconds
  11. stddev = 38.99 milliseconds
  12. median = 5.77 milliseconds
  13. 75% <= 6.42 milliseconds
  14. 95% <= 7.84 milliseconds
  15. 98% <= 10.03 milliseconds
  16. 99% <= 46.69 milliseconds
  17. 99.9% <= 503.15 milliseconds
  18. get_benchmark_timer
  19. count = 1738130
  20. mean rate = 5776.01 calls/second
  21. 1-minute rate = 4670.78 calls/second
  22. 5-minute rate = 5404.37 calls/second
  23. 15-minute rate = 5409.38 calls/second
  24. min = 0.17 milliseconds
  25. max = 499.86 milliseconds
  26. mean = 1.28 milliseconds
  27. stddev = 21.15 milliseconds
  28. median = 0.35 milliseconds
  29. 75% <= 0.43 milliseconds
  30. 95% <= 0.61 milliseconds
  31. 98% <= 0.76 milliseconds
  32. 99% <= 0.86 milliseconds
  33. 99.9% <= 499.86 milliseconds
  34. put_benchmark_timer
  35. count = 15642993
  36. mean rate = 51983.20 calls/second
  37. 1-minute rate = 42034.80 calls/second
  38. 5-minute rate = 48626.90 calls/second
  39. 15-minute rate = 48661.16 calls/second
  40. min = 3.55 milliseconds
  41. max = 574.44 milliseconds
  42. mean = 9.21 milliseconds
  43. stddev = 36.41 milliseconds
  44. median = 5.87 milliseconds
  45. 75% <= 6.43 milliseconds
  46. 95% <= 7.89 milliseconds
  47. 98% <= 13.10 milliseconds
  48. 99% <= 16.65 milliseconds
  49. 99.9% <= 522.46 milliseconds

client4:5.3w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 12779528
  4. mean rate = 53033.34 calls/second
  5. 1-minute rate = 45941.89 calls/second
  6. 5-minute rate = 44791.65 calls/second
  7. 15-minute rate = 40517.18 calls/second
  8. min = 0.18 milliseconds
  9. max = 525.26 milliseconds
  10. mean = 8.36 milliseconds
  11. stddev = 34.28 milliseconds
  12. median = 5.73 milliseconds
  13. 75% <= 6.35 milliseconds
  14. 95% <= 7.97 milliseconds
  15. 98% <= 11.73 milliseconds
  16. 99% <= 36.88 milliseconds
  17. 99.9% <= 522.17 milliseconds
  18. get_benchmark_timer
  19. count = 1277965
  20. mean rate = 5303.33 calls/second
  21. 1-minute rate = 4594.36 calls/second
  22. 5-minute rate = 4480.36 calls/second
  23. 15-minute rate = 4053.67 calls/second
  24. min = 0.18 milliseconds
  25. max = 514.77 milliseconds
  26. mean = 0.54 milliseconds
  27. stddev = 8.97 milliseconds
  28. median = 0.36 milliseconds
  29. 75% <= 0.43 milliseconds
  30. 95% <= 0.60 milliseconds
  31. 98% <= 0.73 milliseconds
  32. 99% <= 0.83 milliseconds
  33. 99.9% <= 3.02 milliseconds
  34. put_benchmark_timer
  35. count = 11501763
  36. mean rate = 47729.82 calls/second
  37. 1-minute rate = 41347.87 calls/second
  38. 5-minute rate = 40311.33 calls/second
  39. 15-minute rate = 36463.51 calls/second
  40. min = 4.02 milliseconds
  41. max = 581.53 milliseconds
  42. mean = 9.83 milliseconds
  43. stddev = 37.75 milliseconds
  44. median = 5.88 milliseconds
  45. 75% <= 6.49 milliseconds
  46. 95% <= 8.57 milliseconds
  47. 98% <= 14.53 milliseconds
  48. 99% <= 141.18 milliseconds
  49. 99.9% <= 581.53 milliseconds

client5:4.8w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 8787260
  4. mean rate = 48528.52 calls/second
  5. 1-minute rate = 45517.95 calls/second
  6. 5-minute rate = 44081.76 calls/second
  7. 15-minute rate = 42392.68 calls/second
  8. min = 0.16 milliseconds
  9. max = 508.39 milliseconds
  10. mean = 6.80 milliseconds
  11. stddev = 24.29 milliseconds
  12. median = 5.65 milliseconds
  13. 75% <= 6.14 milliseconds
  14. 95% <= 7.44 milliseconds
  15. 98% <= 9.67 milliseconds
  16. 99% <= 11.37 milliseconds
  17. 99.9% <= 508.39 milliseconds
  18. get_benchmark_timer
  19. count = 878755
  20. mean rate = 4852.96 calls/second
  21. 1-minute rate = 4552.01 calls/second
  22. 5-minute rate = 4410.87 calls/second
  23. 15-minute rate = 4243.26 calls/second
  24. min = 0.14 milliseconds
  25. max = 122.35 milliseconds
  26. mean = 0.62 milliseconds
  27. stddev = 5.50 milliseconds
  28. median = 0.34 milliseconds
  29. 75% <= 0.44 milliseconds
  30. 95% <= 0.59 milliseconds
  31. 98% <= 0.72 milliseconds
  32. 99% <= 0.85 milliseconds
  33. 99.9% <= 122.35 milliseconds
  34. put_benchmark_timer
  35. count = 7908790
  36. mean rate = 43676.00 calls/second
  37. 1-minute rate = 40965.79 calls/second
  38. 5-minute rate = 39667.82 calls/second
  39. 15-minute rate = 38144.83 calls/second
  40. min = 4.04 milliseconds
  41. max = 593.34 milliseconds
  42. mean = 9.75 milliseconds
  43. stddev = 41.98 milliseconds
  44. median = 5.75 milliseconds
  45. 75% <= 6.25 milliseconds
  46. 95% <= 7.62 milliseconds
  47. 98% <= 9.24 milliseconds
  48. 99% <= 17.10 milliseconds
  49. 99.9% <= 519.11 milliseconds

client6:4.5w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 6867485
  4. mean rate = 45507.64 calls/second
  5. 1-minute rate = 45064.38 calls/second
  6. 5-minute rate = 38615.42 calls/second
  7. 15-minute rate = 35853.66 calls/second
  8. min = 0.15 milliseconds
  9. max = 575.29 milliseconds
  10. mean = 8.81 milliseconds
  11. stddev = 39.29 milliseconds
  12. median = 5.53 milliseconds
  13. 75% <= 6.05 milliseconds
  14. 95% <= 7.28 milliseconds
  15. 98% <= 9.24 milliseconds
  16. 99% <= 116.75 milliseconds
  17. 99.9% <= 496.56 milliseconds
  18. get_benchmark_timer
  19. count = 686765
  20. mean rate = 4550.82 calls/second
  21. 1-minute rate = 4506.83 calls/second
  22. 5-minute rate = 3863.49 calls/second
  23. 15-minute rate = 3588.02 calls/second
  24. min = 0.14 milliseconds
  25. max = 1.42 milliseconds
  26. mean = 0.36 milliseconds
  27. stddev = 0.12 milliseconds
  28. median = 0.34 milliseconds
  29. 75% <= 0.41 milliseconds
  30. 95% <= 0.59 milliseconds
  31. 98% <= 0.70 milliseconds
  32. 99% <= 0.75 milliseconds
  33. 99.9% <= 1.27 milliseconds
  34. put_benchmark_timer
  35. count = 6180720
  36. mean rate = 40955.74 calls/second
  37. 1-minute rate = 40557.85 calls/second
  38. 5-minute rate = 34751.96 calls/second
  39. 15-minute rate = 32265.65 calls/second
  40. min = 4.26 milliseconds
  41. max = 552.87 milliseconds
  42. mean = 8.20 milliseconds
  43. stddev = 31.77 milliseconds
  44. median = 5.61 milliseconds
  45. 75% <= 6.18 milliseconds
  46. 95% <= 7.27 milliseconds
  47. 98% <= 9.02 milliseconds
  48. 99% <= 22.27 milliseconds
  49. 99.9% <= 552.87 milliseconds

client7:4.4w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 7999875
  4. mean rate = 44215.63 calls/second
  5. min = 0.17 milliseconds
  6. max = 499.51 milliseconds
  7. mean = 9.70 milliseconds
  8. stddev = 44.64 milliseconds
  9. median = 5.61 milliseconds
  10. 75% <= 6.11 milliseconds
  11. 95% <= 7.38 milliseconds
  12. 98% <= 10.21 milliseconds
  13. 99% <= 116.78 milliseconds
  14. 99.9% <= 499.51 milliseconds
  15. get_benchmark_timer
  16. count = 676095
  17. mean rate = 4479.54 calls/second
  18. 1-minute rate = 4521.71 calls/second
  19. 5-minute rate = 3948.66 calls/second
  20. 15-minute rate = 3724.38 calls/second
  21. min = 0.15 milliseconds
  22. max = 499.29 milliseconds
  23. mean = 1.16 milliseconds
  24. stddev = 19.78 milliseconds
  25. median = 0.34 milliseconds
  26. 75% <= 0.42 milliseconds
  27. 95% <= 0.60 milliseconds
  28. 98% <= 0.71 milliseconds
  29. 99% <= 0.85 milliseconds
  30. 99.9% <= 499.29 milliseconds
  31. put_benchmark_timer
  32. count = 6084841
  33. mean rate = 40315.02 calls/second
  34. 1-minute rate = 40691.08 calls/second
  35. 5-minute rate = 35518.40 calls/second
  36. 15-minute rate = 33492.93 calls/second
  37. min = 4.22 milliseconds
  38. max = 518.07 milliseconds
  39. mean = 10.22 milliseconds
  40. stddev = 44.75 milliseconds
  41. median = 5.70 milliseconds
  42. 75% <= 6.20 milliseconds
  43. 95% <= 7.37 milliseconds
  44. 98% <= 10.08 milliseconds
  45. 99% <= 120.90 milliseconds
  46. 99.9% <= 518.07 milliseconds

client8:4.2w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 7747941
  4. mean rate = 42811.02 calls/second
  5. 1-minute rate = 42423.09 calls/second
  6. 5-minute rate = 35193.32 calls/second
  7. 15-minute rate = 31374.94 calls/second
  8. min = 0.17 milliseconds
  9. max = 589.28 milliseconds
  10. mean = 10.22 milliseconds
  11. stddev = 44.35 milliseconds
  12. median = 5.86 milliseconds
  13. 75% <= 6.64 milliseconds
  14. 95% <= 9.41 milliseconds
  15. 98% <= 13.28 milliseconds
  16. 99% <= 164.26 milliseconds
  17. 99.9% <= 589.28 milliseconds
  18. get_benchmark_timer
  19. count = 774815
  20. mean rate = 4281.18 calls/second
  21. 1-minute rate = 4242.54 calls/second
  22. 5-minute rate = 3521.90 calls/second
  23. 15-minute rate = 3141.29 calls/second
  24. min = 0.18 milliseconds
  25. max = 1.48 milliseconds
  26. mean = 0.34 milliseconds
  27. stddev = 0.12 milliseconds
  28. median = 0.32 milliseconds
  29. 75% <= 0.39 milliseconds
  30. 95% <= 0.55 milliseconds
  31. 98% <= 0.65 milliseconds
  32. 99% <= 0.80 milliseconds
  33. 99.9% <= 1.48 milliseconds
  34. put_benchmark_timer
  35. count = 6973126
  36. mean rate = 38529.00 calls/second
  37. 1-minute rate = 38180.75 calls/second
  38. 5-minute rate = 31671.44 calls/second
  39. 15-minute rate = 28233.65 calls/second
  40. min = 4.42 milliseconds
  41. max = 511.03 milliseconds
  42. mean = 7.98 milliseconds
  43. stddev = 25.88 milliseconds
  44. median = 6.01 milliseconds
  45. 75% <= 6.73 milliseconds
  46. 95% <= 9.71 milliseconds
  47. 98% <= 11.06 milliseconds
  48. 99% <= 15.29 milliseconds
  49. 99.9% <= 511.03 milliseconds

测试场景2

场景2: 测试条件

Client 数量Client-BatchingStorage-Type读写比例Replicator-Pipelinekey 大小value 大小
8开启RocksDB1:9开启16 字节16字节

场景2: 结果汇总

  • 8 个 client 一共达到 25w+ ops, p95 RT 在 20ms 以内
  • 3 个 server 节点负载没达到极限 load 10 左右,cpu 30% 左右,还可以增加 client,client 发送能力受限

场景2: 3 个 server 机器负载

场景2: server1

  1. top - 21:01:33 up 10 days, 23:59, 1 user, load average: 10.52, 9.01, 5.85
  2. Tasks: 28 total, 1 running, 27 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 16.9 us, 13.9 sy, 0.0 ni, 63.2 id, 1.3 wa, 0.0 hi, 0.0 si, 4.6 st
  4. %Cpu1 : 16.6 us, 13.0 sy, 0.0 ni, 62.5 id, 1.3 wa, 0.0 hi, 0.0 si, 6.6 st
  5. %Cpu2 : 15.8 us, 13.5 sy, 0.0 ni, 64.6 id, 0.7 wa, 0.0 hi, 0.0 si, 5.4 st
  6. %Cpu3 : 16.7 us, 13.7 sy, 0.0 ni, 63.7 id, 1.7 wa, 0.0 hi, 0.0 si, 4.3 st
  7. %Cpu4 : 16.7 us, 14.0 sy, 0.0 ni, 63.7 id, 1.3 wa, 0.0 hi, 0.0 si, 4.3 st
  8. %Cpu5 : 16.0 us, 13.0 sy, 0.0 ni, 66.0 id, 1.3 wa, 0.0 hi, 0.0 si, 3.7 st
  9. %Cpu6 : 16.3 us, 13.3 sy, 0.0 ni, 64.7 id, 1.7 wa, 0.0 hi, 0.0 si, 4.0 st
  10. %Cpu7 : 16.3 us, 14.7 sy, 0.0 ni, 62.7 id, 1.7 wa, 0.0 hi, 0.0 si, 4.7 st
  11. %Cpu8 : 17.3 us, 13.6 sy, 0.0 ni, 65.1 id, 1.7 wa, 0.0 hi, 0.0 si, 2.3 st
  12. %Cpu9 : 18.3 us, 13.3 sy, 0.0 ni, 66.0 id, 1.0 wa, 0.0 hi, 0.0 si, 1.3 st
  13. %Cpu10 : 16.9 us, 13.6 sy, 0.0 ni, 65.9 id, 1.7 wa, 0.0 hi, 0.0 si, 2.0 st
  14. %Cpu11 : 18.1 us, 14.0 sy, 0.0 ni, 64.9 id, 1.3 wa, 0.0 hi, 0.0 si, 1.7 st
  15. %Cpu12 : 17.9 us, 14.3 sy, 0.0 ni, 64.5 id, 1.3 wa, 0.0 hi, 0.0 si, 2.0 st
  16. %Cpu13 : 56.7 us, 7.3 sy, 0.0 ni, 34.3 id, 0.7 wa, 0.0 hi, 0.0 si, 1.0 st
  17. %Cpu14 : 17.7 us, 13.3 sy, 0.0 ni, 64.0 id, 1.3 wa, 0.0 hi, 0.0 si, 3.7 st
  18. %Cpu15 : 19.2 us, 12.3 sy, 0.0 ni, 65.2 id, 1.3 wa, 0.0 hi, 0.0 si, 2.0 st
  19. KiB Mem : 62914560 total, 3414404 free, 39254896 used, 20245260 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 3414404 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 33887 root 20 0 13.831g 9.054g 24048 S 513.6 15.1 77:08.73 java

场景2: server2

  1. top - 21:02:18 up 10 days, 23:53, 1 user, load average: 9.57, 9.66, 6.38
  2. Tasks: 34 total, 1 running, 33 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 12.0 us, 8.7 sy, 0.0 ni, 58.7 id, 7.3 wa, 0.0 hi, 0.0 si, 13.3 st
  4. %Cpu1 : 12.1 us, 10.8 sy, 0.0 ni, 65.3 id, 7.4 wa, 0.0 hi, 0.0 si, 4.4 st
  5. %Cpu2 : 11.7 us, 10.3 sy, 0.0 ni, 66.0 id, 8.0 wa, 0.0 hi, 0.0 si, 4.0 st
  6. %Cpu3 : 15.0 us, 10.7 sy, 0.0 ni, 62.3 id, 8.0 wa, 0.0 hi, 0.0 si, 4.0 st
  7. %Cpu4 : 10.0 us, 8.7 sy, 0.0 ni, 60.3 id, 8.0 wa, 0.0 hi, 0.0 si, 13.0 st
  8. %Cpu5 : 12.4 us, 10.7 sy, 0.0 ni, 65.9 id, 7.4 wa, 0.0 hi, 0.0 si, 3.7 st
  9. %Cpu6 : 12.4 us, 10.4 sy, 0.0 ni, 64.2 id, 8.4 wa, 0.0 hi, 0.0 si, 4.7 st
  10. %Cpu7 : 12.4 us, 11.4 sy, 0.0 ni, 63.5 id, 8.0 wa, 0.0 hi, 0.0 si, 4.7 st
  11. %Cpu8 : 13.0 us, 10.0 sy, 0.0 ni, 66.0 id, 8.3 wa, 0.0 hi, 0.0 si, 2.7 st
  12. %Cpu9 : 18.9 us, 16.9 sy, 0.0 ni, 53.0 id, 8.6 wa, 0.0 hi, 0.0 si, 2.6 st
  13. %Cpu10 : 13.0 us, 11.0 sy, 0.0 ni, 65.8 id, 7.6 wa, 0.0 hi, 0.0 si, 2.7 st
  14. %Cpu11 : 12.7 us, 9.3 sy, 0.0 ni, 66.7 id, 8.7 wa, 0.0 hi, 0.0 si, 2.7 st
  15. %Cpu12 : 14.3 us, 11.6 sy, 0.0 ni, 62.5 id, 9.0 wa, 0.0 hi, 0.0 si, 2.7 st
  16. %Cpu13 : 35.7 us, 6.0 sy, 0.0 ni, 50.7 id, 6.0 wa, 0.0 hi, 0.0 si, 1.7 st
  17. %Cpu14 : 12.9 us, 8.6 sy, 0.0 ni, 65.9 id, 9.6 wa, 0.0 hi, 0.0 si, 3.0 st
  18. %Cpu15 : 16.9 us, 15.3 sy, 0.0 ni, 56.5 id, 9.0 wa, 0.0 hi, 0.0 si, 2.3 st
  19. KiB Mem : 62914560 total, 3618372 free, 37721664 used, 21574524 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 3618372 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 5610 root 20 0 13.875g 7.683g 24144 S 397.7 12.8 78:40.24 java

场景2: server3

  1. top - 21:02:40 up 6 days, 5:01, 1 user, load average: 10.67, 9.64, 6.02
  2. Tasks: 33 total, 1 running, 32 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 23.3 us, 8.3 sy, 0.0 ni, 63.1 id, 1.0 wa, 0.0 hi, 0.0 si, 4.3 st
  4. %Cpu1 : 15.6 us, 11.6 sy, 0.0 ni, 67.4 id, 1.0 wa, 0.0 hi, 0.0 si, 4.3 st
  5. %Cpu2 : 14.3 us, 10.7 sy, 0.0 ni, 69.0 id, 1.0 wa, 0.0 hi, 0.0 si, 5.0 st
  6. %Cpu3 : 14.3 us, 10.6 sy, 0.0 ni, 69.4 id, 1.0 wa, 0.0 hi, 0.0 si, 4.7 st
  7. %Cpu4 : 30.5 us, 7.6 sy, 0.0 ni, 57.0 id, 0.7 wa, 0.0 hi, 0.0 si, 4.3 st
  8. %Cpu5 : 13.3 us, 10.7 sy, 0.0 ni, 69.3 id, 1.3 wa, 0.0 hi, 0.0 si, 5.3 st
  9. %Cpu6 : 23.5 us, 6.6 sy, 0.0 ni, 64.6 id, 1.0 wa, 0.0 hi, 0.0 si, 4.3 st
  10. %Cpu7 : 13.0 us, 10.3 sy, 0.0 ni, 69.8 id, 1.0 wa, 0.0 hi, 0.0 si, 6.0 st
  11. %Cpu8 : 16.6 us, 10.6 sy, 0.0 ni, 69.8 id, 1.0 wa, 0.0 hi, 0.0 si, 2.0 st
  12. %Cpu9 : 14.7 us, 10.7 sy, 0.0 ni, 71.0 id, 1.0 wa, 0.0 hi, 0.0 si, 2.7 st
  13. %Cpu10 : 17.1 us, 10.4 sy, 0.0 ni, 69.6 id, 1.0 wa, 0.0 hi, 0.0 si, 2.0 st
  14. %Cpu11 : 22.9 us, 7.6 sy, 0.0 ni, 66.8 id, 1.0 wa, 0.0 hi, 0.0 si, 1.7 st
  15. %Cpu12 : 16.6 us, 10.3 sy, 0.0 ni, 70.4 id, 1.0 wa, 0.0 hi, 0.0 si, 1.7 st
  16. %Cpu13 : 16.0 us, 10.0 sy, 0.0 ni, 70.3 id, 1.3 wa, 0.0 hi, 0.0 si, 2.3 st
  17. %Cpu14 : 15.6 us, 11.0 sy, 0.0 ni, 70.1 id, 1.0 wa, 0.0 hi, 0.0 si, 2.3 st
  18. %Cpu15 : 24.3 us, 7.3 sy, 0.0 ni, 66.4 id, 0.3 wa, 0.0 hi, 0.0 si, 1.7 st
  19. KiB Mem : 62914560 total, 25306552 free, 11674440 used, 25933568 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 25306552 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 28781 root 20 0 14.315g 7.042g 23948 S 436.0 11.7 83:37.86 java

场景2: 8 个 client 的 ops 数据

client1:3.4w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 27063068
  4. mean rate = 34618.47 calls/second
  5. 1-minute rate = 27821.82 calls/second
  6. 5-minute rate = 30759.93 calls/second
  7. 15-minute rate = 29420.06 calls/second
  8. min = 0.26 milliseconds
  9. max = 651.76 milliseconds
  10. mean = 12.97 milliseconds
  11. stddev = 35.78 milliseconds
  12. median = 10.89 milliseconds
  13. 75% <= 12.24 milliseconds
  14. 95% <= 14.66 milliseconds
  15. 98% <= 16.69 milliseconds
  16. 99% <= 80.38 milliseconds
  17. 99.9% <= 638.77 milliseconds
  18. get_benchmark_timer
  19. count = 2706330
  20. mean rate = 3461.87 calls/second
  21. 1-minute rate = 2782.23 calls/second
  22. 5-minute rate = 3076.27 calls/second
  23. 15-minute rate = 2943.51 calls/second
  24. min = 0.22 milliseconds
  25. max = 7.80 milliseconds
  26. mean = 1.00 milliseconds
  27. stddev = 0.81 milliseconds
  28. median = 0.76 milliseconds
  29. 75% <= 1.34 milliseconds
  30. 95% <= 2.53 milliseconds
  31. 98% <= 3.44 milliseconds
  32. 99% <= 4.13 milliseconds
  33. 99.9% <= 4.62 milliseconds
  34. put_benchmark_timer
  35. count = 24356738
  36. mean rate = 31156.48 calls/second
  37. 1-minute rate = 25039.79 calls/second
  38. 5-minute rate = 27683.70 calls/second
  39. 15-minute rate = 26476.56 calls/second
  40. min = 7.02 milliseconds
  41. max = 669.96 milliseconds
  42. mean = 14.28 milliseconds
  43. stddev = 39.64 milliseconds
  44. median = 11.29 milliseconds
  45. 75% <= 12.46 milliseconds
  46. 95% <= 14.87 milliseconds
  47. 98% <= 16.10 milliseconds
  48. 99% <= 30.76 milliseconds
  49. 99.9% <= 644.50 milliseconds

client2:3.3w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 26958831
  4. mean rate = 33228.27 calls/second
  5. 1-minute rate = 24381.18 calls/second
  6. 5-minute rate = 29993.96 calls/second
  7. 15-minute rate = 33389.17 calls/second
  8. stddev = 58.01 milliseconds
  9. median = 11.18 milliseconds
  10. 75% <= 12.93 milliseconds
  11. 95% <= 21.01 milliseconds
  12. 98% <= 54.43 milliseconds
  13. 99% <= 130.17 milliseconds
  14. 99.9% <= 720.17 milliseconds
  15. get_benchmark_timer
  16. count = 2624025
  17. mean rate = 3358.44 calls/second
  18. 1-minute rate = 2464.79 calls/second
  19. 5-minute rate = 3062.32 calls/second
  20. 15-minute rate = 3372.49 calls/second
  21. min = 0.21 milliseconds
  22. max = 31.64 milliseconds
  23. mean = 1.27 milliseconds
  24. stddev = 1.95 milliseconds
  25. median = 0.79 milliseconds
  26. 75% <= 1.37 milliseconds
  27. 95% <= 3.43 milliseconds
  28. 98% <= 5.91 milliseconds
  29. 99% <= 7.85 milliseconds
  30. 99.9% <= 31.64 milliseconds
  31. put_benchmark_timer
  32. count = 23616175
  33. mean rate = 30225.79 calls/second
  34. 1-minute rate = 22183.57 calls/second
  35. 5-minute rate = 27557.31 calls/second
  36. 15-minute rate = 30331.83 calls/second
  37. min = 7.17 milliseconds
  38. max = 909.81 milliseconds
  39. mean = 16.54 milliseconds
  40. stddev = 42.13 milliseconds
  41. median = 11.70 milliseconds
  42. 75% <= 13.49 milliseconds
  43. 95% <= 23.07 milliseconds
  44. 98% <= 42.74 milliseconds
  45. 99% <= 74.63 milliseconds
  46. 99.9% <= 656.58 milliseconds

client3:3.3w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 26253677
  4. mean rate = 33617.12 calls/second
  5. 1-minute rate = 25187.58 calls/second
  6. 5-minute rate = 30816.67 calls/second
  7. 15-minute rate = 33638.31 calls/second
  8. min = 0.17 milliseconds
  9. max = 767.76 milliseconds
  10. mean = 18.57 milliseconds
  11. stddev = 64.92 milliseconds
  12. median = 11.57 milliseconds
  13. 75% <= 13.46 milliseconds
  14. 95% <= 19.38 milliseconds
  15. 98% <= 50.42 milliseconds
  16. 99% <= 183.79 milliseconds
  17. 99.9% <= 767.76 milliseconds
  18. get_benchmark_timer
  19. count = 2625390
  20. mean rate = 3361.74 calls/second
  21. 1-minute rate = 2518.65 calls/second
  22. 5-minute rate = 3081.81 calls/second
  23. 15-minute rate = 3364.76 calls/second
  24. min = 0.18 milliseconds
  25. max = 28.92 milliseconds
  26. mean = 0.97 milliseconds
  27. stddev = 1.49 milliseconds
  28. median = 0.60 milliseconds
  29. 75% <= 1.16 milliseconds
  30. 95% <= 2.51 milliseconds
  31. 98% <= 3.91 milliseconds
  32. 99% <= 4.87 milliseconds
  33. 99.9% <= 28.92 milliseconds
  34. put_benchmark_timer
  35. count = 23628287
  36. mean rate = 30255.31 calls/second
  37. 1-minute rate = 22669.02 calls/second
  38. 5-minute rate = 27734.88 calls/second
  39. 15-minute rate = 30273.56 calls/second
  40. min = 6.54 milliseconds
  41. max = 669.53 milliseconds
  42. mean = 16.63 milliseconds
  43. stddev = 42.65 milliseconds
  44. median = 12.04 milliseconds
  45. 75% <= 13.89 milliseconds
  46. 95% <= 21.06 milliseconds
  47. 98% <= 54.34 milliseconds
  48. 99% <= 91.46 milliseconds
  49. 99.9% <= 669.53 milliseconds

client4:3.2w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 24419381
  4. mean rate = 32518.91 calls/second
  5. 1-minute rate = 24838.53 calls/second
  6. 5-minute rate = 29790.41 calls/second
  7. 15-minute rate = 29588.00 calls/second
  8. min = 0.17 milliseconds
  9. max = 725.16 milliseconds
  10. mean = 16.39 milliseconds
  11. stddev = 54.53 milliseconds
  12. median = 11.47 milliseconds
  13. 75% <= 13.78 milliseconds
  14. 95% <= 19.28 milliseconds
  15. 98% <= 30.65 milliseconds
  16. 99% <= 105.18 milliseconds
  17. 99.9% <= 725.16 milliseconds
  18. get_benchmark_timer
  19. count = 2441960
  20. mean rate = 3251.92 calls/second
  21. 1-minute rate = 2483.80 calls/second
  22. 5-minute rate = 2979.44 calls/second
  23. 15-minute rate = 2960.96 calls/second
  24. min = 0.20 milliseconds
  25. max = 46.11 milliseconds
  26. mean = 1.10 milliseconds
  27. stddev = 2.07 milliseconds
  28. median = 0.65 milliseconds
  29. 75% <= 1.17 milliseconds
  30. 95% <= 3.21 milliseconds
  31. 98% <= 4.67 milliseconds
  32. 99% <= 7.19 milliseconds
  33. 99.9% <= 46.11 milliseconds
  34. put_benchmark_timer
  35. count = 21977431
  36. mean rate = 29266.92 calls/second
  37. 1-minute rate = 22354.78 calls/second
  38. 5-minute rate = 26810.98 calls/second
  39. 15-minute rate = 26627.05 calls/second
  40. min = 6.26 milliseconds
  41. max = 686.77 milliseconds
  42. mean = 19.55 milliseconds
  43. stddev = 60.66 milliseconds
  44. median = 11.64 milliseconds
  45. 75% <= 13.79 milliseconds
  46. 95% <= 20.49 milliseconds
  47. 98% <= 51.49 milliseconds
  48. 99% <= 158.31 milliseconds
  49. 99.9% <= 686.77 milliseconds

client5:3.1w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 24433633
  4. mean rate = 31287.05 calls/second
  5. 1-minute rate = 25593.06 calls/second
  6. 5-minute rate = 28834.95 calls/second
  7. 15-minute rate = 28794.75 calls/second
  8. min = 0.14 milliseconds
  9. max = 669.59 milliseconds
  10. mean = 16.72 milliseconds
  11. stddev = 57.19 milliseconds
  12. median = 11.28 milliseconds
  13. 75% <= 12.70 milliseconds
  14. 95% <= 16.14 milliseconds
  15. 98% <= 20.97 milliseconds
  16. 99% <= 165.17 milliseconds
  17. 99.9% <= 657.06 milliseconds
  18. get_benchmark_timer
  19. count = 2443392
  20. mean rate = 3128.74 calls/second
  21. 1-minute rate = 2559.29 calls/second
  22. 5-minute rate = 2883.82 calls/second
  23. 15-minute rate = 2881.30 calls/second
  24. min = 0.16 milliseconds
  25. max = 149.27 milliseconds
  26. mean = 1.18 milliseconds
  27. stddev = 6.21 milliseconds
  28. median = 0.61 milliseconds
  29. 75% <= 1.14 milliseconds
  30. 95% <= 2.81 milliseconds
  31. 98% <= 3.75 milliseconds
  32. 99% <= 4.15 milliseconds
  33. 99.9% <= 149.27 milliseconds
  34. put_benchmark_timer
  35. count = 21990253
  36. mean rate = 28158.26 calls/second
  37. 1-minute rate = 23033.89 calls/second
  38. 5-minute rate = 25951.15 calls/second
  39. 15-minute rate = 25913.45 calls/second
  40. min = 6.84 milliseconds
  41. max = 763.56 milliseconds
  42. mean = 16.03 milliseconds
  43. stddev = 47.95 milliseconds
  44. median = 11.42 milliseconds
  45. 75% <= 12.76 milliseconds
  46. 95% <= 16.15 milliseconds
  47. 98% <= 19.67 milliseconds
  48. 99% <= 114.30 milliseconds
  49. 99.9% <= 699.76 milliseconds

client6:3.1w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 20830219
  4. mean rate = 31516.69 calls/second
  5. 1-minute rate = 25118.51 calls/second
  6. 5-minute rate = 29308.51 calls/second
  7. 15-minute rate = 27446.80 calls/second
  8. min = 0.18 milliseconds
  9. max = 691.10 milliseconds
  10. mean = 18.82 milliseconds
  11. stddev = 64.48 milliseconds
  12. median = 11.52 milliseconds
  13. 75% <= 13.91 milliseconds
  14. 95% <= 20.09 milliseconds
  15. 98% <= 44.41 milliseconds
  16. 99% <= 514.14 milliseconds
  17. 99.9% <= 691.10 milliseconds
  18. get_benchmark_timer
  19. count = 2083040
  20. mean rate = 3151.69 calls/second
  21. 1-minute rate = 2511.87 calls/second
  22. 5-minute rate = 2931.39 calls/second
  23. 15-minute rate = 2747.00 calls/second
  24. min = 0.17 milliseconds
  25. max = 133.46 milliseconds
  26. mean = 1.07 milliseconds
  27. stddev = 2.73 milliseconds
  28. median = 0.62 milliseconds
  29. 75% <= 1.22 milliseconds
  30. 95% <= 3.12 milliseconds
  31. 98% <= 4.54 milliseconds
  32. 99% <= 6.36 milliseconds
  33. 99.9% <= 12.35 milliseconds
  34. put_benchmark_timer
  35. count = 18747261
  36. mean rate = 28365.02 calls/second
  37. 1-minute rate = 22606.89 calls/second
  38. 5-minute rate = 26377.15 calls/second
  39. 15-minute rate = 24699.81 calls/second
  40. min = 6.56 milliseconds
  41. max = 610.15 milliseconds
  42. mean = 14.15 milliseconds
  43. stddev = 19.10 milliseconds
  44. median = 11.71 milliseconds
  45. 75% <= 13.72 milliseconds
  46. 95% <= 19.98 milliseconds
  47. 98% <= 33.31 milliseconds
  48. 99% <= 49.35 milliseconds
  49. 99.9% <= 256.55 milliseconds

client7: 2.7w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 1662169
  4. mean rate = 27281.91 calls/second
  5. 1-minute rate = 26451.07 calls/second
  6. 5-minute rate = 25433.35 calls/second
  7. 15-minute rate = 25184.94 calls/second
  8. min = 0.19 milliseconds
  9. max = 653.89 milliseconds
  10. mean = 17.29 milliseconds
  11. stddev = 63.40 milliseconds
  12. median = 10.89 milliseconds
  13. 75% <= 12.30 milliseconds
  14. 95% <= 15.52 milliseconds
  15. 98% <= 20.74 milliseconds
  16. 99% <= 609.59 milliseconds
  17. 99.9% <= 653.89 milliseconds
  18. get_benchmark_timer
  19. count = 166240
  20. mean rate = 2728.46 calls/second
  21. 1-minute rate = 2645.56 calls/second
  22. 5-minute rate = 2544.26 calls/second
  23. 15-minute rate = 2519.52 calls/second
  24. min = 0.15 milliseconds
  25. max = 7.26 milliseconds
  26. mean = 0.90 milliseconds
  27. stddev = 0.85 milliseconds
  28. median = 0.62 milliseconds
  29. 75% <= 1.14 milliseconds
  30. 95% <= 2.56 milliseconds
  31. 98% <= 3.32 milliseconds
  32. 99% <= 3.98 milliseconds
  33. 99.9% <= 7.26 milliseconds
  34. put_benchmark_timer
  35. count = 1496129
  36. mean rate = 24554.50 calls/second
  37. 1-minute rate = 23805.54 calls/second
  38. 5-minute rate = 22889.10 calls/second
  39. 15-minute rate = 22665.42 calls/second
  40. min = 6.81 milliseconds
  41. max = 676.67 milliseconds
  42. mean = 16.38 milliseconds
  43. stddev = 53.58 milliseconds
  44. median = 11.06 milliseconds
  45. 75% <= 12.45 milliseconds
  46. 95% <= 15.37 milliseconds
  47. 98% <= 18.17 milliseconds
  48. 99% <= 80.63 milliseconds
  49. 99.9% <= 676.67 milliseconds

client8:2.6w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 1600602
  4. mean rate = 26257.11 calls/second
  5. 1-minute rate = 24534.93 calls/second
  6. 5-minute rate = 21923.66 calls/second
  7. 15-minute rate = 21298.22 calls/second
  8. min = 0.19 milliseconds
  9. max = 682.09 milliseconds
  10. mean = 16.39 milliseconds
  11. stddev = 58.25 milliseconds
  12. median = 10.92 milliseconds
  13. 75% <= 12.34 milliseconds
  14. 95% <= 14.90 milliseconds
  15. 98% <= 19.40 milliseconds
  16. 99% <= 157.02 milliseconds
  17. 99.9% <= 677.99 milliseconds
  18. get_benchmark_timer
  19. count = 160080
  20. mean rate = 2625.94 calls/second
  21. 1-minute rate = 2455.55 calls/second
  22. 5-minute rate = 2196.64 calls/second
  23. 15-minute rate = 2134.65 calls/second
  24. min = 0.17 milliseconds
  25. max = 641.93 milliseconds
  26. mean = 2.25 milliseconds
  27. stddev = 28.01 milliseconds
  28. median = 0.65 milliseconds
  29. 75% <= 1.17 milliseconds
  30. 95% <= 2.72 milliseconds
  31. 98% <= 3.65 milliseconds
  32. 99% <= 4.17 milliseconds
  33. 99.9% <= 641.93 milliseconds
  34. put_benchmark_timer
  35. count = 1440522
  36. mean rate = 23629.27 calls/second
  37. 1-minute rate = 22077.83 calls/second
  38. 5-minute rate = 19726.70 calls/second
  39. 15-minute rate = 19163.46 calls/second
  40. min = 7.15 milliseconds
  41. max = 669.63 milliseconds
  42. mean = 16.56 milliseconds
  43. stddev = 55.32 milliseconds
  44. median = 11.10 milliseconds
  45. 75% <= 12.30 milliseconds
  46. 95% <= 14.86 milliseconds
  47. 98% <= 17.11 milliseconds
  48. 99% <= 83.11 milliseconds
  49. 99.9% <= 655.09 milliseconds

测试场景3

场景3: 测试条件

Client 数量Client-BatchingStorage-Type读写比例Replicator-Pipelinekey 大小value 大小
2关闭Memory1:9开启16 字节16字节

场景3: 结果汇总

  • 2 个 client 一共达到 10w+ ops,p95 RT 在 10ms 以内
  • 3 个 server 节点负载达到极限 load 接近 20,cpu 超过 50%
  • ops 降低的原因在于关闭了 client-batching 后 RPC(bolt) 层面也出现了瓶颈,所以还是建议打开 client-batching 开关,client-batching 没有副作用,完全不会影响单个消息的延迟

场景3: 3 个 server 机器负载

场景3: server1

  1. top - 22:39:42 up 11 days, 1:37, 1 user, load average: 15.82, 8.18, 4.28
  2. Tasks: 33 total, 1 running, 32 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 49.3 us, 18.5 sy, 0.0 ni, 24.2 id, 1.0 wa, 0.0 hi, 0.0 si, 7.0 st
  4. %Cpu1 : 33.7 us, 21.2 sy, 0.0 ni, 36.7 id, 1.0 wa, 0.0 hi, 0.0 si, 7.4 st
  5. %Cpu2 : 33.8 us, 22.7 sy, 0.0 ni, 35.1 id, 1.3 wa, 0.0 hi, 0.0 si, 7.0 st
  6. %Cpu3 : 34.9 us, 22.3 sy, 0.0 ni, 34.2 id, 1.7 wa, 0.0 hi, 0.0 si, 7.0 st
  7. %Cpu4 : 36.5 us, 21.6 sy, 0.0 ni, 35.5 id, 1.0 wa, 0.0 hi, 0.0 si, 5.3 st
  8. %Cpu5 : 35.3 us, 22.0 sy, 0.0 ni, 35.0 id, 1.3 wa, 0.0 hi, 0.0 si, 6.3 st
  9. %Cpu6 : 27.9 us, 24.2 sy, 0.0 ni, 34.3 id, 1.0 wa, 0.0 hi, 0.0 si, 12.5 st
  10. %Cpu7 : 35.8 us, 22.4 sy, 0.0 ni, 35.1 id, 1.0 wa, 0.0 hi, 0.0 si, 5.7 st
  11. %Cpu8 : 61.1 us, 9.0 sy, 0.0 ni, 27.9 id, 0.7 wa, 0.0 hi, 0.0 si, 1.3 st
  12. %Cpu9 : 50.3 us, 12.7 sy, 0.0 ni, 34.3 id, 0.7 wa, 0.0 hi, 0.0 si, 2.0 st
  13. %Cpu10 : 49.3 us, 15.3 sy, 0.0 ni, 32.3 id, 1.0 wa, 0.0 hi, 0.0 si, 2.0 st
  14. %Cpu11 : 42.5 us, 20.3 sy, 0.0 ni, 33.6 id, 1.0 wa, 0.0 hi, 0.0 si, 2.7 st
  15. %Cpu12 : 50.3 us, 14.9 sy, 0.0 ni, 31.8 id, 0.7 wa, 0.0 hi, 0.0 si, 2.3 st
  16. %Cpu13 : 39.5 us, 21.3 sy, 0.0 ni, 35.2 id, 1.3 wa, 0.0 hi, 0.0 si, 2.7 st
  17. %Cpu14 : 37.5 us, 21.6 sy, 0.0 ni, 36.9 id, 1.3 wa, 0.0 hi, 0.0 si, 2.7 st
  18. %Cpu15 : 38.7 us, 21.7 sy, 0.0 ni, 36.0 id, 1.0 wa, 0.0 hi, 0.0 si, 2.7 st
  19. KiB Mem : 62914560 total, 23990152 free, 36612212 used, 2312196 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 23990152 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 93733 root 20 0 10.266g 6.475g 23748 S 968.0 10.8 13:03.22 java

场景3: server2

  1. top - 22:39:55 up 11 days, 1:31, 1 user, load average: 17.92, 9.24, 4.61
  2. Tasks: 34 total, 1 running, 33 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 36.5 us, 21.9 sy, 0.0 ni, 25.9 id, 0.7 wa, 0.0 hi, 0.0 si, 15.0 st
  4. %Cpu1 : 45.8 us, 22.4 sy, 0.0 ni, 24.4 id, 0.7 wa, 0.0 hi, 0.0 si, 6.7 st
  5. %Cpu2 : 67.1 us, 13.6 sy, 0.0 ni, 12.3 id, 0.7 wa, 0.0 hi, 0.0 si, 6.3 st
  6. %Cpu3 : 44.3 us, 22.7 sy, 0.0 ni, 25.7 id, 1.0 wa, 0.0 hi, 0.0 si, 6.3 st
  7. %Cpu4 : 40.1 us, 21.4 sy, 0.0 ni, 24.1 id, 0.7 wa, 0.0 hi, 0.0 si, 13.7 st
  8. %Cpu5 : 53.2 us, 17.9 sy, 0.0 ni, 21.3 id, 1.0 wa, 0.0 hi, 0.0 si, 6.6 st
  9. %Cpu6 : 47.2 us, 20.9 sy, 0.0 ni, 24.9 id, 1.0 wa, 0.0 hi, 0.0 si, 6.0 st
  10. %Cpu7 : 51.5 us, 18.6 sy, 0.0 ni, 24.6 id, 0.7 wa, 0.0 hi, 0.0 si, 4.7 st
  11. %Cpu8 : 48.3 us, 21.3 sy, 0.0 ni, 26.7 id, 0.7 wa, 0.0 hi, 0.0 si, 3.0 st
  12. %Cpu9 : 52.5 us, 17.6 sy, 0.0 ni, 26.9 id, 0.7 wa, 0.0 hi, 0.0 si, 2.3 st
  13. %Cpu10 : 52.2 us, 20.1 sy, 0.0 ni, 24.4 id, 0.7 wa, 0.0 hi, 0.0 si, 2.7 st
  14. %Cpu11 : 50.2 us, 19.5 sy, 0.0 ni, 26.4 id, 1.0 wa, 0.0 hi, 0.0 si, 3.0 st
  15. %Cpu12 : 56.3 us, 15.2 sy, 0.0 ni, 25.2 id, 1.0 wa, 0.0 hi, 0.0 si, 2.3 st
  16. %Cpu13 : 61.3 us, 16.0 sy, 0.0 ni, 19.7 id, 0.7 wa, 0.0 hi, 0.0 si, 2.3 st
  17. %Cpu14 : 49.8 us, 19.6 sy, 0.0 ni, 26.6 id, 1.3 wa, 0.0 hi, 0.0 si, 2.7 st
  18. %Cpu15 : 52.8 us, 14.6 sy, 0.0 ni, 29.2 id, 1.0 wa, 0.0 hi, 0.0 si, 2.3 st
  19. KiB Mem : 62914560 total, 24122012 free, 36653792 used, 2138756 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 24122012 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 64477 root 20 0 10.243g 6.651g 24096 S 1108 11.1 15:44.10 java

场景3: server3

  1. top - 22:40:06 up 6 days, 6:39, 1 user, load average: 18.66, 8.40, 4.05
  2. Tasks: 33 total, 1 running, 32 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 35.1 us, 21.7 sy, 0.0 ni, 34.1 id, 1.0 wa, 0.0 hi, 0.0 si, 8.0 st
  4. %Cpu1 : 31.9 us, 20.6 sy, 0.0 ni, 38.5 id, 1.0 wa, 0.0 hi, 0.0 si, 8.0 st
  5. %Cpu2 : 33.6 us, 20.9 sy, 0.0 ni, 36.9 id, 1.3 wa, 0.0 hi, 0.0 si, 7.3 st
  6. %Cpu3 : 29.7 us, 21.3 sy, 0.0 ni, 37.5 id, 1.4 wa, 0.0 hi, 0.0 si, 10.1 st
  7. %Cpu4 : 32.8 us, 21.7 sy, 0.0 ni, 35.1 id, 1.3 wa, 0.0 hi, 0.0 si, 9.0 st
  8. %Cpu5 : 32.1 us, 21.7 sy, 0.0 ni, 36.5 id, 1.7 wa, 0.0 hi, 0.0 si, 8.0 st
  9. %Cpu6 : 34.4 us, 21.2 sy, 0.0 ni, 36.4 id, 2.0 wa, 0.0 hi, 0.0 si, 6.0 st
  10. %Cpu7 : 32.6 us, 22.3 sy, 0.0 ni, 36.2 id, 1.7 wa, 0.0 hi, 0.0 si, 7.3 st
  11. %Cpu8 : 39.4 us, 19.5 sy, 0.0 ni, 36.1 id, 1.3 wa, 0.0 hi, 0.0 si, 3.6 st
  12. %Cpu9 : 39.3 us, 18.7 sy, 0.0 ni, 37.0 id, 1.3 wa, 0.0 hi, 0.0 si, 3.7 st
  13. %Cpu10 : 32.3 us, 20.7 sy, 0.0 ni, 42.3 id, 1.0 wa, 0.0 hi, 0.0 si, 3.7 st
  14. %Cpu11 : 36.9 us, 19.9 sy, 0.0 ni, 38.2 id, 1.7 wa, 0.0 hi, 0.0 si, 3.3 st
  15. %Cpu12 : 35.5 us, 21.3 sy, 0.0 ni, 38.2 id, 1.3 wa, 0.0 hi, 0.0 si, 3.7 st
  16. %Cpu13 : 36.4 us, 21.5 sy, 0.0 ni, 36.8 id, 1.7 wa, 0.0 hi, 0.0 si, 3.6 st
  17. %Cpu14 : 37.9 us, 20.3 sy, 0.0 ni, 36.9 id, 1.7 wa, 0.0 hi, 0.0 si, 3.3 st
  18. %Cpu15 : 38.2 us, 20.3 sy, 0.0 ni, 36.9 id, 1.3 wa, 0.0 hi, 0.0 si, 3.3 st
  19. KiB Mem : 62914560 total, 46373164 free, 11221960 used, 5319436 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 46373164 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 80602 root 20 0 10.248g 6.683g 23852 S 892.4 11.1 17:00.73 java

场景3: 2 个 client 的 ops 数据

client1:6.7w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 4179630
  4. mean rate = 67738.98 calls/second
  5. 1-minute rate = 50557.06 calls/second
  6. 5-minute rate = 29763.94 calls/second
  7. 15-minute rate = 24315.01 calls/second
  8. min = 0.26 milliseconds
  9. max = 180.11 milliseconds
  10. mean = 3.41 milliseconds
  11. stddev = 12.61 milliseconds
  12. median = 1.95 milliseconds
  13. 75% <= 2.81 milliseconds
  14. 95% <= 5.50 milliseconds
  15. 98% <= 8.18 milliseconds
  16. 99% <= 13.32 milliseconds
  17. 99.9% <= 175.37 milliseconds
  18. get_benchmark_timer
  19. count = 417963
  20. mean rate = 6773.36 calls/second
  21. 1-minute rate = 5055.28 calls/second
  22. 5-minute rate = 2975.47 calls/second
  23. 15-minute rate = 2430.44 calls/second
  24. min = 0.21 milliseconds
  25. max = 172.13 milliseconds
  26. mean = 1.92 milliseconds
  27. stddev = 8.21 milliseconds
  28. median = 1.17 milliseconds
  29. 75% <= 1.81 milliseconds
  30. 95% <= 3.59 milliseconds
  31. 98% <= 5.11 milliseconds
  32. 99% <= 7.12 milliseconds
  33. 99.9% <= 170.09 milliseconds
  34. put_benchmark_timer
  35. count = 3761900
  36. mean rate = 60957.97 calls/second
  37. 1-minute rate = 45499.16 calls/second
  38. 5-minute rate = 26782.83 calls/second
  39. 15-minute rate = 21878.18 calls/second
  40. min = 0.52 milliseconds
  41. max = 378.67 milliseconds
  42. mean = 3.61 milliseconds
  43. stddev = 14.97 milliseconds
  44. median = 1.99 milliseconds
  45. 75% <= 2.83 milliseconds
  46. 95% <= 5.18 milliseconds
  47. 98% <= 9.14 milliseconds
  48. 99% <= 25.40 milliseconds
  49. 99.9% <= 170.50 milliseconds

client2:5w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 4654332
  4. mean rate = 50967.66 calls/second
  5. 1-minute rate = 45068.29 calls/second
  6. 5-minute rate = 30478.74 calls/second
  7. 15-minute rate = 25927.10 calls/second
  8. min = 0.23 milliseconds
  9. max = 137.20 milliseconds
  10. mean = 4.46 milliseconds
  11. stddev = 12.00 milliseconds
  12. median = 2.13 milliseconds
  13. 75% <= 3.30 milliseconds
  14. 95% <= 9.63 milliseconds
  15. 98% <= 31.22 milliseconds
  16. 99% <= 78.35 milliseconds
  17. 99.9% <= 137.20 milliseconds
  18. get_benchmark_timer
  19. count = 465471
  20. mean rate = 5096.89 calls/second
  21. 1-minute rate = 4507.05 calls/second
  22. 5-minute rate = 3048.50 calls/second
  23. 15-minute rate = 2593.46 calls/second
  24. min = 0.17 milliseconds
  25. max = 171.40 milliseconds
  26. mean = 2.84 milliseconds
  27. stddev = 11.44 milliseconds
  28. median = 1.11 milliseconds
  29. 75% <= 1.88 milliseconds
  30. 95% <= 5.50 milliseconds
  31. 98% <= 10.88 milliseconds
  32. 99% <= 63.25 milliseconds
  33. 99.9% <= 149.80 milliseconds
  34. put_benchmark_timer
  35. count = 4189457
  36. mean rate = 45872.23 calls/second
  37. 1-minute rate = 40560.19 calls/second
  38. 5-minute rate = 27427.51 calls/second
  39. 15-minute rate = 23330.54 calls/second
  40. min = 0.50 milliseconds
  41. max = 180.62 milliseconds
  42. mean = 5.11 milliseconds
  43. stddev = 15.16 milliseconds
  44. median = 2.27 milliseconds
  45. 75% <= 3.38 milliseconds
  46. 95% <= 9.20 milliseconds
  47. 98% <= 61.84 milliseconds
  48. 99% <= 102.72 milliseconds
  49. 99.9% <= 180.62 milliseconds

场景4: 测试场景4

场景4: 测试条件

Client 数量Client-BatchingStorage-Type读写比例Replicator-Pipelinekey 大小value 大小
2关闭Memory1:9关闭16 字节16字节

场景4: 结果汇总

  • 2 个 client 一共达到 7.5w ops,p95 RT 在 15ms 以内
  • 3 个 server 节点负载达到极限 load 接近 20 (有一台超过 20), cpu 超过 50%
  • ops 降低的原因在于关闭了 client-batching 后 RPC(bolt) 层面也出现了瓶颈,所以还是建议打开 client-batching 开关,client-batching 没有副作用,完全不会影响单个消息的延迟

场景4: 3 个 server 机器负载

场景4: server1

  1. top - 23:19:52 up 11 days, 2:18, 1 user, load average: 17.16, 9.82, 5.15
  2. Tasks: 28 total, 1 running, 27 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 20.1 us, 13.4 sy, 0.0 ni, 60.2 id, 1.3 wa, 0.0 hi, 0.0 si, 5.0 st
  4. %Cpu1 : 15.6 us, 12.0 sy, 0.0 ni, 66.1 id, 1.0 wa, 0.0 hi, 0.0 si, 5.3 st
  5. %Cpu2 : 19.3 us, 12.3 sy, 0.0 ni, 63.7 id, 1.0 wa, 0.0 hi, 0.0 si, 3.7 st
  6. %Cpu3 : 19.3 us, 12.7 sy, 0.0 ni, 62.3 id, 1.7 wa, 0.0 hi, 0.0 si, 4.0 st
  7. %Cpu4 : 18.6 us, 12.3 sy, 0.0 ni, 63.8 id, 1.3 wa, 0.0 hi, 0.0 si, 4.0 st
  8. %Cpu5 : 73.1 us, 6.0 sy, 0.0 ni, 16.9 id, 0.3 wa, 0.0 hi, 0.0 si, 3.7 st
  9. %Cpu6 : 22.6 us, 10.6 sy, 0.0 ni, 60.8 id, 1.0 wa, 0.0 hi, 0.0 si, 5.0 st
  10. %Cpu7 : 18.7 us, 12.0 sy, 0.0 ni, 62.7 id, 1.3 wa, 0.0 hi, 0.0 si, 5.3 st
  11. %Cpu8 : 20.6 us, 11.0 sy, 0.0 ni, 65.1 id, 1.7 wa, 0.0 hi, 0.0 si, 1.7 st
  12. %Cpu9 : 19.0 us, 11.3 sy, 0.0 ni, 66.3 id, 1.3 wa, 0.0 hi, 0.0 si, 2.0 st
  13. %Cpu10 : 33.9 us, 8.6 sy, 0.0 ni, 54.8 id, 1.0 wa, 0.0 hi, 0.0 si, 1.7 st
  14. %Cpu11 : 19.6 us, 12.0 sy, 0.0 ni, 65.1 id, 1.7 wa, 0.0 hi, 0.0 si, 1.7 st
  15. %Cpu12 : 19.7 us, 11.7 sy, 0.0 ni, 66.0 id, 1.0 wa, 0.0 hi, 0.0 si, 1.7 st
  16. %Cpu13 : 17.3 us, 11.6 sy, 0.0 ni, 68.4 id, 1.0 wa, 0.0 hi, 0.0 si, 1.7 st
  17. %Cpu14 : 21.7 us, 11.3 sy, 0.0 ni, 64.0 id, 1.3 wa, 0.0 hi, 0.0 si, 1.7 st
  18. %Cpu15 : 18.8 us, 11.9 sy, 0.0 ni, 65.7 id, 1.7 wa, 0.0 hi, 0.0 si, 2.0 st
  19. KiB Mem : 62914560 total, 21751724 free, 36954048 used, 4208788 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 21751724 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 112036 root 20 0 10.756g 6.728g 23928 S 556.8 11.2 33:35.32 java

场景4: server2

  1. top - 23:20:07 up 11 days, 2:11, 1 user, load average: 11.21, 8.29, 5.24
  2. Tasks: 34 total, 1 running, 33 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 25.5 us, 14.4 sy, 0.0 ni, 44.3 id, 1.0 wa, 0.0 hi, 0.0 si, 14.8 st
  4. %Cpu1 : 27.1 us, 17.1 sy, 0.0 ni, 47.8 id, 1.7 wa, 0.0 hi, 0.0 si, 6.4 st
  5. %Cpu2 : 27.5 us, 17.5 sy, 0.0 ni, 47.0 id, 2.0 wa, 0.0 hi, 0.0 si, 6.0 st
  6. %Cpu3 : 38.8 us, 13.7 sy, 0.0 ni, 40.1 id, 1.3 wa, 0.0 hi, 0.0 si, 6.0 st
  7. %Cpu4 : 23.1 us, 17.7 sy, 0.0 ni, 42.8 id, 1.3 wa, 0.0 hi, 0.0 si, 15.1 st
  8. %Cpu5 : 25.3 us, 18.7 sy, 0.0 ni, 48.0 id, 1.3 wa, 0.0 hi, 0.0 si, 6.7 st
  9. %Cpu6 : 38.3 us, 14.3 sy, 0.0 ni, 40.0 id, 1.7 wa, 0.0 hi, 0.0 si, 5.7 st
  10. %Cpu7 : 26.2 us, 17.9 sy, 0.0 ni, 49.2 id, 1.3 wa, 0.0 hi, 0.0 si, 5.3 st
  11. %Cpu8 : 55.5 us, 10.6 sy, 0.0 ni, 30.9 id, 1.0 wa, 0.0 hi, 0.0 si, 2.0 st
  12. %Cpu9 : 40.7 us, 13.0 sy, 0.0 ni, 43.3 id, 1.0 wa, 0.0 hi, 0.0 si, 2.0 st
  13. %Cpu10 : 40.4 us, 14.2 sy, 0.0 ni, 40.7 id, 1.3 wa, 0.0 hi, 0.0 si, 3.3 st
  14. %Cpu11 : 24.5 us, 18.5 sy, 0.0 ni, 52.3 id, 2.0 wa, 0.0 hi, 0.0 si, 2.6 st
  15. %Cpu12 : 26.5 us, 17.2 sy, 0.0 ni, 51.7 id, 1.7 wa, 0.0 hi, 0.0 si, 3.0 st
  16. %Cpu13 : 29.2 us, 16.9 sy, 0.0 ni, 49.8 id, 1.7 wa, 0.0 hi, 0.0 si, 2.3 st
  17. %Cpu14 : 29.1 us, 16.9 sy, 0.0 ni, 49.3 id, 2.0 wa, 0.0 hi, 0.0 si, 2.6 st
  18. %Cpu15 : 29.5 us, 17.2 sy, 0.0 ni, 49.3 id, 1.3 wa, 0.0 hi, 0.0 si, 2.6 st
  19. KiB Mem : 62914560 total, 22364924 free, 37127368 used, 3422268 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 22364924 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 82346 root 20 0 10.710g 7.086g 23864 S 770.7 11.8 27:31.04 java

场景4: server3

  1. top - 23:20:36 up 6 days, 7:19, 1 user, load average: 23.60, 11.37, 5.58
  2. Tasks: 36 total, 1 running, 35 sleeping, 0 stopped, 0 zombie
  3. %Cpu0 : 61.0 us, 15.3 sy, 0.0 ni, 15.3 id, 0.3 wa, 0.0 hi, 0.0 si, 8.0 st
  4. %Cpu1 : 53.3 us, 20.0 sy, 0.0 ni, 17.7 id, 0.7 wa, 0.0 hi, 0.0 si, 8.3 st
  5. %Cpu2 : 63.3 us, 16.7 sy, 0.0 ni, 11.7 id, 0.3 wa, 0.0 hi, 0.0 si, 8.0 st
  6. %Cpu3 : 76.4 us, 11.0 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 0.0 si, 5.0 st
  7. %Cpu4 : 61.1 us, 15.9 sy, 0.0 ni, 15.9 id, 0.3 wa, 0.0 hi, 0.0 si, 6.6 st
  8. %Cpu5 : 73.6 us, 10.7 sy, 0.0 ni, 9.4 id, 0.0 wa, 0.0 hi, 0.0 si, 6.4 st
  9. %Cpu6 : 52.3 us, 19.3 sy, 0.0 ni, 15.7 id, 0.3 wa, 0.0 hi, 0.0 si, 12.3 st
  10. %Cpu7 : 74.3 us, 11.0 sy, 0.0 ni, 8.7 id, 0.0 wa, 0.0 hi, 0.0 si, 6.0 st
  11. %Cpu8 : 70.1 us, 12.3 sy, 0.0 ni, 14.6 id, 0.3 wa, 0.0 hi, 0.0 si, 2.7 st
  12. %Cpu9 : 72.1 us, 12.3 sy, 0.0 ni, 13.0 id, 0.3 wa, 0.0 hi, 0.0 si, 2.3 st
  13. %Cpu10 : 70.4 us, 14.3 sy, 0.0 ni, 12.6 id, 0.0 wa, 0.0 hi, 0.0 si, 2.7 st
  14. %Cpu11 : 59.5 us, 18.3 sy, 0.0 ni, 18.3 id, 0.3 wa, 0.0 hi, 0.0 si, 3.7 st
  15. %Cpu12 : 66.2 us, 13.9 sy, 0.0 ni, 16.9 id, 0.3 wa, 0.0 hi, 0.0 si, 2.6 st
  16. %Cpu13 : 66.4 us, 15.3 sy, 0.0 ni, 15.0 id, 0.3 wa, 0.0 hi, 0.0 si, 3.0 st
  17. %Cpu14 : 85.1 us, 6.0 sy, 0.0 ni, 7.3 id, 0.0 wa, 0.0 hi, 0.0 si, 1.7 st
  18. %Cpu15 : 76.9 us, 11.2 sy, 0.0 ni, 9.2 id, 0.3 wa, 0.0 hi, 0.0 si, 2.3 st
  19. KiB Mem : 62914560 total, 44333956 free, 12042248 used, 6538356 buff/cache
  20. KiB Swap: 2097148 total, 2097148 free, 0 used. 44333956 avail Mem
  21. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  22. 94729 root 20 0 10.931g 7.454g 23852 S 1300 12.4 39:10.99 java

场景4: 2 个 client 的 ops 数据

client1:5w ops

  1. -- Timers ----------------------------------------------------------------------
  2. -- Timers ----------------------------------------------------------------------
  3. benchmark_timer
  4. count = 13865824
  5. mean rate = 50991.58 calls/second
  6. 1-minute rate = 36409.73 calls/second
  7. 5-minute rate = 36313.26 calls/second
  8. 15-minute rate = 27293.11 calls/second
  9. min = 0.22 milliseconds
  10. max = 176.44 milliseconds
  11. mean = 4.76 milliseconds
  12. stddev = 10.70 milliseconds
  13. median = 1.83 milliseconds
  14. 75% <= 3.94 milliseconds
  15. 95% <= 14.85 milliseconds
  16. 98% <= 42.14 milliseconds
  17. 99% <= 51.69 milliseconds
  18. 99.9% <= 120.15 milliseconds
  19. get_benchmark_timer
  20. count = 1386585
  21. mean rate = 5099.09 calls/second
  22. 1-minute rate = 3640.95 calls/second
  23. 5-minute rate = 3631.72 calls/second
  24. 15-minute rate = 2730.02 calls/second
  25. min = 0.18 milliseconds
  26. max = 163.21 milliseconds
  27. mean = 2.59 milliseconds
  28. stddev = 10.82 milliseconds
  29. median = 0.61 milliseconds
  30. 75% <= 1.33 milliseconds
  31. 95% <= 7.23 milliseconds
  32. 98% <= 15.76 milliseconds
  33. 99% <= 38.85 milliseconds
  34. 99.9% <= 163.21 milliseconds
  35. put_benchmark_timer
  36. count = 12479370
  37. mean rate = 45891.51 calls/second
  38. 1-minute rate = 32768.72 calls/second
  39. 5-minute rate = 32678.93 calls/second
  40. 15-minute rate = 24558.33 calls/second
  41. min = 0.47 milliseconds
  42. max = 194.99 milliseconds
  43. mean = 5.52 milliseconds
  44. stddev = 14.86 milliseconds
  45. median = 1.88 milliseconds
  46. 75% <= 3.98 milliseconds
  47. 95% <= 17.87 milliseconds
  48. 98% <= 44.19 milliseconds
  49. 99% <= 64.29 milliseconds
  50. 99.9% <= 194.99 milliseconds

client2:2.6w ops

  1. -- Timers ----------------------------------------------------------------------
  2. benchmark_timer
  3. count = 3517274
  4. mean rate = 26758.85 calls/second
  5. 1-minute rate = 25361.61 calls/second
  6. 5-minute rate = 9553.50 calls/second
  7. 15-minute rate = 3606.93 calls/second
  8. min = 0.16 milliseconds
  9. max = 244.37 milliseconds
  10. mean = 5.79 milliseconds
  11. stddev = 15.54 milliseconds
  12. median = 2.02 milliseconds
  13. 75% <= 4.67 milliseconds
  14. 95% <= 16.92 milliseconds
  15. 98% <= 44.83 milliseconds
  16. 99% <= 84.95 milliseconds
  17. 99.9% <= 244.37 milliseconds
  18. get_benchmark_timer
  19. count = 351743
  20. mean rate = 2675.91 calls/second
  21. 1-minute rate = 2536.20 calls/second
  22. 5-minute rate = 955.37 calls/second
  23. 15-minute rate = 360.70 calls/second
  24. min = 0.16 milliseconds
  25. max = 107.90 milliseconds
  26. mean = 2.00 milliseconds
  27. stddev = 6.64 milliseconds
  28. median = 0.57 milliseconds
  29. 75% <= 1.32 milliseconds
  30. 95% <= 7.01 milliseconds
  31. 98% <= 10.18 milliseconds
  32. 99% <= 14.93 milliseconds
  33. 99.9% <= 107.90 milliseconds
  34. put_benchmark_timer
  35. count = 3165875
  36. mean rate = 24083.79 calls/second
  37. 1-minute rate = 22825.39 calls/second
  38. 5-minute rate = 8598.12 calls/second
  39. 15-minute rate = 3246.22 calls/second
  40. min = 0.51 milliseconds
  41. max = 192.06 milliseconds
  42. mean = 5.92 milliseconds
  43. stddev = 13.93 milliseconds
  44. median = 2.18 milliseconds
  45. 75% <= 4.99 milliseconds
  46. 95% <= 18.09 milliseconds
  47. 98% <= 48.30 milliseconds
  48. 99% <= 71.10 milliseconds
  49. 99.9% <= 192.06 milliseconds