如何设置熔断规则

安装示例程序

如果你还没有安装示例程序,请参照 快速开始 安装 Aeraki,Istio 及示例程序。

安装完成后,可以看到集群中增加了下面两个 NS,这两个 NS 中分别安装了基于 MetaProtocol 实现的 Dubbo 和 Thrift 协议的示例程序。 你可以选用任何一个程序进行测试。

  1. ~ kubectl get ns|grep meta
  2. meta-dubbo Active 16m
  3. meta-thrift Active 16m

模拟 thrift 服务调用失败

通过下面的命令创建一个 thrift-sample-server-fake deployment,该 deployment 将在 thrift-sample-server service 中增加一个 endpoint。从下面的 yaml 可以看到,该 deployment 中并没有 thrift sample server,而是一个 nginx 容器。

  1. kubectl apply -f- <<EOF
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5. name: thrift-sample-server-fake
  6. namespace: meta-thrift
  7. labels:
  8. app: thrift-sample-server
  9. spec:
  10. selector:
  11. matchLabels:
  12. app: thrift-sample-server
  13. replicas: 1
  14. template:
  15. metadata:
  16. annotations:
  17. sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
  18. sidecar.istio.io/proxyImage: aeraki/meta-protocol-proxy:1.0.1
  19. sidecar.istio.io/rewriteAppHTTPProbers: "false"
  20. labels:
  21. app: thrift-sample-server
  22. spec:
  23. containers:
  24. - name: thrift-sample-server
  25. image: nginx
  26. ports:
  27. - containerPort: 9090
  28. EOF

现在 thrift-sample-server service 中有三个 endpoint,其中 thrift-sample-server-fake 这个 deployment 中没有 thrift sample server,因此其对应的 endpoint “172.19.0.102” 并不能处理客户端的请求。从客户端的日志中可以看到每三次请求中就有一次错误信息:

  1. ~ aerakictl_app_log client meta-thrift --tail 0 -f
  2. Connected to thrift-sample-server
  3. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  4. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  5. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  6. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  7. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  8. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  9. at org.aeraki.HelloClient.main(HelloClient.java:44)
  10. Connected to thrift-sample-server
  11. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  12. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  13. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  14. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  15. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  16. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  17. at org.aeraki.HelloClient.main(HelloClient.java:44)

创建熔断规则

通过下面的命令创建一个 Destination Rule:当请求连续出现 5 次错误后,就将出错的 upstream host 从 cluster 的负载均衡分组中移除。

  1. kubectl apply -f- <<EOF
  2. apiVersion: networking.istio.io/v1beta1
  3. kind: DestinationRule
  4. metadata:
  5. name: thrift-sample-server
  6. namespace: meta-thrift
  7. spec:
  8. host: thrift-sample-server
  9. trafficPolicy:
  10. outlierDetection:
  11. baseEjectionTime: 15m
  12. consecutive5xxErrors: 5
  13. interval: 5m
  14. EOF

此时查看客户端的输出,可以看到客户端在熔断规则指定的错误次数后,不再将请求发送到出错的 endpoint “172.19.0.102”。

  1. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  2. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  3. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  4. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  5. at org.aeraki.HelloClient.main(HelloClient.java:44)
  6. Connected to thrift-sample-server
  7. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  8. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  9. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  10. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  11. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  12. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  13. at org.aeraki.HelloClient.main(HelloClient.java:44)
  14. Connected to thrift-sample-server
  15. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  16. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  17. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  18. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  19. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  20. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  21. at org.aeraki.HelloClient.main(HelloClient.java:44)
  22. Connected to thrift-sample-server
  23. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  24. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  25. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  26. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  27. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  28. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  29. at org.aeraki.HelloClient.main(HelloClient.java:44)
  30. Connected to thrift-sample-server
  31. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  32. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  33. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  34. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  35. at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  36. at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  37. at org.aeraki.HelloClient.main(HelloClient.java:44)
  38. Connected to thrift-sample-server
  39. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  40. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  41. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  42. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  43. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  44. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  45. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  46. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  47. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  48. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  49. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10

理解原理

通过查看 envoy 的 stats 输出,可以看到 thrift-sample-server 发生了连续的请求错误,该出错的 host 被移除了该 cluster 的负载均衡分组。

  1. aerakictl_sidecar_stats client meta-thrift|grep -i outlier
  2. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_active: 1
  3. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_consecutive_5xx: 1
  4. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_detected_consecutive_5xx: 1
  5. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_consecutive_5xx: 1
  6. luster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_total: 1
  7. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_total: 1

最后修改 March 22, 2022: circuit breaking (f12858b)