使用 SDS 为 Gateway 提供 HTTPS 加密支持

控制 Ingress 流量任务中描述了如何进行配置,通过 Ingress Gateway 把服务的 HTTP 端点暴露给外部。这里更进一步,使用单向或者双向 TLS 来完成开放服务的任务。双向 TLS 所需的私钥、服务器证书以及根证书都由 Secret 发现服务(SDS)完成配置。

开始之前

  1. $ curl --version | grep LibreSSL
  2. curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

如果上面的命令输出了一段 LibreSSL 的版本信息,就说明你的 curl 命令可以完成本任务的内容。否则就要想办法换一个不同的 curl 了,例如可以换用一台运行 Linux 的工作站。

为服务器和客户端生成证书

可以使用各种常用工具来生成证书和私钥。这个例子中用了一个来自 https://github.com/nicholasjackson/mtls-go-example脚本来完成工作。

  1. $ git clone https://github.com/nicholasjackson/mtls-go-example
  • 进入代码库文件夹:
  1. $ pushd mtls-go-example
  • httpbin.example.com 生成证书。注意要把下面命令中的 password 替换为其它值。
  1. $ ./generate.sh httpbin.example.com password

看到提示后,所有问题都输入 Y 即可。这个命令会生成四个目录:1_root2_intermediate3_application 以及 4_client。这些目录中包含了后续过程所需的客户端和服务端证书。

  • 把证书移动到 httpbin.example.com 目录之中:
  1. $ mkdir ~+1/httpbin.example.com && mv 1_root 2_intermediate 3_application 4_client ~+1/httpbin.example.com
  • 返回之前的目录:
  1. $ popd

使用 SDS 配置 TLS Ingress 网关

可以配置 TLS Ingress 网关,让它从 Ingress 网关代理通过 SDS 获取凭据。Ingress 网关代理和 Ingress 网关在同一个 Pod 中运行,监视 Ingress 网关所在命名空间中新建的 Secret。在 Ingress 网关中启用 SDS 具有如下好处:

  • Ingress 网关无需重启,就可以动态的新增、删除或者更新密钥/证书对以及根证书。

  • 无需加载 Secret 卷。创建了 kubernetes Secret 之后,这个 Secret 就会被网关代理捕获,并以密钥/证书对和根证书的形式发送给 Ingress 网关。

  • 网关代理能够监视多个密钥/证书对。只需要为每个主机名创建 Secret 并更新网关定义就可以了。

  • 在 Ingress 网关上启用 SDS,并部署 Ingress 网关代理。

这个功能缺省是禁用的,因此需要在 Helm 中打开 istio-ingressgateway.sds.enabled 开关,然后生成 istio-ingressgateway.yaml 文件:

  1. $ helm template install/kubernetes/helm/istio/ --name istio \
  2. --namespace istio-system -x charts/gateways/templates/deployment.yaml \
  3. --set gateways.istio-egressgateway.enabled=false \
  4. --set gateways.istio-ingressgateway.sds.enabled=true > \
  5. $HOME/istio-ingressgateway.yaml
  6. $ kubectl apply -f $HOME/istio-ingressgateway.yaml
  • 设置两个环境变量:INGRESS_HOSTSECURE_INGRESS_PORT
  1. $ export SECURE_INGRESS_PORT=$(kubectl -n istio-system \
  2. get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
  3. $ export INGRESS_HOST=$(kubectl -n istio-system \
  4. get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

为单一主机配置 TLS Ingress 网关

  • 启动 httpbin 样例:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. name: httpbin
  6. labels:
  7. app: httpbin
  8. spec:
  9. ports:
  10. - name: http
  11. port: 8000
  12. selector:
  13. app: httpbin
  14. ---
  15. apiVersion: extensions/v1beta1
  16. kind: Deployment
  17. metadata:
  18. name: httpbin
  19. spec:
  20. replicas: 1
  21. template:
  22. metadata:
  23. labels:
  24. app: httpbin
  25. version: v1
  26. spec:
  27. containers:
  28. - image: docker.io/citizenstig/httpbin
  29. imagePullPolicy: IfNotPresent
  30. name: httpbin
  31. ports:
  32. - containerPort: 8000
  33. EOF
  • 为 Ingress 网关创建 Secret:
  1. $ kubectl create -n istio-system secret generic httpbin-credential \
  2. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
  3. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
  • 创建一个网关,其 servers: 字段的端口为 443,设置 credentialName 的值为 httpbin-credential。这个值就是 Secret 的名字。TLS 模式设置为 SIMPLE
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5. name: mygateway
  6. spec:
  7. selector:
  8. istio: ingressgateway # 使用缺省的 Ingress 网关。
  9. servers:
  10. - port:
  11. number: 443
  12. name: https
  13. protocol: HTTPS
  14. tls:
  15. mode: SIMPLE
  16. credentialName: "httpbin-credential" # 和 Secret 名称一致
  17. hosts:
  18. - "httpbin.example.com"
  19. EOF
  • 配置网关的 Ingress 流量路由,并配置对应的 VirtualService::
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5. name: httpbin
  6. spec:
  7. hosts:
  8. - "httpbin.example.com"
  9. gateways:
  10. - mygateway
  11. http:
  12. - match:
  13. - uri:
  14. prefix: /status
  15. - uri:
  16. prefix: /delay
  17. route:
  18. - destination:
  19. port:
  20. number: 8000
  21. host: httpbin
  22. EOF
  • 用 HTTPS 协议访问 httpbin 服务:
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418

httpbin 服务会返回 418 I’m a Teapot

  • 删除网关的 Secret,并新建另外一个,然后修改 Ingress 网关的凭据:
  1. $ kubectl -n istio-system delete secret httpbin-credential
  1. $ pushd mtls-go-example
  2. $ ./generate.sh httpbin.example.com <password>
  3. $ mkdir ~+1/httpbin.new.example.com && mv 1_root 2_intermediate \
  4. 3_application 4_client ~+1/httpbin.new.example.com
  5. $ popd
  6. $ kubectl create -n istio-system secret generic httpbin-credential \
  7. --from-file=key=httpbin.new.example.com/3_application/private/httpbin.example.com.key.pem \
  8. --from-file=cert=httpbin.new.example.com/3_application/certs/httpbin.example.com.cert.pem
  • 使用 curl 访问 httpbin 服务:
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.new.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
  5. ...
  6. HTTP/2 418
  7. ...
  8. -=[ teapot ]=-
  9. _...._
  10. .' _ _ `.
  11. | ."` ^ `". _,
  12. \_;`"---"`|//
  13. | ;/
  14. \_ _/
  15. `"""`
  • 如果尝试使用之前的证书链来再次访问 httpbin,就会得到失败的结果:
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
  5. ...
  6. * TLSv1.2 (OUT), TLS handshake, Client hello (1):
  7. * TLSv1.2 (IN), TLS handshake, Server hello (2):
  8. * TLSv1.2 (IN), TLS handshake, Certificate (11):
  9. * TLSv1.2 (OUT), TLS alert, Server hello (2):
  10. * SSL certificate problem: unable to get local issuer certificate

为 TLS Ingress 网关配置多个主机名

可以把多个主机名配置到同一个 Ingress 网关上,例如 httpbin.example.comhelloworld-v1.example.com。Ingress 网关会为每个 credentialName 获取一个唯一的凭据。

  • 启动 hellowworld-v1 示例:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. name: helloworld-v1
  6. labels:
  7. app: helloworld-v1
  8. spec:
  9. ports:
  10. - name: http
  11. port: 8000
  12. selector:
  13. app: helloworld-v1
  14. ---
  15. apiVersion: extensions/v1beta1
  16. kind: Deployment
  17. metadata:
  18. name: helloworld-v1
  19. spec:
  20. replicas: 1
  21. template:
  22. metadata:
  23. labels:
  24. app: helloworld-v1
  25. spec:
  26. containers:
  27. - name: helloworld
  28. image: istio/examples-helloworld-v1
  29. resources:
  30. requests:
  31. cpu: "100m"
  32. imagePullPolicy: IfNotPresent #Always
  33. ports:
  34. - containerPort: 5000
  35. EOF
  • 为 Ingress 网关创建一个 Ingress。如果已经创建了 httpbin-credential,就可以创建 helloworld-credential Secret 了。
  1. $ pushd mtls-go-example
  2. $ ./generate.sh helloworld-v1.example.com <password>
  3. $ mkdir ~+1/helloworld-v1.example.com && mv 1_root 2_intermediate \
  4. 3_application 4_client ~+1/helloworld-v1.example.com
  5. $ popd
  6. $ kubectl create -n istio-system secret generic helloworld-credential \
  7. --from-file=key=helloworld-v1.example.com/3_application/private/helloworld-v1.example.com.key.pem \
  8. --from-file=cert=helloworld-v1.example.com/3_application/certs/helloworld-v1.example.com.cert.pem
  • 定义一个网关,其中包含了两个 server,都开放了 443 端口。两个 credentialName 字段分别赋值为 httpbin-credentialhelloworld-credentialserverCertificate 以及 privateKey 应该为空。
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5. name: mygateway
  6. spec:
  7. selector:
  8. istio: ingressgateway # 使用缺省的 Ingress 网关
  9. servers:
  10. - port:
  11. number: 443
  12. name: https-httpbin
  13. protocol: HTTPS
  14. tls:
  15. mode: SIMPLE
  16. credentialName: "httpbin-credential"
  17. hosts:
  18. - "httpbin.example.com"
  19. - port:
  20. number: 443
  21. name: https-helloworld
  22. protocol: HTTPS
  23. tls:
  24. mode: SIMPLE
  25. credentialName: "helloworld-credential"
  26. hosts:
  27. - "helloworld-v1.example.com"
  28. EOF
  • 配置网关的流量路由,配置 VirtualService:、
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: VirtualService
  4. metadata:
  5. name: helloworld-v1
  6. spec:
  7. hosts:
  8. - "helloworld-v1.example.com"
  9. gateways:
  10. - mygateway
  11. http:
  12. - match:
  13. - uri:
  14. exact: /hello
  15. route:
  16. - destination:
  17. host: helloworld-v1
  18. port:
  19. number: 5000
  20. EOF
  • helloworld-v1.example.com 发送 HTTPS 请求:
  1. $ curl -v -HHost:helloworld-v1.example.com \
  2. --resolve helloworld-v1.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert helloworld-v1.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://helloworld-v1.example.com:$SECURE_INGRESS_PORT/hello
  5. HTTP/2 200
  • 发送 HTTPS 请求到 httpbin.example.com,还是会看到茶壶:
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
  5. -=[ teapot ]=-
  6. _...._
  7. .' _ _ `.
  8. | ."` ^ `". _,
  9. \_;`"---"`|//
  10. | ;/
  11. \_ _/
  12. `"""`

配置双向 TLS Ingress 网关

可以对网关的定义进行扩展,加入双向 TLS 的支持。要修改 Ingress 网关的凭据,就要删除并重建对应的 Secret。服务器会使用 CA 证书对客户端进行校验,因此需要使用 cacert 字段来保存 CA 证书:

  1. $ kubectl -n istio-system delete secret httpbin-credential
  2. $ kubectl create -n istio-system secret generic httpbin-credential  \
  3. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
  4. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem \
  5. --from-file=cacert=httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem
  • 修改网关定义,设置 TLS 的模式为 MUTUAL
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5. name: mygateway
  6. spec:
  7. selector:
  8. istio: ingressgateway # Istio 的缺省 Ingress 网关
  9. servers:
  10. - port:
  11. number: 443
  12. name: https
  13. protocol: HTTPS
  14. tls:
  15. mode: MUTUAL
  16. credentialName: "httpbin-credential" # 和 Secret 名称一致
  17. hosts:
  18. - "httpbin.example.com"
  19. EOF
  • 使用前面的方式尝试发出 HTTPS 请求,会看到失败的过程:
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
  5. * TLSv1.2 (OUT), TLS header, Certificate Status (22):
  6. * TLSv1.2 (OUT), TLS handshake, Client hello (1):
  7. * TLSv1.2 (IN), TLS handshake, Server hello (2):
  8. * TLSv1.2 (IN), TLS handshake, Certificate (11):
  9. * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  10. * TLSv1.2 (IN), TLS handshake, Request CERT (13):
  11. * TLSv1.2 (IN), TLS handshake, Server finished (14):
  12. * TLSv1.2 (OUT), TLS handshake, Certificate (11):
  13. * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  14. * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  15. * TLSv1.2 (OUT), TLS handshake, Finished (20):
  16. * TLSv1.2 (IN), TLS alert, Server hello (2):
  17. * error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
  • curl 命令中加入客户端证书和私钥的参数,重新发送请求。(客户端证书参数为 —cert,私钥参数为 —key
  1. $ curl -v -HHost:httpbin.example.com \
  2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
  3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
  4. --cert httpbin.example.com/4_client/certs/httpbin.example.com.cert.pem \
  5. --key httpbin.example.com/4_client/private/httpbin.example.com.key.pem \
  6. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
  7. -=[ teapot ]=-
  8. _...._
  9. .' _ _ `.
  10. | ."` ^ `". _,
  11. \_;`"---"`|//
  12. | ;/
  13. \_ _/

故障排查

  • 查看 INGRESS_HOSTSECURE_INGRESS_PORT 环境变量。根据下面的输出内容,确认其中是否包含了有效的值:
  1. $ kubectl get svc -n istio-system
  2. $ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT
  • 检查 istio-ingressgateway 控制器的日志,搜寻其中的错误信息:
  1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=ingressgateway \
  2. -n istio-system -o jsonpath='{.items[0].metadata.name}') -c istio-proxy
  • 如果使用的是 macOS,检查其编译信息,确认其中包含 LibreSSL,具体步骤在开始之前一节中有具体描述。

  • istio-system 命名空间中是否成功创建了 Secret

  1. $ kubectl -n istio-system get secrets

httpbin-credentialhelloworld-credential 都应该出现在列表之中。

  • 检查日志,看 Ingress 网关代理是否已经成功的把密钥和证书对推送给了 Ingress 网关:
  1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=ingressgateway \
  2. -n istio-system -o jsonpath='{.items[0].metadata.name}') -c ingress-sds

正常情况下,日志中应该显示 httpbin-credential 已经成功创建。如果使用的是双向 TLS,还应该看到 httpbin-credential-cacert。通过对日志的查看,能够验证 Ingress 网关代理从 Ingress 网关收到了 SDS 请求,资源名称是 httpbin-credential,Ingress 网关最后得到了应有的密钥/证书对。如果使用的是双向 TLS,日志会显示出密钥/证书对已经发送给 Ingress 网关,网关代理接收到了资源名为 httpbin-credential-cacert 的 SDS 请求,Ingress 网关用这种方式获取根证书。

清理

  • 删除网关配置、VirtualService 以及 Secret
  1. $ kubectl delete gateway mygateway
  2. $ kubectl delete virtualservice httpbin
  3. $ kubectl delete --ignore-not-found=true -n istio-system secret httpbin-credential \
  4. helloworld-credential
  5. $ kubectl delete --ignore-not-found=true virtualservice helloworld-v1
  • 删除证书目录以及用于生成证书的代码库:
  1. $ rm -rf httpbin.example.com helloworld-v1.example.com mtls-go-example
  • 删除用于重新部署 Ingress 网关的文件:
  1. $ rm -f $HOME/istio-ingressgateway.yaml
  • 关闭 httpbinhelloworld-v1 服务:
  1. $ kubectl delete service --ignore-not-found=true helloworld-v1
  2. $ kubectl delete service --ignore-not-found=true httpbin