Certificate

APISIX supports to load multiple SSL certificates by TLS extension Server Name Indication (SNI).

Single SNI

It is most common for an SSL certificate to contain only one domain. We can create an ssl object. Here is a simple case, creates a ssl object and route object.

  • cert: PEM-encoded public certificate of the SSL key pair.
  • key: PEM-encoded private key of the SSL key pair.
  • snis: Hostname(s) to associate with this certificate as SNIs. To set this attribute this certificate must have a valid private key associated with it.

We will use the Python script below to simplify the example:

create-ssl.py

  1. #!/usr/bin/env python
  2. # coding: utf-8
  3. import sys
  4. # sudo pip install requests
  5. import requests
  6. if len(sys.argv) <= 3:
  7. print("bad argument")
  8. sys.exit(1)
  9. with open(sys.argv[1]) as f:
  10. cert = f.read()
  11. with open(sys.argv[2]) as f:
  12. key = f.read()
  13. sni = sys.argv[3]
  14. api_key = "edd1c9f034335f136f87ad84b625c8f1"
  15. resp = requests.put("http://127.0.0.1:9180/apisix/admin/ssls/1", json={
  16. "cert": cert,
  17. "key": key,
  18. "snis": [sni],
  19. }, headers={
  20. "X-API-KEY": api_key,
  21. })
  22. print(resp.status_code)
  23. print(resp.text)
  1. # create SSL object
  2. ./create-ssl.py t.crt t.key test.com
  3. # create Router object
  4. curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
  5. {
  6. "uri": "/hello",
  7. "hosts": ["test.com"],
  8. "methods": ["GET"],
  9. "upstream": {
  10. "type": "roundrobin",
  11. "nodes": {
  12. "127.0.0.1:1980": 1
  13. }
  14. }
  15. }'
  16. # make a test
  17. curl --resolve 'test.com:9443:127.0.0.1' https://test.com:9443/hello -vvv
  18. * Added test.com:9443:127.0.0.1 to DNS cache
  19. * About to connect() to test.com port 9443 (#0)
  20. * Trying 127.0.0.1...
  21. * Connected to test.com (127.0.0.1) port 9443 (#0)
  22. * Initializing NSS with certpath: sql:/etc/pki/nssdb
  23. * skipping SSL peer certificate verification
  24. * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  25. * Server certificate:
  26. * subject: CN=test.com,O=iresty,L=ZhuHai,ST=GuangDong,C=CN
  27. * start date: Jun 24 22:18:05 2019 GMT
  28. * expire date: May 31 22:18:05 2119 GMT
  29. * common name: test.com
  30. * issuer: CN=test.com,O=iresty,L=ZhuHai,ST=GuangDong,C=CN
  31. > GET /hello HTTP/1.1
  32. > User-Agent: curl/7.29.0
  33. > Host: test.com:9443
  34. > Accept: */*

wildcard SNI

Sometimes, one SSL certificate may contain a wildcard domain like *.test.com, that means it can accept more than one domain, eg: www.test.com or mail.test.com.

Here is an example, note that the value we pass as sni is *.test.com.

  1. ./create-ssl.py t.crt t.key '*.test.com'
  2. curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
  3. {
  4. "uri": "/hello",
  5. "hosts": ["*.test.com"],
  6. "methods": ["GET"],
  7. "upstream": {
  8. "type": "roundrobin",
  9. "nodes": {
  10. "127.0.0.1:1980": 1
  11. }
  12. }
  13. }'
  14. # make a test
  15. curl --resolve 'www.test.com:9443:127.0.0.1' https://www.test.com:9443/hello -vvv
  16. * Added test.com:9443:127.0.0.1 to DNS cache
  17. * About to connect() to test.com port 9443 (#0)
  18. * Trying 127.0.0.1...
  19. * Connected to test.com (127.0.0.1) port 9443 (#0)
  20. * Initializing NSS with certpath: sql:/etc/pki/nssdb
  21. * skipping SSL peer certificate verification
  22. * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  23. * Server certificate:
  24. * subject: CN=test.com,O=iresty,L=ZhuHai,ST=GuangDong,C=CN
  25. * start date: Jun 24 22:18:05 2019 GMT
  26. * expire date: May 31 22:18:05 2119 GMT
  27. * common name: test.com
  28. * issuer: CN=test.com,O=iresty,L=ZhuHai,ST=GuangDong,C=CN
  29. > GET /hello HTTP/1.1
  30. > User-Agent: curl/7.29.0
  31. > Host: test.com:9443
  32. > Accept: */*

multiple domain

If your SSL certificate may contain more than one domain, like www.test.com and mail.test.com, then you can add them into the snis array. For example:

  1. {
  2. "snis": ["www.test.com", "mail.test.com"]
  3. }

multiple certificates for a single domain

If you want to configure multiple certificate for a single domain, for instance, supporting both the ECC and RSA key-exchange algorithm, then just configure the extra certificates (the first certificate and private key should be still put in cert and key) and private keys by certs and keys.

  • certs: PEM-encoded certificate array.
  • keys: PEM-encoded private key array.

APISIX will pair certificate and private key with the same indice as a SSL key pair. So the length of certs and keys must be same.

set up multiple CA certificates

APISIX currently uses CA certificates in several places, such as Protect Admin API, etcd with mTLS, and Deployment Modes.

In these places, ssl_trusted_certificate or trusted_ca_cert will be used to set up the CA certificate, but these configurations will eventually be translated into lua_ssl_trusted_certificate directive in OpenResty.

If you need to set up different CA certificates in different places, then you can package these CA certificates into a CA bundle file and point to this file when you need to set up CAs. This will avoid the problem that the generated lua_ssl_trusted_certificate has multiple locations and overwrites each other.

The following is a complete example to show how to set up multiple CA certificates in APISIX.

Suppose we let client and APISIX Admin API, APISIX and ETCD communicate with each other using mTLS protocol, and currently there are two CA certificates, foo_ca.crt and bar_ca.crt, and use each of these two CA certificates to issue client and server certificate pairs, foo_ca.crt and its issued certificate pair are used to protect Admin API, and bar_ca.crt and its issued certificate pair are used to protect ETCD.

The following table details the configurations involved in this example and what they do:

ConfigurationTypeDescription
foo_ca.crtCA certIssues the secondary certificate required for the client to communicate with the APISIX Admin API over mTLS.
foo_client.crtcertA certificate issued by foo_ca.crt and used by the client to prove its identity when accessing the APISIX Admin API.
foo_client.keykeyIssued by foo_ca.crt, used by the client, the key file required to access the APISIX Admin API.
foo_server.crtcertIssued by foo_ca.crt, used by APISIX, corresponding to the admin_api_mtls.admin_ssl_cert configuration entry.
foo_server.keykeyIssued by foo_ca.crt, used by APISIX, corresponding to the admin_api_mtls.admin_ssl_cert_key configuration entry.
admin.apisix.devdonameCommon Name used in issuing foo_server.crt certificate, through which the client accesses APISIX Admin API
bar_ca.crtCA certIssues the secondary certificate required for APISIX to communicate with ETCD over mTLS.
bar_etcd.crtcertIssued by bar_ca.crt and used by ETCD, corresponding to the -cert-file option in the ETCD startup command.
bar_etcd.keykeyIssued by bar_ca.crt and used by ETCD, corresponding to the —key-file option in the ETCD startup command.
bar_apisix.crtcertIssued by bar_ca.crt, used by APISIX, corresponding to the etcd.tls.cert configuration entry.
bar_apisix.keykeyIssued by bar_ca.crt, used by APISIX, corresponding to the etcd.tls.key configuration entry.
etcd.cluster.devkeyCommon Name used in issuing bar_etcd.crt certificate, which is used as SNI when APISIX communicates with ETCD over mTLS. corresponds to etcd.tls.sni configuration item.
apisix.ca-bundleCA bundleMerged from foo_ca.crt and bar_ca.crt, replacing foo_ca.crt and bar_ca.crt.
  1. Create CA bundle files
  1. cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle
  1. Start the ETCD cluster and enable client authentication

Start by writing a goreman configuration named Procfile-single-enable-mtls, the content as:

  1. # Use goreman to run `go get github.com/mattn/goreman`
  2. etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379 --advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle
  3. etcd2: etcd --name infra2 --listen-client-urls https://127.0.0.1:22379 --advertise-client-urls https://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle
  4. etcd3: etcd --name infra3 --listen-client-urls https://127.0.0.1:32379 --advertise-client-urls https://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle

Use goreman to start the ETCD cluster:

  1. goreman -f Procfile-single-enable-mtls start > goreman.log 2>&1 &
  1. Update config.yaml

conf/config.yaml

  1. deployment:
  2. admin:
  3. admin_key
  4. - name: admin
  5. key: edd1c9f034335f136f87ad84b625c8f1
  6. role: admin
  7. admin_listen:
  8. ip: 127.0.0.1
  9. port: 9180
  10. https_admin: true
  11. admin_api_mtls:
  12. admin_ssl_ca_cert: /path/to/apisix.ca-bundle
  13. admin_ssl_cert: /path/to/foo_server.crt
  14. admin_ssl_cert_key: /path/to/foo_server.key
  15. apisix:
  16. ssl:
  17. ssl_trusted_certificate: /path/to/apisix.ca-bundle
  18. deployment:
  19. role: traditional
  20. role_traditional:
  21. config_provider: etcd
  22. etcd:
  23. host:
  24. - "https://127.0.0.1:12379"
  25. - "https://127.0.0.1:22379"
  26. - "https://127.0.0.1:32379"
  27. tls:
  28. cert: /path/to/bar_apisix.crt
  29. key: /path/to/bar_apisix.key
  30. sni: etcd.cluster.dev
  1. Test APISIX Admin API

Start APISIX, if APISIX starts successfully and there is no abnormal output in logs/error.log, it means that mTLS communication between APISIX and ETCD is normal.

Use curl to simulate a client, communicate with APISIX Admin API with mTLS, and create a route:

  1. curl -vvv \
  2. --resolve 'admin.apisix.dev:9180:127.0.0.1' https://admin.apisix.dev:9180/apisix/admin/routes/1 \
  3. --cert /path/to/foo_client.crt \
  4. --key /path/to/foo_client.key \
  5. --cacert /path/to/apisix.ca-bundle \
  6. -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
  7. {
  8. "uri": "/get",
  9. "upstream": {
  10. "type": "roundrobin",
  11. "nodes": {
  12. "httpbin.org:80": 1
  13. }
  14. }
  15. }'

A successful mTLS communication between curl and the APISIX Admin API is indicated if the following SSL handshake process is output:

  1. * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  2. * TLSv1.3 (IN), TLS handshake, Server hello (2):
  3. * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  4. * TLSv1.3 (IN), TLS handshake, Request CERT (13):
  5. * TLSv1.3 (IN), TLS handshake, Certificate (11):
  6. * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  7. * TLSv1.3 (IN), TLS handshake, Finished (20):
  8. * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  9. * TLSv1.3 (OUT), TLS handshake, Certificate (11):
  10. * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
  11. * TLSv1.3 (OUT), TLS handshake, Finished (20):
  12. * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  1. Verify APISIX proxy
  1. curl http://127.0.0.1:9080/get -i
  2. HTTP/1.1 200 OK
  3. Content-Type: application/json
  4. Content-Length: 298
  5. Connection: keep-alive
  6. Date: Tue, 26 Jul 2022 16:31:00 GMT
  7. Access-Control-Allow-Origin: *
  8. Access-Control-Allow-Credentials: true
  9. Server: APISIX/2.14.1
  10. ...

APISIX proxied the request to the /get path of the upstream httpbin.org and returned HTTP/1.1 200 OK. The whole process is working fine using CA bundle instead of CA certificate.