GEP-1911: Backend Protocol Selection

  • Issue: #1911
  • Status: Experimental

(See status definitions here.)

TLDR

Not all implementations support automatic protocol selection. Even in some cases protocols are disabled without an explicit opt-in (eg. websockets with Contour & NGINX). Thus application developers need the ability to specify the protocol(s) that their application supports.

Goals

  • Support protocols that can have a Gateway *Route resource as a frontend
  • Standardize Gateway API implementations on the protocols & constants defined by the Kubernetes Standard Application Protocols (KEP-3726)
  • Support backends with multiple protocols on the same port (ie. tcp/udp)

Non-Goals

  • Backend TLS (covered in GEP-1897)
  • Additional protocol specific configuration
  • Disabling Protocols

Introduction

Since Kubernetes 1.20 the core/v1.Service and core/v1.EndpointSlice resource has a stable appProtocol field. It’s purpose is to allow end-users to specify an application protocol (L7) for each service port.

Originally the use of this field in the Gateway API was rejected in GEP-1282:

v1.Service’s appProtocol field is not fit for purpose, because it is defined as accepting values either from the IANA Service Name registry, or domain-prefixed values and we need more flexibility than that.

Since then a Kubernetes enhancement proposal was created KEP-3726 to repurpose appProtocol to include a convention for protocols that are not IANA service names. This would involve prefixing protocol names with kubernetes.io/*.

Note: Kubernetes will automatically create EndpointSlices for Services that have a selector. Custom EndpointSlices can manually be created.

API Semantics

A Gateway implementation MUST recognize the Kubernetes Standard Application Protocols (KEP-3726) for specifying the protocol for a backend reference in a Gateway API *Route resource

Thus when a *Route points to a Kubernetes Service, implementations SHOULD honor the appProtocol field if it is set for the target Service Port.

At the moment there exists three defined constants:

  • kubernetes.io/h2c - HTTP/2 over cleartext as described in RFC7540
  • kubernetes.io/ws - WebSocket over cleartext as described in RFC6445
  • kubernetes.io/wss - WebSocket over TLS as described in RFC6455

New Protocols & Reserved Prefix

To add support for a new protocol it should first become a Kubernetes Standard Application Protocol by updating the KEP-3726. KEP-3726 also states the appProtocol field accepts a domain-prefixed implementation specific value. Thus, if the suggested protocol is not suited to have a kubernetes.io/* prefix, then the Gateway API MAY support the new protocol using it’s own prefix gateway.networking.k8s.io/*. Please make a PR to this GEP.

For example we may want to add a sentinel appProtocol value that prevents Gateway implementations from discovering the protocol of the application. Instead they should just refer to the Service’s protocol field. Such a constant was rejected upstream (https://github.com/kubernetes/enhancements/pull/4106) but as an example it could be defined in a future addition to this GEP as gateway.networking.k8s.io/no-sniff.

Default Protocols

If a Service appProtocol isn’t specified an implementation MAY infer the backend protocol through its own means. Implementations MAY infer the protocol from the Route type referring to the backend Service.

Absence of the appProtocol field does not imply the implementation should disable any features (eg. websocket upgrades).

Multiple Protocols on the Same Port

Only the Kubernetes Service protocol field supports multiple protocols on the same port. See the details in KEP-1435.

Implementations MAY support Kubernetes Service BackendRefs that are multiplexing TCP and UDP on the same port. Otherwise implementations MUST set *Route ResolvedRefs condition to False with the “UnsupportedProtocol” Reason with a clear message that multiplexing is not supported.

Currently Kubernetes Service API only allows different appProtocol values for the same port when protocol fields differs. At this time there seems to be interest in changing appProtocol to be a list in order to facilitate this use-case.

Supporting Protocols

If a Route is not able to send traffic to the backend using the specified protocol then the backend is considered invalid. Implementations MUST set ResolvedRefs condition to False with the Reason UnsupportedProtocol.

Implementations MAY support the following combinations below:

ServicePort ProtocolServicePort AppProtocolRoute TypeSupported
TCPkubernetes.io/h2cGRPCRouteYes [1]
TCPkubernetes.io/h2cHTTPRouteYes
TCPkubernetes.io/wsHTTPRouteYes
TCPkubernetes.io/wssTLSRouteYes
  1. GRPC works over h2c - so a GRPCRoute should be able to connect to an h2c backend

Implementations MAY support the following combinations below:

ServicePort ProtocolServicePort AppProtocolRoute TypeSupported
ServicePort ProtocolServicePort AppProtocolRoute TypeSupported
TCPkubernetes.io/wssHTTPRouteConditional [1]
  1. Only if there is a corresponding BackendTLSPolicy - see GEP-1897

Open Questions

  1. TLSRoute & UDP protocol

TLS over UDP seems to be a thing via QUIC/HTTP3 ref. Likewise there’s also DTLS. But it’s unclear if Gateway’s TLSRoute applies to an underlying UDP protocol.

  1. Websockets & HTTP/2/3

Should we upstream new constants for websocket over HTTP/2 & HTTP/3 ? HTTP/3 makes things more complicated since its supports UDP as the underlying protocol.

Alternatives

Single Meta-resource

The first pass of this GEP proposed a new meta-resource GEP-713 called BackendProtocol.

This allows end-users to specify a list of ports and a list of corresponding protocols that that single port supports.

This was dropped in favour of supporting Kubernetes Standard Application Protocols.

Multiple Protocol Meta-resources

Rather than bundle protocol details into a single resource an alternative would be to create distinct meta resources. ie. HTTP2Backend, GPRCBackend, WebsocketBackend.

The advantages of this approach are:

  • Easy to introduce new protocols
  • Definitions/types would be simpler

The disadvantages of this approach are:

  • N resources for N protocols need to be created to describe a single backend
  • No easy mechanic to specify priority of protocols

Adding Properties on Gateway Route Objects

From GEP-1282:

some types of configuration requested by users are more about defining functionality that describes capabilities of the backend more than the route you take to get to the backend.

Backend protocol is specifying capabilities. This configuration is less about routing.

Kubernetes Service - Expanding Protocol field

The protocol field on a Kubernetes service is used to specify a L4 protocol over IP. This field isn’t appropriate to describe protocols that operate at a higher ‘application’ level (eg. HTTP/GRPC etc.)

Extending Kubernetes Service

This is considered untenable due to the ‘the turnaround time for those changes can be years.’ (ref-1282)

Unstructured Data/Special Values

Unstructured data refers to using labels and annotations.

From GEP-1282:

these are very sticky and hard to get rid of once you start using them.

Special values refers to using special strings in existing Kubernetes Resources. For example Istio allows for protocol to be specified by prefixing the Kubernetes Service’s port name with the protocol (ie. http-, grpc-). This approach is limiting as it doesn’t allow for multiple protocols on the same port and future configuration per protocol. One protocol per port may be relaxed in the future see KEP 1435

Additionally, annotations are not self-documenting unlike CRD fields which can display documentation via kubectl explain

References