Graceful Pod Shutdown

When Kubernetes begins to terminate a pod, it starts by sending all containersin that pod a TERM signal. When the Linkerd proxy sidecar receives this signal,it will immediately begin a graceful shutdown where it refuses all new requestsand allows existing requests to complete before shutting down.

This means that if the pod's main container attempts to make any new networkcalls after the proxy has received the TERM signal, those network calls willfail. This also has implications for clients of the terminating pod and forjob resources.

Slow Updating Clients

Before Kubernetes terminates a pod, it first removes that pod from the endpointsresource of any services that pod is a member of. This means that clients ofthat service should stop sending traffic to the pod before it is terminated.However, certain clients can be slow to receive the endpoints update and mayattempt to send requests to the terminating pod after that pod's proxy hasalready received the TERM signal and begun graceful shutdown. Those requestswill fail.

To mitigate this, use the —wait-before-exit-seconds flag withlinkerd inject to delay the Linkerd proxy's handling of the TERM signal fora given number of seconds using a preStop hook. This delay gives slow clientsadditional time to receive the endpoints update before beginning gracefulshutdown. To achieve max benefit from the option, the main container should haveits own preStop hook with the sleep command inside which has a smaller periodthan is set for the proxy sidecar. And none of them must be bigger thanterminationGracePeriodSeconds configured for the entire pod.

For example,

       # application container
        lifecycle:
          preStop:
            exec:
              command:
                - /bin/bash
                - -c
                - sleep 20
    # for entire pod
    terminationGracePeriodSeconds: 160

Job Resources

Pods which are part of a job resource run until all of the containers in thepod complete. However, the Linkerd proxy container runs continuously until itreceives a TERM signal. This means that job pods which have been injected willcontinue to run, even once the main container has completed.

Better support forsidecar containers in Kuberneteshas been proposed and Linkerd will take advantage of this support when itbecomes available.