PodProbeMarker

FEATURE STATE: Kruise v1.3.0

Kubernetes provides three Pod lifecycle management:

  • Readiness Probe Used to determine whether the business container is ready to respond to user requests. If the probe fails, the Pod will be removed from Service Endpoints.
  • Liveness Probe Used to determine the health status of the container. If the probe fails, the kubelet will restart the container.
  • Startup Probe Used to know when a container application has started. If such a probe is configured, it disables liveness and readiness checks until it succeeds.

So the Probe capabilities provided in Kubernetes have defined specific semantics and related behaviors. In addition, there is actually a need to customize Probe semantics and related behaviors, such as:

  • GameServer defines Idle Probe to determine whether the Pod currently has a game match, if not, from the perspective of cost optimization, the Pod can be scaled down.
  • K8S Operator defines the main-secondary probe to determine the role of the current Pod (main or secondary). When upgrading, the secondary can be upgraded first, so as to achieve the behavior of selecting the main only once during the upgrade process, reducing the service interruption time during the upgrade process.

OpenKruise provides the ability to customize the Probe and return the result to the Pod Status, and the user can decide the follow-up behavior based on the probe result.

Feature-gate

PodProbeMarker feature is turned off by default, if you want to turn it on set feature-gate PodProbeMarkerGate.

  1. $ helm install kruise https://... --set featureGates="PodProbeMarkerGate=true"

Usage

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: PodProbeMarker
  3. metadata:
  4.   name: game-server-probe
  5.   namespace: ns
  6. spec:
  7.   selector:
  8.     matchLabels:
  9.       app: game-server
  10.   probes:
  11.   - name: Idle
  12.     containerName: game-server
  13.     probe:
  14.       exec: 
  15.         command:
  16.         - /home/game/idle.sh      
  17.       initialDelaySeconds: 10
  18.       timeoutSeconds: 3
  19.       periodSeconds: 10
  20.       successThreshold: 1
  21.       failureThreshold: 3
  22.     markerPolicy:
  23.     - state: Succeeded
  24.       labels:
  25.         gameserver-idle: 'true'
  26.       annotations:
  27.         controller.kubernetes.io/pod-deletion-cost: '-10'
  28.     - state: Failed
  29.       labels:
  30.         gameserver-idle: 'false'
  31.       annotations:
  32.         controller.kubernetes.io/pod-deletion-cost: '10'
  33.     podConditionType: game.io/idle
  • spec.selector: Select matching Pods based on Labels, both MatchLabels and MatchExpressions are supported. For details, please refer to: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels. Once specified, selector cannot be changed for a PodProbeMarker.
  • spec.probes
    • name: The probe name needs to be unique within the Pod and between different containers
    • containerName: The container that executes the probe
    • probe: The API definition related to probe is consistent with the native K8S probe (currently only Exec is supported). For details, please refer to: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
    • markerPolicy: According to the Probe execution result (Succeeded or Failed), patch specific Labels and Annotations to the Pod.
      • state: probe result, Succeeded or Failed
      • labels: If the result is satisfied, patch labels to the Pod
      • annotations: If the result is satisfied, patch annotations to the Pod
    • podConditionType: Save the Probe execution result (Succeeded or Failed) to the pod condition. When probe is Succeeded, pod.status.condition.status=True. Otherwise, when the probe fails to execute, pod.status.condition.status=False. When podConditionType is empty, probe execution result will not be saved to pod condition.

Note: If only one Marker Policy is defined, for example: only define State=Succeeded, Patch Labels[healthy]\=’true’. When the probe execution success, kruise will patch labels[healthy]\=’true’ to pod. And when the probe execution fails, Label[healthy] will be deleted.

How to view Probe results?

Pod Status Conditions

If podConditionType is defined, the probe result will be saved to the pod condition, where condition.type=podConditionType, as follows:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   labels:
  5.     app: game-server
  6.   name: game-server-58cb9f5688-7sbd8
  7.   namespace: ns
  8. ...
  9. status:
  10.   conditions:
  11.     # podConditionType
  12.   - type: game.io/idle
  13.     # Probe State 'Succeeded' indicates 'True', and 'Failed' indicates 'False'
  14.     status: "True"
  15.     lastProbeTime: "2022-09-09T07:13:04Z"
  16.     lastTransitionTime: "2022-09-09T07:13:04Z"
  17.     # If the probe fails to execute, the message is stderr
  18.     message: ""

This method can be used in combination with Kubernetes Readiness Gate to flexibly control whether the Pod is Ready.

Pod Metadata

If the user defines the MarkerPolicy, OpenKruise will patch the specific Metadata to the Pod, as follows:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   labels:
  5.     app: game-server
  6.     gameserver-idle: 'true'
  7.   annotations:
  8.     controller.kubernetes.io/pod-deletion-cost: '-10'
  9.   name: game-server-58cb9f5688-7sbd8
  10.   namespace: ns

OpenKruise CloneSet and Advanced StatefulSet support the ability to control upgrade priorities based on Pod Labels. At the same time, community-native Deployment and Kruise CloneSet also support scaling down priority and upgrade order based on Deletion Cost. Therefore, the Custom Probe MarkerPolicy can be combined with the above capabilities to achieve the effect of scaling down or upgrading the priority.

Pod Event

Through the pod event, you can view the historical probe execution results, as follows:

  1. $ kubectl describe pods -n ns game-server-58cb9f5688-7sbd8
  2. Events:
  3.   Type    Reason                Age                From                         Message
  4.   ----    ------                ----               ----                         -------
  5.   Normal  KruiseProbeFailed     37s (x2 over 47s)  kruise-daemon-podprobe
  6.   Normal  KruiseProbeSucceeded  36s (x2 over 37s)  kruise-daemon-podprobe