Pod Group Status

In Coscheduling v1alph1 design, PodGroup’s statusonly includes counters of related pods which is not enough for PodGroup lifecycle management. More informationabout PodGroup’s status will be introduced in this design doc for lifecycle management, e.g. PodGroupPhase.

Function Detail

To include more information for PodGroup current status/phase, the following types are introduced:

  1. // PodGroupPhase is the phase of a pod group at the current time.
  2. type PodGroupPhase string
  4. // These are the valid phase of podGroups.
  5. const (
  6.     // PodPending means the pod group has been accepted by the system, but scheduler can not allocate
  7.     // enough resources to it.
  8.     PodGroupPending PodGroupPhase = "Pending"
  10.     // PodRunning means `spec.minMember` pods of PodGroups has been in running phase.
  11.     PodGroupRunning PodGroupPhase = "Running"
  13.     // PodGroupUnknown means part of `spec.minMember` pods are running but the other part can not
  14.     // be scheduled, e.g. not enough resource; scheduler will wait for related controller to recover it.
  15.     PodGroupUnknown PodGroupPhase = "Unknown"
  16. )
  18. type PodGroupConditionType string
  20. const (
  21.     PodGroupUnschedulableType PodGroupConditionType = "Unschedulable"
  22. )
  24. // PodGroupCondition contains details for the current state of this pod group.
  25. type PodGroupCondition struct {
  26.     // Type is the type of the condition
  27.     Type PodGroupConditionType `json:"type,omitempty" protobuf:"bytes,1,opt,name=type"`
  29.     // Status is the status of the condition.
  30.     Status v1.ConditionStatus `json:"status,omitempty" protobuf:"bytes,2,opt,name=status"`
  32.     // The ID of condition transition.
  33.     TransitionID string `json:"transitionID,omitempty" protobuf:"bytes,3,opt,name=transitionID"`
  35.     // Last time the phase transitioned from another to current phase.
  36.     // +optional
  37.     LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"`
  39.     // Unique, one-word, CamelCase reason for the phase's last transition.
  40.     // +optional
  41.     Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"`
  43.     // Human-readable message indicating details about last transition.
  44.     // +optional
  45.     Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"`
  46. }
  48. const (
  49.     // PodFailedReason is probed if pod of PodGroup failed
  50.     PodFailedReason string = "PodFailed"
  52.     // PodDeletedReason is probed if pod of PodGroup deleted
  53.     PodDeletedReason string = "PodDeleted"
  55.     // NotEnoughResourcesReason is probed if there're not enough resources to schedule pods
  56.     NotEnoughResourcesReason string = "NotEnoughResources"
  58.     // NotEnoughPodsReason is probed if there're not enough tasks compared to `spec.minMember`
  59.     NotEnoughPodsReason string = "NotEnoughTasks"
  60. )
  62. // PodGroupStatus represents the current state of a pod group.
  63. type PodGroupStatus struct {
  64.     // Current phase of PodGroup.
  65.     Phase PodGroupPhase `json:"phase,omitempty" protobuf:"bytes,1,opt,name=phase"`
  67.     // The conditions of PodGroup.
  68.     // +optional
  69.     Conditions []PodGroupCondition `json:"conditions,omitempty" protobuf:"bytes,2,opt,name=conditions"`
  71.     // The number of actively running pods.
  72.     // +optional
  73.     Running int32 `json:"running,omitempty" protobuf:"bytes,3,opt,name=running"`
  75.     // The number of pods which reached phase Succeeded.
  76.     // +optional
  77.     Succeeded int32 `json:"succeeded,omitempty" protobuf:"bytes,4,opt,name=succeeded"`
  79.     // The number of pods which reached phase Failed.
  80.     // +optional
  81.     Failed int32 `json:"failed,omitempty" protobuf:"bytes,5,opt,name=failed"`
  82. }

According to the PodGroup’s lifecycle, the following phase/state transactions are reasonable. And relatedreasons will be appended to Reason field.

FromToReason
PendingRunningWhen every pods of spec.minMember are running
RunningUnknownWhen some pods of spec.minMember are restarted but can not be rescheduled
UnknownPendingWhen all pods (spec.minMember) in PodGroups are deleted

Feature Interaction

Cluster AutoScale

Cluster Autoscaler is a tool thatautomatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

  • there are pods that failed to run in the cluster due to insufficient resources,
  • there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.When Cluster-Autoscaler scale-out a new node, it leverage predicates in scheduler to check whether the new node can bescheduled. But Coscheduling is not an implementation of predicates for now; so it’ll not work well together withCluster-Autoscaler right now. Alternative solution will be proposed later for that.

Operators/Controllers

The lifecycle of PodGroup are managed by operators/controllers, the scheduler only probes related state forcontrollers. For example, if PodGroup is Unknown for MPI job, the controller need to re-start all pods in PodGroup.

Reference