This document describes how to check the status of your HPAs after scaling them up or down with your load testing tool. For information on how to check the status from the Rancher UI (at least version 2.3.x), refer to Managing HPAs with the Rancher UI.

    For HPA to work correctly, service deployments should have resources request definitions for containers. Follow this hello-world example to test if HPA is working correctly.

    1. Configure kubectl to connect to your Kubernetes cluster.

    2. Copy the hello-world deployment manifest below.

      Hello World Manifest

      1. apiVersion: apps/v1beta2
      2. kind: Deployment
      3. metadata:
      4.   labels:
      5.     app: hello-world
      6.   name: hello-world
      7.   namespace: default
      8. spec:
      9.   replicas: 1
      10.   selector:
      11.     matchLabels:
      12.       app: hello-world
      13.   strategy:
      14.     rollingUpdate:
      15.       maxSurge: 1
      16.       maxUnavailable: 0
      17.     type: RollingUpdate
      18.   template:
      19.     metadata:
      20.       labels:
      21.         app: hello-world
      22.     spec:
      23.       containers:
      24.       - image: rancher/hello-world
      25.         imagePullPolicy: Always
      26.         name: hello-world
      27.         resources:
      28.           requests:
      29.             cpu: 500m
      30.             memory: 64Mi
      31.         ports:
      32.         - containerPort: 80
      33.           protocol: TCP
      34.       restartPolicy: Always
      35. ---
      36. apiVersion: v1
      37. kind: Service
      38. metadata:
      39.   name: hello-world
      40.   namespace: default
      41. spec:
      42.   ports:
      43.   - port: 80
      44.     protocol: TCP
      45.     targetPort: 80
      46.   selector:
      47.     app: hello-world

    3. Deploy it to your cluster.

      1. # kubectl create -f <HELLO_WORLD_MANIFEST>

    4. Copy one of the HPAs below based on the metric type you’re using:

      Hello World HPA: Resource Metrics

      1. apiVersion: autoscaling/v2beta1
      2. kind: HorizontalPodAutoscaler
      3. metadata:
      4.   name: hello-world
      5.   namespace: default
      6. spec:
      7.   scaleTargetRef:
      8.     apiVersion: extensions/v1beta1
      9.     kind: Deployment
      10.     name: hello-world
      11.   minReplicas: 1
      12.   maxReplicas: 10
      13.   metrics:
      14.   - type: Resource
      15.     resource:
      16.       name: cpu
      17.       targetAverageUtilization: 50
      18.   - type: Resource
      19.     resource:
      20.       name: memory
      21.       targetAverageValue: 1000Mi

      Hello World HPA: Custom Metrics

      1. apiVersion: autoscaling/v2beta1
      2. kind: HorizontalPodAutoscaler
      3. metadata:
      4.   name: hello-world
      5.   namespace: default
      6. spec:
      7.   scaleTargetRef:
      8.     apiVersion: extensions/v1beta1
      9.     kind: Deployment
      10.     name: hello-world
      11.   minReplicas: 1
      12.   maxReplicas: 10
      13.   metrics:
      14.   - type: Resource
      15.     resource:
      16.       name: cpu
      17.       targetAverageUtilization: 50
      18.   - type: Resource
      19.     resource:
      20.       name: memory
      21.       targetAverageValue: 100Mi
      22.   - type: Pods
      23.     pods:
      24.       metricName: cpu_system
      25.       targetAverageValue: 20m

    5. View the HPA info and description. Confirm that metric data is shown.

      Resource Metrics

      1. Enter the following commands.

        1. # kubectl get hpa
        2. NAME          REFERENCE                TARGETS                     MINPODS   MAXPODS   REPLICAS   AGE
        3. hello-world   Deployment/hello-world   1253376 / 100Mi, 0% / 50%   1         10        1          6m
        4. # kubectl describe hpa
        5. Name:                                                  hello-world
        6. Namespace:                                             default
        7. Labels:                                                <none>
        8. Annotations:                                           <none>
        9. CreationTimestamp:                                     Mon, 23 Jul 2018 20:21:16 +0200
        10. Reference:                                             Deployment/hello-world
        11. Metrics:                                               ( current / target )
        12.   resource memory on pods:                             1253376 / 100Mi
        13.   resource cpu on pods  (as a percentage of request):  0% (0) / 50%
        14. Min replicas:                                          1
        15. Max replicas:                                          10
        16. Conditions:
        17.   Type            Status  Reason              Message
        18.   ----            ------  ------              -------
        19.   AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
        20.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        21.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        22. Events:           <none>

        Custom Metrics

      2. Enter the following command.

        1. # kubectl describe hpa

        You should receive the output that follows.

        1. Name:                                                  hello-world
        2. Namespace:                                             default
        3. Labels:                                                <none>
        4. Annotations:                                           <none>
        5. CreationTimestamp:                                     Tue, 24 Jul 2018 18:36:28 +0200
        6. Reference:                                             Deployment/hello-world
        7. Metrics:                                               ( current / target )
        8.   resource memory on pods:                             3514368 / 100Mi
        9.   "cpu_system" on pods:                                0 / 20m
        10.   resource cpu on pods  (as a percentage of request):  0% (0) / 50%
        11. Min replicas:                                          1
        12. Max replicas:                                          10
        13. Conditions:
        14.   Type            Status  Reason              Message
        15.   ----            ------  ------              -------
        16.   AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
        17.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        18.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        19. Events:           <none>

    6. Generate a load for the service to test that your pods autoscale as intended. You can use any load-testing tool (Hey, Gatling, etc.), but we’re using Hey.

    7. Test that pod autoscaling works as intended.

      To Test Autoscaling Using Resource Metrics:

      Upscale to 2 Pods: CPU Usage Up to Target

      Use your load testing tool to scale up to two pods based on CPU Usage.

      1. View your HPA.

        1. # kubectl describe hpa

        You should receive output similar to what follows.

        1. Name:                                                  hello-world
        2. Namespace:                                             default
        3. Labels:                                                <none>
        4. Annotations:                                           <none>
        5. CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
        6. Reference:                                             Deployment/hello-world
        7. Metrics:                                               ( current / target )
        8.   resource memory on pods:                             10928128 / 100Mi
        9.   resource cpu on pods  (as a percentage of request):  56% (280m) / 50%
        10. Min replicas:                                          1
        11. Max replicas:                                          10
        12. Conditions:
        13.   Type            Status  Reason              Message
        14.   ----            ------  ------              -------
        15.   AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
        16.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
        17.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        18. Events:
        19.   Type    Reason             Age   From                       Message
        20.   ----    ------             ----  ----                       -------
        21.   Normal  SuccessfulRescale  13s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

      2. Enter the following command to confirm you’ve scaled to two pods.

        1.   # kubectl get pods

        You should receive output similar to what follows:

        1.   NAME                                                     READY     STATUS    RESTARTS   AGE
        2.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          1m
        3.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

        Upscale to 3 pods: CPU Usage Up to Target

      Use your load testing tool to upscale to 3 pods based on CPU usage with horizontal-pod-autoscaler-upscale-delay set to 3 minutes.

      1. Enter the following command.

        1. # kubectl describe hpa

        You should receive output similar to what follows

        1.   Name:                                                  hello-world
        2.   Namespace:                                             default
        3.   Labels:                                                <none>
        4.   Annotations:                                           <none>
        5.   CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
        6.   Reference:                                             Deployment/hello-world
        7.   Metrics:                                               ( current / target )
        8.     resource memory on pods:                             9424896 / 100Mi
        9.     resource cpu on pods  (as a percentage of request):  66% (333m) / 50%
        10.   Min replicas:                                          1
        11.   Max replicas:                                          10
        12.   Conditions:
        13.     Type            Status  Reason              Message
        14.     ----            ------  ------              -------
        15.     AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
        16.     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
        17.     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        18.   Events:
        19.     Type    Reason             Age   From                       Message
        20.     ----    ------             ----  ----                       -------
        21.     Normal  SuccessfulRescale  4m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
        22.     Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target

      2. Enter the following command to confirm three pods are running.

        1. # kubectl get pods

        You should receive output similar to what follows.

        1.   NAME                                                     READY     STATUS    RESTARTS   AGE
        2.   hello-world-54764dfbf8-f46kh                             0/1       Running   0          1m
        3.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          5m
        4.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

        Downscale to 1 Pod: All Metrics Below Target

      Use your load testing to scale down to 1 pod when all metrics are below target for horizontal-pod-autoscaler-downscale-delay (5 minutes by default).

      1. Enter the following command.

        1. # kubectl describe hpa

        You should receive output similar to what follows.

        1.   Name:                                                  hello-world
        2.   Namespace:                                             default
        3.   Labels:                                                <none>
        4.   Annotations:                                           <none>
        5.   CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
        6.   Reference:                                             Deployment/hello-world
        7.   Metrics:                                               ( current / target )
        8.     resource memory on pods:                             10070016 / 100Mi
        9.     resource cpu on pods  (as a percentage of request):  0% (0) / 50%
        10.   Min replicas:                                          1
        11.   Max replicas:                                          10
        12.   Conditions:
        13.     Type            Status  Reason              Message
        14.     ----            ------  ------              -------
        15.     AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
        16.     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        17.     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        18.   Events:
        19.     Type    Reason             Age   From                       Message
        20.     ----    ------             ----  ----                       -------
        21.     Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
        22.     Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
        23.     Normal  SuccessfulRescale  1s    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
    1. **To Test Autoscaling Using Custom Metrics:**
    3.  Upscale to 2 Pods: CPU Usage Up to Target
    5. Use your load testing tool to upscale two pods based on CPU usage.
    7. 1.  Enter the following command.
    9.     ```
    10.     # kubectl describe hpa
    11.     ```
    13.     You should receive output similar to what follows.
    15.     ```
    16.     Name:                                                  hello-world
    17.     Namespace:                                             default
    18.     Labels:                                                <none>
    19.     Annotations:                                           <none>
    20.     CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    21.     Reference:                                             Deployment/hello-world
    22.     Metrics:                                               ( current / target )
    23.       resource memory on pods:                             8159232 / 100Mi
    24.       "cpu_system" on pods:                                7m / 20m
    25.       resource cpu on pods  (as a percentage of request):  64% (321m) / 50%
    26.     Min replicas:                                          1
    27.     Max replicas:                                          10
    28.     Conditions:
    29.       Type            Status  Reason              Message
    30.       ----            ------  ------              -------
    31.       AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
    32.       ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    33.       ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    34.     Events:
    35.       Type    Reason             Age   From                       Message
    36.       ----    ------             ----  ----                       -------
    37.       Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    38.     ```
    40. 2.  Enter the following command to confirm two pods are running.
    42.     ```
    43.     # kubectl get pods
    44.     ```
    46.     You should receive output similar to what follows.
    48.     ```
    49.         NAME                           READY     STATUS    RESTARTS   AGE
    50.         hello-world-54764dfbf8-5pfdr   1/1       Running   0          3s
    51.         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    52.     ```
    54.  Upscale to 3 Pods: CPU Usage Up to Target
    56. Use your load testing tool to scale up to three pods when the cpu\_system usage limit is up to target.
    58. 1.  Enter the following command.
    60.     ```
    61.     # kubectl describe hpa
    62.     ```
    64.     You should receive output similar to what follows:
    66.     ```
    67.       Name:                                                  hello-world
    68.       Namespace:                                             default
    69.       Labels:                                                <none>
    70.       Annotations:                                           <none>
    71.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    72.       Reference:                                             Deployment/hello-world
    73.       Metrics:                                               ( current / target )
    74.         resource memory on pods:                             8374272 / 100Mi
    75.         "cpu_system" on pods:                                27m / 20m
    76.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
    77.       Min replicas:                                          1
    78.       Max replicas:                                          10
    79.       Conditions:
    80.         Type            Status  Reason              Message
    81.         ----            ------  ------              -------
    82.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
    83.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    84.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    85.       Events:
    86.         Type    Reason             Age   From                       Message
    87.         ----    ------             ----  ----                       -------
    88.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    89.         Normal  SuccessfulRescale  3s    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    90.     ```
    92. 2.  Enter the following command to confirm three pods are running.
    94.     ```
    95.     # kubectl get pods
    96.     ```
    98.     You should receive output similar to what follows:
    100.     ```
    101.       # kubectl get pods
    102.       NAME                           READY     STATUS    RESTARTS   AGE
    103.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          3m
    104.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
    105.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    106.     ```
    108.  Upscale to 4 Pods: CPU Usage Up to Target
    110. Use your load testing tool to upscale to four pods based on CPU usage. `horizontal-pod-autoscaler-upscale-delay` is set to three minutes by default.
    112. 1.  Enter the following command.
    114.     ```
    115.     # kubectl describe hpa
    116.     ```
    118.     You should receive output similar to what follows.
    120.     ```
    121.       Name:                                                  hello-world
    122.       Namespace:                                             default
    123.       Labels:                                                <none>
    124.       Annotations:                                           <none>
    125.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    126.       Reference:                                             Deployment/hello-world
    127.       Metrics:                                               ( current / target )
    128.         resource memory on pods:                             8374272 / 100Mi
    129.         "cpu_system" on pods:                                27m / 20m
    130.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
    131.       Min replicas:                                          1
    132.       Max replicas:                                          10
    133.       Conditions:
    134.         Type            Status  Reason              Message
    135.         ----            ------  ------              -------
    136.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
    137.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    138.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    139.       Events:
    140.         Type    Reason             Age   From                       Message
    141.         ----    ------             ----  ----                       -------
    142.         Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    143.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    144.         Normal  SuccessfulRescale  4s    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
    145.     ```
    147. 2.  Enter the following command to confirm four pods are running.
    149.     ```
    150.     # kubectl get pods
    151.     ```
    153.     You should receive output similar to what follows.
    155.     ```
    156.       NAME                           READY     STATUS    RESTARTS   AGE
    157.       hello-world-54764dfbf8-2p9xb   1/1       Running   0          5m
    158.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          2m
    159.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
    160.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    161.     ```
    163.  Downscale to 1 Pod: All Metrics Below Target
    165. Use your load testing tool to scale down to one pod when all metrics below target for `horizontal-pod-autoscaler-downscale-delay`.
    167. 1.  Enter the following command.
    169.     ```
    170.     # kubectl describe hpa
    171.     ```
    173.     You should receive similar output to what follows.
    175.     ```
    176.         Name:                                                  hello-world
    177.         Namespace:                                             default
    178.         Labels:                                                <none>
    179.         Annotations:                                           <none>
    180.         CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    181.         Reference:                                             Deployment/hello-world
    182.         Metrics:                                               ( current / target )
    183.           resource memory on pods:                             8101888 / 100Mi
    184.           "cpu_system" on pods:                                8m / 20m
    185.           resource cpu on pods  (as a percentage of request):  0% (0) / 50%
    186.         Min replicas:                                          1
    187.         Max replicas:                                          10
    188.         Conditions:
    189.           Type            Status  Reason              Message
    190.           ----            ------  ------              -------
    191.           AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
    192.           ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
    193.           ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    194.         Events:
    195.           Type    Reason             Age   From                       Message
    196.           ----    ------             ----  ----                       -------
    197.           Normal  SuccessfulRescale  10m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    198.           Normal  SuccessfulRescale  8m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    199.           Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
    200.           Normal   SuccessfulRescale             13s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
    201.     ```
    203. 2.  Enter the following command to confirm a single pods is running.
    205.     ```
    206.         # kubectl get pods
    207.     ```
    209.     You should receive output similar to what follows.
    211.     ```
    212.         NAME                           READY     STATUS    RESTARTS   AGE
    213.         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    214.     ```