Automatically scaling pods based on custom metrics

As a developer, you can use the custom metrics autoscaler to specify how OKD should automatically increase or decrease the number of pods for a deployment, stateful set, custom resource, or job based on custom metrics that are not based only on CPU or memory.

The Custom Metrics Autoscaler Operator for Red Hat OpenShift is an optional operator, based on the Kubernetes Event Driven Autoscaler (KEDA), that allows workloads to be scaled using additional metrics sources other than pod metrics.

Custom Metrics Autoscaler Operator release notes

The release notes for the Custom Metrics Autoscaler Operator for Red Hat Openshift describe new features and enhancements, deprecated features, and known issues.

The Custom Metrics Autoscaler Operator uses the Kubernetes-based Event Driven Autoscaler (KEDA) and is built on top of the OKD horizontal pod autoscaler (HPA).

Supported versions

The following table defines the Custom Metrics Autoscaler Operator versions for each OKD version.

Custom Metrics Autoscaler Operator 2.8.2-174 release notes

This release of the Custom Metrics Autoscaler Operator 2.8.2-174 provides new features and bug fixes for running the Operator in an OKD cluster. The components of the Custom Metrics Autoscaler Operator 2.8.2-174 were released in RHEA-2023:1683.

New features and enhancements

Operator upgrade support

You can now upgrade from a prior version of the Custom Metrics Autoscaler Operator. See “Changing the update channel for an Operator” in the “Additional resources” for information on upgrading an Operator.

must-gather support

You can now collect data about the Custom Metrics Autoscaler Operator and its components by using the OKD must-gather tool. Currently, the process for using the must-gather tool with the Custom Metrics Autoscaler is different than for other operators. See “Gathering debugging data in the “Additional resources” for more information.

Custom Metrics Autoscaler Operator 2.8.2 release notes

This release of the Custom Metrics Autoscaler Operator 2.8.2 provides new features and bug fixes for running the Operator in an OKD cluster. The components of the Custom Metrics Autoscaler Operator 2.8.2 were released in RHSA-2023:1042.

New features and enhancements

Audit Logging

You can now gather and view audit logs for the Custom Metrics Autoscaler Operator and its associated components. Audit logs are security-relevant chronological sets of records that document the sequence of activities that have affected the system by individual users, administrators, or other components of the system.

Scale applications based on Apache Kafka metrics

You can now use the KEDA Apache kafka trigger/scaler to scale deployments based on an Apache Kafka topic.

Scale applications based on CPU metrics

You can now use the KEDA CPU trigger/scaler to scale deployments based on CPU metrics.

Scale applications based on memory metrics

You can now use the KEDA memory trigger/scaler to scale deployments based on memory metrics.

Additional resources

• Changing the update channel for an Operator

• Gathering debugging data

• Configuring audit logging

• Understanding the CPU trigger

• Understanding the memory trigger

• Understanding the Kafka trigger

Understanding the custom metrics autoscaler

The Custom Metrics Autoscaler Operator scales your pods up and down based on custom, external metrics from specific applications. Your other applications continue to use other scaling methods. You configure triggers, also known as scalers, which are the source of events and metrics that the custom metrics autoscaler uses to determine how to scale. The custom metrics autoscaler uses a metrics API to convert the external metrics to a form that OKD can use. The custom metrics autoscaler creates a horizontal pod autoscaler (HPA) that performs the actual scaling.

To use the custom metrics autoscaler, you create a ScaledObject or ScaledJob object, which is a custom resource (CR) that defines the scaling metadata. You specify the deployment or job to scale, the source of the metrics to scale on (trigger), and other parameters such as the minimum and maximum replica counts allowed.

The custom metrics autoscaler, unlike the HPA, can scale to zero. If you set the minReplicaCount value in the custom metrics autoscaler CR to 0, the custom metrics autoscaler scales the workload down from 1 to 0 replicas to or up from 0 replicas to 1. This is known as the activation phase. After scaling up to 1 replica, the HPA takes control of the scaling. This is known as the scaling phase.

Some triggers allow you to change the number of replicas that are scaled by the cluster metrics autoscaler. In all cases, the parameter to configure the activation phase always uses the same phrase, prefixed with activation. For example, if the threshold parameter configures scaling, activationThreshold would configure activation. Configuring the activation and scaling phases allows you more flexibility with your scaling policies. For example, you could configure a higher activation phase to prevent scaling up or down if the metric is particularly low.

The activation value has more priority than the scaling value in case of different decisions for each. For example, if the threshold is set to 10, and the activationThreshold is 50, if the metric reports 40, the scaler is not active and the pods are scaled to zero even if the HPA requires 4 instances.

You can verify that the autoscaling has taken place by reviewing the number of pods in your custom resource or by reviewing the Custom Metrics Autoscaler Operator logs for messages similar to the following:

Successfully set ScaleTarget replica count

Successfully updated ScaleTarget

You can temporarily pause the autoscaling of a workload object, if needed. For example, you could pause autoscaling before performing cluster maintenance.

Installing the custom metrics autoscaler

You can use the OKD web console to install the Custom Metrics Autoscaler Operator.

The installation creates five CRDs:

• ClusterTriggerAuthentication

• KedaController

• ScaledJob

• ScaledObject

• TriggerAuthentication

Prerequisites

• Ensure that you have downloaded the pull secret from the Red Hat OpenShift Cluster Manager as shown in Obtaining the installation program in the installation documentation for your platform.

  If you have the pull secret, add the redhat-operators catalog to the OperatorHub custom resource (CR) as shown in Configuring OKD to use Red Hat Operators.

• If you use the community KEDA:

  • Uninstall the community KEDA. You cannot run both KEDA and the custom metrics autoscaler on the same OKD cluster.

  • Remove the KEDA 1.x custom resource definitions by running the following commands:

    $ oc delete crd scaledobjects.keda.k8s.io

    $ oc delete crd triggerauthentications.keda.k8s.io

Procedure

1. In the OKD web console, click Operators → OperatorHub.

2. Choose Custom Metrics Autoscaler from the list of available Operators, and click Install.

3. On the Install Operator page, ensure that the All namespaces on the cluster (default) option is selected for Installation Mode. This installs the Operator in all namespaces.

4. Ensure that the openshift-keda namespace is selected for Installed Namespace. OKD creates the namespace, if not present in your cluster.

5. Click Install.

6. Verify the installation by listing the Custom Metrics Autoscaler Operator components:

   1. Navigate to Workloads → Pods.

   2. Select the openshift-keda project from the drop-down menu and verify that the custom-metrics-autoscaler-operator-* pod is running.

   3. Navigate to Workloads → Deployments to verify that the custom-metrics-autoscaler-operator deployment is running.

7. Optional: Verify the installation in the OpenShift CLI using the following commands:

   $ oc get all -n openshift-keda

   The output appears similar to the following:

   Example output

   NAME                                                      READY   STATUS    RESTARTS   AGE
   pod/custom-metrics-autoscaler-operator-5fd8d9ffd8-xt4xp   1/1     Running   0          18m
   NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/custom-metrics-autoscaler-operator   1/1     1            1           18m
   NAME                                                            DESIRED   CURRENT   READY   AGE
   replicaset.apps/custom-metrics-autoscaler-operator-5fd8d9ffd8   1         1         1       18m

8. Install the KedaController custom resource, which creates the required CRDs:

   1. In the OKD web console, click Operators → Installed Operators.

   2. Click Custom Metrics Autoscaler.

   3. On the Operator Details page, click the KedaController tab.

   4. On the KedaController tab, click Create KedaController and edit the file.

      kind: KedaController
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: keda
        namespace: openshift-keda
      spec:
        watchNamespace: '' (1)
        operator:
          logLevel: info (2)
          logEncoder: console (3)
        metricsServer:
          logLevel: '0' (4)
          auditConfig: (5)
            logFormat: "json"
            logOutputVolumeClaim: "persistentVolumeClaimName"
            policy:
              rules:
              - level: Metadata
              omitStages: "RequestReceived"
              omitManagedFields: false
            lifetime:
              maxAge: "2"
              maxBackup: "1"
              maxSize: "50"
        serviceAccount: {}

      1	Specifies the namespaces that the custom autoscaler should watch. Enter names in a comma-separated list. Omit or set empty to watch all namespaces. The default is empty.
      2	Specifies the level of verbosity for the Custom Metrics Autoscaler Operator log messages. The allowed values are debug, info, error. The default is info.
      3	Specifies the logging format for the Custom Metrics Autoscaler Operator log messages. The allowed values are console or json. The default is console.
      4	Specifies the logging level for the Custom Metrics Autoscaler Metrics Server. The allowed values are 0 for info and 4 or debug. The default is 0.
      5	Activates audit logging for the Custom Metrics Autoscaler Operator and specifies the audit policy to use, as described in the “Configuring audit logging” section.

   5. Click Create to create the KEDAController.

Understanding the custom metrics autoscaler triggers

Triggers, also known as scalers, provide the metrics that the Custom Metrics Autoscaler Operator uses to scale your pods.

You use a ScaledObject or ScaledJob custom resource to configure triggers for specific objects, as described in the sections that follow.

Understanding the Prometheus trigger

You can scale pods based on Prometheus metrics, which can use the installed OKD monitoring or an external Prometheus server as the metrics source. See Additional resources for information on the configurations required to use the OKD monitoring as a source for metrics.

Example scaled object with a Prometheus target

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prom-scaledobject
  namespace: my-namespace
spec:
 ...
  triggers:
  - type: prometheus (1)
    metadata:
      serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 (2)
      namespace: kedatest (3)
      metricName: http_requests_total (4)
      threshold: '5' (5)
      query: sum(rate(http_requests_total{job="test-app"}[1m])) (6)
      authModes: "basic" (7)
      cortexOrgID: my-org (8)
      ignoreNullValues: false (9)
      unsafeSsl: "false" (10)

Understanding the CPU trigger

You can scale pods based on CPU metrics. This trigger uses cluster metrics as the source for metrics.

The custom metrics autoscaler scales the pods associated with an object to maintain the CPU usage that you specify. The autoscaler increases or decreases the number of replicas between the minimum and maximum numbers to maintain the specified CPU utilization across all pods. The memory trigger considers the memory utilization of the entire pod. If the pod has multiple containers, the memory utilization is the sum of all of the containers.

Example scaled object with a CPU target

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: cpu-scaledobject
  namespace: my-namespace
spec:
 ...
  triggers:
  - type: cpu (1)
    metricType: Utilization (2)
    metadata:
      value: "60" (3)
      containerName: "api" (4)

Understanding the memory trigger

You can scale pods based on memory metrics. This trigger uses cluster metrics as the source for metrics.

The custom metrics autoscaler scales the pods associated with an object to maintain the average memory usage that you specify. The autoscaler increases and decreases the number of replicas between the minimum and maximum numbers to maintain the specified memory utilization across all pods. The memory trigger considers the memory utilization of entire pod. If the pod has multiple containers, the memory utilization is the sum of all of the containers.

Example scaled object with a memory target

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: memory-scaledobject
  namespace: my-namespace
spec:
 ...
  triggers:
  - type: memory (1)
    metricType: Utilization (2)
    metadata:
      value: "60" (3)
      containerName: "api" (4)

Understanding the Kafka trigger

You can scale pods based on an Apache Kafka topic or other services that support the Kafka protocol. The custom metrics autoscaler does not scale higher than the number of Kafka partitions, unless you set the allowIdleConsumers parameter to true in the scaled object or scaled job.

Example scaled object with a Kafka target

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: my-namespace
spec:
 ...
  triggers:
  - type: kafka (1)
    metadata:
      topic: my-topic (2)
      bootstrapServers: my-cluster-kafka-bootstrap.openshift-operators.svc:9092 (3)
      consumerGroup: my-group (4)
      lagThreshold: '10' (5)
      activationLagThreshold (6)
      offsetResetPolicy: 'latest' (7)
      allowIdleConsumers: true (8)
      scaleToZeroOnInvalidOffset: false (9)
      excludePersistentLag: false (10)
      version: 1.0.0 (11)
      partitionLimitation: '1,2,10-20,31' (12)

Additional resources

• Configuring the custom metrics autoscaler to use OKD monitoring

Understanding custom metrics autoscaler trigger authentications

A trigger authentication allows you to include authentication information in a scaled object or a scaled job that can be used by the associated containers. You can use trigger authentications to pass OKD secrets, platform-native pod authentication mechanisms, environment variables, and so on.

You define a TriggerAuthentication object in the same namespace as the object that you want to scale. That trigger authentication can be used only by objects in that namespace.

Alternatively, to share credentials between objects in multiple namespaces, you can create a ClusterTriggerAuthentication object that can be used across all namespaces.

Trigger authentications and cluster trigger authentication use the same configuration. However, a cluster trigger authentication requires an additional kind parameter in the authentication reference of the scaled object.

Example trigger authentication with a secret

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: secret-triggerauthentication
  namespace: my-namespace (1)
spec:
  secretTargetRef: (2)
  - parameter: user-name (3)
    name: my-secret (4)
    key: USER_NAME (5)
  - parameter: password
    name: my-secret
    key: USER_PASSWORD

Example cluster trigger authentication with a secret

kind: ClusterTriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata: (1)
  name: secret-cluster-triggerauthentication
spec:
  secretTargetRef: (2)
  - parameter: user-name (3)
    name: secret-name (4)
    key: USER_NAME (5)
  - parameter: user-password
    name: secret-name
    key: USER_PASSWORD

Example trigger authentication with a token

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: token-triggerauthentication
  namespace: my-namespace (1)
spec:
  secretTargetRef: (2)
  - parameter: bearerToken (3)
    name: my-token-2vzfq (4)
    key: token (5)
  - parameter: ca
    name: my-token-2vzfq
    key: ca.crt

Example trigger authentication with an environment variable

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: env-var-triggerauthentication
  namespace: my-namespace (1)
spec:
  env: (2)
  - parameter: access_key (3)
    name: ACCESS_KEY (4)
    containerName: my-container (5)

Example trigger authentication with pod authentication providers

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: pod-id-triggerauthentication
  namespace: my-namespace (1)
spec:
  podIdentity: (2)
    provider: aws-eks (3)

Additional resources

• For information on OKD secrets, see Providing sensitive data to pods.

Using trigger authentications

You use trigger authentications and cluster trigger authentications by using a custom resource to create the authentication, then add a reference to a scaled object or scaled job.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

• If you are using a secret, the Secret object must exist, for example:

  Example secret

  apiVersion: v1
  kind: Secret
  metadata:
    name: my-secret
  data:
    user-name: <base64_USER_NAME>
    password: <base64_USER_PASSWORD>

Procedure

1. Create the TriggerAuthentication or ClusterTriggerAuthentication object.

   1. Create a YAML file that defines the object:

      Example trigger authentication with a secret

      kind: TriggerAuthentication
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: prom-triggerauthentication
        namespace: my-namespace
      spec:
        secretTargetRef:
        - parameter: user-name
          name: my-secret
          key: USER_NAME
        - parameter: password
          name: my-secret
          key: USER_PASSWORD

   2. Create the TriggerAuthentication object:

      $ oc create -f <file-name>.yaml

2. Create or edit a ScaledObject YAML file:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: scaledobject
     namespace: my-namespace
   spec:
     scaleTargetRef:
       name: example-deployment
     maxReplicaCount: 100
     minReplicaCount: 0
     pollingInterval: 30
     triggers:
     - authenticationRef:
       type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest # replace <NAMESPACE>
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "basic"
       - authenticationRef: (1)
           name: prom-triggerauthentication
         metadata:
           name: prom-triggerauthentication
         type: object
       - authenticationRef: (2)
           name: prom-cluster-triggerauthentication
           kind: ClusterTriggerAuthentication
         metadata:
           name: prom-cluster-triggerauthentication
         type: object

   1	Optional: Specify a trigger authentication.
   2	Optional: Specify a cluster trigger authentication. You must include the kind: ClusterTriggerAuthentication parameter.

   It is not necessary to specify both a namespace trigger authentication and a cluster trigger authentication.

3. Create the object. For example:

   $ oc apply -f <file-name>

Configuring the custom metrics autoscaler to use OKD monitoring

You can use the installed OKD Prometheus monitoring as a source for the metrics used by the custom metrics autoscaler. However, there are some additional configurations you must perform.

You must perform the following tasks, as described in this section:

• Create a service account to get a token.

• Create a role.

• Add that role to the service account.

• Reference the token in the trigger authentication object used by Prometheus.

Prerequisites

• OKD monitoring must be installed.

• Monitoring of user-defined workloads must be enabled in OKD monitoring, as described in the Creating a user-defined workload monitoring config map section.

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Change to the project with the object you want to scale:

   $ oc project my-project

2. Use the following command to create a service account, if your cluster does not have one:

   $ oc create serviceaccount <service_account>

   where:

   <service_account>

   Specifies the name of the service account.

3. Use the following command to locate the token assigned to the service account:

   $ oc describe serviceaccount <service_account>

   where:

   <service_account>

   Specifies the name of the service account.

   Example output

   Name:                thanos
   Namespace:           my-project
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-nnwgj
   Mountable secrets:   thanos-dockercfg-nnwgj
   Tokens:              thanos-token-9g4n5 (1)
   Events:              <none>

   1	Use this token in the trigger authentication.

4. Create a trigger authentication with the service account token:

   1. Create a YAML file similar to the following:

      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
      spec:
        secretTargetRef: (1)
        - parameter: bearerToken (2)
          name: thanos-token-9g4n5 (3)
          key: token (4)
        - parameter: ca
          name: thanos-token-9g4n5
          key: ca.crt

      1	Specifies that this object uses a secret for authorization.
      2	Specifies the authentication parameter to supply by using the token.
      3	Specifies the name of the token to use.
      4	Specifies the key in the token to use with the specified parameter.

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

5. Create a role for reading Thanos metrics:

   1. Create a YAML file with the following parameters:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

6. Create a role binding for reading Thanos metrics:

   1. Create a YAML file similar to the following:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: thanos-metrics-reader (1)
        namespace: my-project (2)
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: thanos-metrics-reader
      subjects:
      - kind: ServiceAccount
        name: thanos (3)
        namespace: my-project (4)

      1	Specifies the name of the role you created.
      2	Specifies the namespace of the object you want to scale.
      3	Specifies the name of the service account to bind to the role.
      4	Specifies the namespace of the object you want to scale.

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

You can now deploy a scaled object or scaled job to enable autoscaling for your application, as described in the following sections. To use OKD monitoring as the source, in the trigger, or scaler, specify the prometheus type and use https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 as the serverAddress.

Additional resources

• For information on enabing monitoring of user-defined workloads, see Creating a user-defined workload monitoring config map.

Pausing the custom metrics autoscaler for a workload

You can pause the autoscaling of a workload, as needed, by adding the autoscaling.keda.sh/paused-replicas annotation to the custom metrics autoscaler for that workload. The custom metrics autoscaler scales the replicas for that workload to the specified value and pauses autoscaling until the annotation is removed.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    autoscaling.keda.sh/paused-replicas: "4"
...

To restart autoscaling, edit the ScaledObject CR to remove the annotation.

For example, you might want to pause autoscaling before performing cluster maintenance or to avoid resource starvation by removing non-mission-critical workloads.

Procedure

1. Use the following command to edit the ScaledObject CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Add the autoscaling.keda.sh/paused-replicas annotation with any value:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" (1)
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: "65729"
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1	Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling.

Configuring audit logging

You can gather audit logs, which are a security-relevant chronological set of records documenting the sequence of activities that have affected the system by individual users, administrators, or other components of the system.

For example, audit logs can help you understand where an autoscaling request is coming from. This is key information when backends are getting overloaded by autoscaling requests made by user applications and you need to determine which is the troublesome application. You can configure auditing for the Custom Metrics Autoscaler Operator by editing the KedaController custom resource. The logs are sent to an audit log file on a volume that is secured by using a persistent volume claim in the KedaController CR.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Edit the KedaController custom resource to add the auditConfig stanza:

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: openshift-keda
   spec:
    ...
     metricsServer:
    ...
       auditConfig:
         logFormat: "json" (1)
         logOutputVolumeClaim: "pvc-audit-log" (2)
         policy:
           rules: (3)
           - level: Metadata
           omitStages: "RequestReceived" (4)
           omitManagedFields: false (5)
         lifetime: (6)
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"

   1	Specifies the output format of the audit log, either legacy or json.
   2	Specifies an existing persistent volume claim for storing the log data. All requests coming to the API server are logged to this persistent volume claim. If you leave this field empty, the log data is sent to stdout.
   3	Specifies which events should be recorded and what data they should include: None: Do not log events. Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. This is the default. Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests. RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.
   4	Specifies stages for which no event is created.
   5	Specifies whether to omit the managed fields of the request and response bodies from being written to the API audit log, either true to omit the fields or false to include the fields.
   6	Specifies the size and lifespan of the audit logs. maxAge: The maximum number of days to retain audit log files, based on the timestamp encoded in their filename. maxBackup: The maximum number of audit log files to retain. Set to 0 to retain all audit log files. maxSize: The maximum size in megabytes of an audit log file before it gets rotated.

Verification

1. View the audit log file directly:

   1. Obtain the name of the keda-metrics-apiserver-* pod:

      oc get pod -n openshift-keda

      Example output

      NAME                                                  READY   STATUS    RESTARTS   AGE
      custom-metrics-autoscaler-operator-5cb44cd75d-9v4lv   1/1     Running   0          8m20s
      keda-metrics-apiserver-65c7cc44fd-rrl4r               1/1     Running   0          2m55s
      keda-operator-776cbb6768-zpj5b                        1/1     Running   0          2m55s

   2. View the log data by using a command similar to the following:

      $ oc logs keda-metrics-apiserver-<hash>|grep -i metadata (1)

      1	Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      $ oc logs keda-metrics-apiserver-65c7cc44fd-rrl4r|grep -i metadata

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4c81d41b-3dab-4675-90ce-20b87ce24013","stage":"ResponseComplete","requestURI":"/healthz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.131.0.1"],"userAgent":"kube-probe/1.26","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-02-16T13:00:03.554567Z","stageTimestamp":"2023-02-16T13:00:03.555032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
       ...

2. Alternatively, you can view a specific log:

   1. Use a command similar to the following to log into the keda-metrics-apiserver-* pod:

      $ oc rsh pod/keda-metrics-apiserver-<hash> -n openshift-keda

      For example:

      $ oc rsh pod/keda-metrics-apiserver-65c7cc44fd-rrl4r -n openshift-keda

   2. Change to the /var/audit-policy/ directory:

      sh-4.4$ cd /var/audit-policy/

   3. List the available logs:

      sh-4.4$ ls

      Example output

      log-2023.02.17-14:50  policy.yaml

   4. View the log, as needed:

      sh-4.4$ cat <log_name>/<pvc_name>|grep -i <log_level> (1)

      1	Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      sh-4.4$ cat log-2023.02.17-14:50/pvc-audit-log|grep -i Request

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"63e7f68c-04ec-4f4d-8749-bf1656572a41","stage":"ResponseComplete","requestURI":"/openapi/v2","verb":"get","user":{"username":"system:aggregator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"responseStatus":{"metadata":{},"code":304},"requestReceivedTimestamp":"2023-02-17T13:12:55.035478Z","stageTimestamp":"2023-02-17T13:12:55.038346Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
       ...

Additional resources

• Configuring audit logging

Gathering debugging data

You can use the must-gather tool to collect data about the Custom Metrics Autoscaler Operator and its components, including:

• The openshift-keda namespace and its child objects.

• The Custom Metric Autoscaler Operator installation objects.

• The Custom Metric Autoscaler Operator CRD objects.

The following command runs the must-gather tool for the Custom Metrics Autoscaler Operator:

$ oc adm must-gather --image="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
-n openshift-marketplace \
-o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

Prerequisites

• Access to the cluster as a user with the cluster-admin role.

• The OKD CLI (oc) installed.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

   If your cluster is using a restricted network, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters on restricted networks, you must import the default must-gather image as an image stream by running the following command. $ oc import-image is/must-gather -n openshift

2. Perform one of the following:

   • To get only the Custom Metrics Autoscaler Operator must-gather data, use the following command:

     $ oc adm must-gather --image="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
     -n openshift-marketplace \
     -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     The custom image for the must-gather command is pulled directly from the Operator package manifests, so that it works on any cluster where the Custom Metric Autoscaler Operator is available.

   • To gather the default must-gather data in addition to the Custom Metric Autoscaler Operator information:

     1. Use the following command to obtain the Custom Metrics Autoscaler Operator image and set it as an environment variable:

        $ IMAGE="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
          -n openshift-marketplace \
          -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     2. Use the oc adm must-gather with the Custom Metrics Autoscaler Operator image:

        $ oc adm must-gather --image-stream=openshift/must-gather --image=${IMAGE}

Example must-gather output for the Custom Metric Autoscaler:
```
└── openshift-keda
    ├── apps
    │   ├── daemonsets.yaml
    │   ├── deployments.yaml
    │   ├── replicasets.yaml
    │   └── statefulsets.yaml
    ├── apps.openshift.io
    │   └── deploymentconfigs.yaml
    ├── autoscaling
    │   └── horizontalpodautoscalers.yaml
    ├── batch
    │   ├── cronjobs.yaml
    │   └── jobs.yaml
    ├── build.openshift.io
    │   ├── buildconfigs.yaml
    │   └── builds.yaml
    ├── core
    │   ├── configmaps.yaml
    │   ├── endpoints.yaml
    │   ├── events.yaml
    │   ├── persistentvolumeclaims.yaml
    │   ├── pods.yaml
    │   ├── replicationcontrollers.yaml
    │   ├── secrets.yaml
    │   └── services.yaml
    ├── discovery.k8s.io
    │   └── endpointslices.yaml
    ├── image.openshift.io
    │   └── imagestreams.yaml
    ├── k8s.ovn.org
    │   ├── egressfirewalls.yaml
    │   └── egressqoses.yaml
    ├── keda.sh
    │   ├── kedacontrollers
    │   │   └── keda.yaml
    │   ├── scaledobjects
    │   │   └── example-scaledobject.yaml
    │   └── triggerauthentications
    │       └── example-triggerauthentication.yaml
    ├── monitoring.coreos.com
    │   └── servicemonitors.yaml
    ├── networking.k8s.io
    │   └── networkpolicies.yaml
    ├── openshift-keda.yaml
    ├── pods
    │   ├── custom-metrics-autoscaler-operator-58bd9f458-ptgwx
    │   │   ├── custom-metrics-autoscaler-operator
    │   │   │   └── custom-metrics-autoscaler-operator
    │   │   │       └── logs
    │   │   │           ├── current.log
    │   │   │           ├── previous.insecure.log
    │   │   │           └── previous.log
    │   │   └── custom-metrics-autoscaler-operator-58bd9f458-ptgwx.yaml
    │   ├── custom-metrics-autoscaler-operator-58bd9f458-thbsh
    │   │   └── custom-metrics-autoscaler-operator
    │   │       └── custom-metrics-autoscaler-operator
    │   │           └── logs
    │   ├── keda-metrics-apiserver-65c7cc44fd-6wq4g
    │   │   ├── keda-metrics-apiserver
    │   │   │   └── keda-metrics-apiserver
    │   │   │       └── logs
    │   │   │           ├── current.log
    │   │   │           ├── previous.insecure.log
    │   │   │           └── previous.log
    │   │   └── keda-metrics-apiserver-65c7cc44fd-6wq4g.yaml
    │   └── keda-operator-776cbb6768-fb6m5
    │       ├── keda-operator
    │       │   └── keda-operator
    │       │       └── logs
    │       │           ├── current.log
    │       │           ├── previous.insecure.log
    │       │           └── previous.log
    │       └── keda-operator-776cbb6768-fb6m5.yaml
    ├── policy
    │   └── poddisruptionbudgets.yaml
    └── route.openshift.io
        └── routes.yaml
```

Additional resources

• About the must-gather tool

Accessing performance metrics

The Custom Metrics Autoscaler Operator exposes ready-to-use metrics that it pulls from the on-cluster monitoring component. You can query the metrics by using the Prometheus Query Language (PromQL) to analyze and diagnose issues. All metrics are reset when the controller pod restarts.

You can access the metrics and run queries by using the OKD web console.

Procedure

1. Select the Administrator perspective in the OKD web console.

2. Select Observe → Metrics.

3. To create a custom query, add your PromQL query to the Expression field.

4. To add multiple queries, select Add Query.

Provided metrics

The Custom Metrics Autoscaler Operator exposes the following metrics, which you can view by using the OKD web console.

Custom Metrics Autoscaler Admission webhook metrics

The Custom Metrics Autoscaler Admission webhook also exposes the following Prometheus metrics.

Understanding how to add custom metrics autoscalers

To add a custom metrics autoscaler, create a ScaledObject custom resource for a deployment, stateful set, or custom resource. Create a ScaledJob custom resource for a job.

You can create only one scaled object or scaled job for each workload that you want to scale. Also, you cannot use a scaled object or scaled job and the horizontal pod autoscaler (HPA) on the same workload.

Adding a custom metrics autoscaler to a workload

You can create a custom metrics autoscaler for a workload that is created by a Deployment, StatefulSet, or custom resource object.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

• If you use a custom metrics autoscaler for scaling based on CPU or memory:

  • Your cluster administrator must have properly configured cluster metrics. You can use the oc describe PodMetrics <pod-name> command to determine if metrics are configured. If metrics are configured, the output appears similar to the following, with CPU and Memory displayed under Usage.

    $ oc describe PodMetrics openshift-kube-scheduler-ip-10-0-135-131.ec2.internal

    Example output

    Name:         openshift-kube-scheduler-ip-10-0-135-131.ec2.internal
    Namespace:    openshift-kube-scheduler
    Labels:       <none>
    Annotations:  <none>
    API Version:  metrics.k8s.io/v1beta1
    Containers:
      Name:  wait-for-host-port
      Usage:
        Memory:  0
      Name:      scheduler
      Usage:
        Cpu:     8m
        Memory:  45440Ki
    Kind:        PodMetrics
    Metadata:
      Creation Timestamp:  2019-05-23T18:47:56Z
      Self Link:           /apis/metrics.k8s.io/v1beta1/namespaces/openshift-kube-scheduler/pods/openshift-kube-scheduler-ip-10-0-135-131.ec2.internal
    Timestamp:             2019-05-23T18:47:56Z
    Window:                1m0s
    Events:                <none>

  • The pods associated with the object you want to scale must include specified memory and CPU limits. For example:

    Example pod spec

    apiVersion: v1
    kind: Pod
     ...
    spec:
      containers:
      - name: app
        image: images.my-company.example/app:v4
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Procedure

1. Create a YAML file similar to the following. Only the name <2>, object name <4>, and object kind <5> are required:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "0" (1)
     name: scaledobject (2)
     namespace: my-namespace
   spec:
     scaleTargetRef:
       apiVersion: apps/v1 (3)
       name: example-deployment (4)
       kind: Deployment (5)
       envSourceContainerName: .spec.template.spec.containers[0] (6)
     cooldownPeriod:  200 (7)
     maxReplicaCount: 100 (8)
     minReplicaCount: 0 (9)
     metricsServer: (10)
       auditConfig:
         logFormat: "json"
         logOutputVolumeClaim: "persistentVolumeClaimName"
         policy:
           rules:
           - level: Metadata
           omitStages: "RequestReceived"
           omitManagedFields: false
         lifetime:
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"
     fallback: (11)
       failureThreshold: 3
       replicas: 6
     pollingInterval: 30 (12)
     advanced:
       restoreToOriginalReplicaCount: false (13)
       horizontalPodAutoscalerConfig:
         name: keda-hpa-scale-down (14)
         behavior: (15)
           scaleDown:
             stabilizationWindowSeconds: 300
             policies:
             - type: Percent
               value: 100
               periodSeconds: 15
     triggers:
     - type: prometheus (16)
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "basic"
     - authenticationRef: (17)
         name: prom-triggerauthentication
       metadata:
         name: prom-triggerauthentication
       type: object
     - authenticationRef: (18)
         name: prom-cluster-triggerauthentication
       metadata:
         name: prom-cluster-triggerauthentication
       type: object

   1	Optional: Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling, as described in the “Pausing the custom metrics autoscaler for a workload” section.
   2	Specifies a name for this custom metrics autoscaler.
   3	Optional: Specifies the API version of the target resource. The default is apps/v1.
   4	Specifies the name of the object that you want to scale.
   5	Specifies the kind as Deployment, StatefulSet or CustomResource.
   6	Optional: Specifies the name of the container in the target resource, from which the custom metrics autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].
   7	Optional. Specifies the period in seconds to wait after the last trigger is reported before scaling the deployment back to 0 if the minReplicaCount is set to 0. The default is 300.
   8	Optional: Specifies the maximum number of replicas when scaling up. The default is 100.
   9	Optional: Specifies the minimum number of replicas when scaling down.
   10	Optional: Specifies the parameters for audit logs. as described in the “Configuring audit logging” section.
   11	Optional: Specifies the number of replicas to fall back to if a scaler fails to get metrics from the source for the number of times defined by the failureThreshold parameter. For more information on fallback behavior, see the KEDA documentation.
   12	Optional: Specifies the interval in seconds to check each trigger on. The default is 30.
   13	Optional: Specifies whether to scale back the target resource to the original replica count after the scaled object is deleted. The default is false, which keeps the replica count as it is when the scaled object is deleted.
   14	Optional: Specifies a name for the horizontal pod autoscaler. The default is keda-hpa-{scaled-object-name}.
   15	Optional: Specifies a scaling policy to use to control the rate to scale pods up or down, as described in the “Scaling policies” section.
   16	Specifies the trigger to use as the basis for scaling, as described in the “Understanding the custom metrics autoscaler triggers” section. This example uses OKD monitoring.
   17	Optional: Specifies a trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.
   18	Optional: Specifies a cluster trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.

   It is not necessary to specify both a namespace trigger authentication and a cluster trigger authentication.

2. Create the custom metrics autoscaler:

   $ oc create -f <file-name>.yaml

Verification

• View the command output to verify that the custom metrics autoscaler was created:

  $ oc get scaledobject <scaled_object_name>

  Example output

  NAME            SCALETARGETKIND      SCALETARGETNAME        MIN   MAX   TRIGGERS     AUTHENTICATION               READY   ACTIVE   FALLBACK   AGE
  scaledobject    apps/v1.Deployment   example-deployment     0     50    prometheus   prom-triggerauthentication   True    True     True       17s

  Note the following fields in the output:

• TRIGGERS: Indicates the trigger, or scaler, that is being used.

• AUTHENTICATION: Indicates the name of any trigger authentication being used.

• READY: Indicates whether the scaled object is ready to start scaling:

  • If True, the scaled object is ready.

  • If False, the scaled object is not ready because of a problem in one or more of the objects you created.

• ACTIVE: Indicates whether scaling is taking place:

  • If True, scaling is taking place.

  • If False, scaling is not taking place because there are no metrics or there is a problem in one or more of the objects you created.

• FALLBACK: Indicates whether the custom metrics autoscaler is able to get metrics from the source

  • If False, the custom metrics autoscaler is getting metrics.

  • If True, the custom metrics autoscaler is getting metrics because there are no metrics or there is a problem in one or more of the objects you created.

Additional resources

• Scaling policies

• Understanding the custom metrics autoscaler triggers

• Understanding custom metrics autoscaler trigger authentications

Adding a custom metrics autoscaler to a job

You can create a custom metrics autoscaler for any Job object.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Create a YAML file similar to the following:

   kind: ScaledJob
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: scaledjob
     namespace: my-namespace
   spec:
     failedJobsHistoryLimit: 5
     jobTargetRef:
       activeDeadlineSeconds: 600 (1)
       backoffLimit: 6 (2)
       parallelism: 1 (3)
       completions: 1 (4)
       template:  (5)
         metadata:
           name: pi
         spec:
           containers:
           - name: pi
             image: perl
             command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
     maxReplicaCount: 100 (6)
     pollingInterval: 30 (7)
     successfulJobsHistoryLimit: 5 (8)
     failedJobsHistoryLimit: 5 (9)
     envSourceContainerName: (10)
     rolloutStrategy: gradual (11)
     scalingStrategy: (12)
       strategy: "custom"
       customScalingQueueLengthDeduction: 1
       customScalingRunningJobPercentage: "0.5"
       pendingPodConditions:
         - "Ready"
         - "PodScheduled"
         - "AnyOtherCustomPodCondition"
       multipleScalersCalculation : "max"
     triggers:
     - type: prometheus (13)
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "bearer"
     - authenticationRef: (14)
         name: prom-triggerauthentication
       metadata:
         name: prom-triggerauthentication
        type: object
     - authenticationRef: (15)
         name: prom-cluster-triggerauthentication
       metadata:
         name: prom-cluster-triggerauthentication
       type: object

   1	Specifies the maximum duration the job can run.
   2	Specifies the number of retries for a job. The default is 6.
   3	Optional: Specifies how many pod replicas a job should run in parallel; defaults to 1. For non-parallel jobs, leave unset. When unset, the default is 1.
   4	Optional: Specifies how many successful pod completions are needed to mark a job completed. For non-parallel jobs, leave unset. When unset, the default is 1. For parallel jobs with a fixed completion count, specify the number of completions. For parallel jobs with a work queue, leave unset. When unset the default is the value of the parallelism parameter.
   5	Specifies the template for the pod the controller creates.
   6	Optional: Specifies the maximum number of replicas when scaling up. The default is 100.
   7	Optional: Specifies the interval in seconds to check each trigger on. The default is 30.
   8	Optional: Specifies the number of successful finished jobs should be kept. The default is 100.
   9	Optional: Specifies how many failed jobs should be kept. The default is 100.
   10	Optional: Specifies the name of the container in the target resource, from which the custom autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].
   11	Optional: Specifies whether existing jobs are terminated whenever a scaled job is being updated: default: The autoscaler terminates an existing job if its associated scaled job is updated. The autoscaler recreates the job with the latest specs. gradual: The autoscaler does not terminate an existing job if its associated scaled job is updated. The autoscaler creates new jobs with the latest specs.
   12	Optional: Specifies a scaling strategy: default, custom, or accurate. The default is default. For more information, see the link in the “Additional resources” section that follows.
   13	Specifies the trigger to use as the basis for scaling, as described in the “Understanding the custom metrics autoscaler triggers” section.
   14	Optional: Specifies a trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.
   15	Optional: Specifies a cluster trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section. It is not necessary to specify both a namespace trigger authentication and a cluster trigger authentication.

2. Create the custom metrics autoscaler:

   $ oc create -f <file-name>.yaml

Verification

• View the command output to verify that the custom metrics autoscaler was created:

  $ oc get scaledjob <scaled_job_name>

  Example output

  NAME        MAX   TRIGGERS     AUTHENTICATION              READY   ACTIVE    AGE
  scaledjob   100   prometheus   prom-triggerauthentication  True    True      8s

  Note the following fields in the output:

• TRIGGERS: Indicates the trigger, or scaler, that is being used.

• AUTHENTICATION: Indicates the name of any trigger authentication being used.

• READY: Indicates whether the scaled object is ready to start scaling:

  • If True, the scaled object is ready.

  • If False, the scaled object is not ready because of a problem in one or more of the objects you created.

• ACTIVE: Indicates whether scaling is taking place:

  • If True, scaling is taking place.

  • If False, scaling is not taking place because there are no metrics or there is a problem in one or more of the objects you created.

Additional resources

• Understanding the custom metrics autoscaler triggers

• Understanding custom metrics autoscaler trigger authentications

• Running tasks in pods using jobs

Uninstalling the Custom Metrics Autoscaler Operator

You can remove the custom metrics autoscaler from your OKD cluster. After removing the Custom Metrics Autoscaler Operator, remove other components associated with the Operator to avoid potential issues.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. In the OKD web console, click Operators → Installed Operators.

2. Switch to the openshift-keda project.

3. Remove the KedaController custom resource.

   1. Find the CustomMetricsAutoscaler Operator and click the KedaController tab.

   2. Find the custom resource, and then click Delete KedaController.

   3. Click Uninstall.

4. Remove the Custom Metrics Autoscaler Operator:

   1. Click Operators → Installed Operators.

   2. Find the CustomMetricsAutoscaler Operator and click the Options menu and select Uninstall Operator.

   3. Click Uninstall.

5. Optional: Use the OpenShift CLI to remove the custom metrics autoscaler components:

   1. Delete the custom metrics autoscaler CRDs:

      • clustertriggerauthentications.keda.sh

      • kedacontrollers.keda.sh

      • scaledjobs.keda.sh

      • scaledobjects.keda.sh

      • triggerauthentications.keda.sh

      $ oc delete crd clustertriggerauthentications.keda.sh kedacontrollers.keda.sh scaledjobs.keda.sh scaledobjects.keda.sh triggerauthentications.keda.sh

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings. However, there might be a few cluster roles that must be manually deleted.

   2. List any custom metrics autoscaler cluster roles:

      $ oc get clusterrole | grep keda.sh

   3. Delete the listed custom metrics autoscaler cluster roles. For example:

      $ oc delete clusterrole.keda.sh-v1alpha1-admin

   4. List any custom metrics autoscaler cluster role bindings:

      $ oc get clusterrolebinding | grep keda.sh

   5. Delete the listed custom metrics autoscaler cluster role bindings. For example:

      $ oc delete clusterrolebinding.keda.sh-v1alpha1-admin

6. Delete the custom metrics autoscaler project:

   $ oc delete project openshift-keda

7. Delete the Cluster Metric Autoscaler Operator:

   $ oc delete operator/openshift-custom-metrics-autoscaler-operator.openshift-keda