Validating Operators using the scorecard

Operator authors should validate that their Operator is packaged correctly and free of syntax errors. As an Operator author, you can use the Operator SDK scorecard tool to validate your Operator packaging and run tests.

About the scorecard tool

To validate an Operator, the scorecard tool provided by the Operator SDK begins by creating all resources required by any related custom resources (CRs) and the Operator. The scorecard then creates a proxy container in the deployment of the Operator which is used to record calls to the API server and run some of the tests. The tests performed also examine some of the parameters in the CRs.

Scorecard configuration

The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options.

Configuration file

The default location for the scorecard tool configuration is the <project_dir>/.osdk-scorecard.*. The following is an example of a YAML-formatted configuration file:

Scorecard configuration file

scorecard:
  output: json
  plugins:
    - basic: (1)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
    - olm: (2)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
        csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml"

Configuration methods for global options take the following priority, highest to lowest:

Command arguments (if available) → configuration file → default

The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all operator-sdk subcommands in the future, the scorecard configuration must be under a scorecard subsection.

Command arguments

While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments:

Configuration file options

The scorecard configuration file provides the following options:

Basic and OLM plug-ins

The scorecard supports the internal basic and olm plug-ins, which are configured by a plugins section in the configuration file.

Additional resources

• You can either set cr-manifest or your CSV metadata.annotations['alm-examples'] to provide CRs to the scorecard, but not both. See CRD templates for details.

Tests performed

By default, the scorecard tool has a set of internal tests it can run available across two internal plug-ins. If multiple CRs are specified for a plug-in, the test environment is fully cleaned up after each CR so that each CR gets a clean testing environment.

Each test has a short name that uniquely identifies the test. This is useful when selecting a specific test or tests to run. For example:

$ operator-sdk scorecard -o text --selector=test=checkspectest

$ operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest)'

Basic plug-in

The following basic Operator tests are available from the basic plug-in:

OLM plug-in

The following Operator Lifecycle Manager (OLM) integration tests are available from the olm plug-in:

Additional resources

• Owned CRDs

Running the scorecard

Prerequisites

The following prerequisites for the Operator project are checked by the scorecard tool:

• Access to a cluster running Kubernetes 1.11.3 or later.

• If you want to use the scorecard to check the integration of your Operator project with Operator Lifecycle Manager (OLM), then a cluster service version (CSV) file is also required. This is a requirement when the olm-deployed option is used.

• For Operators that were not generated using the Operator SDK (non-SDK Operators):

  • Resource manifests for installing and configuring the Operator and custom resources (CRs). See the Writing E2E Tests guide for more information on the global and namespaced manifests.

  • Configuration getter that supports reading from the KUBECONFIG environment variable, such as the clientcmd or controller-runtime configuration getters. This is required for the scorecard proxy to work correctly.

Procedure

1. Define a .osdk-scorecard.yaml configuration file in your Operator project.

2. Create the namespace defined in the RBAC files (role_binding).

3. Run the scorecard from the root directory of your Operator project:

   $ operator-sdk scorecard

   The scorecard return code is 1 if any of the executed texts did not pass and 0 if all selected tests passed.

Running the scorecard with an OLM-managed Operator

The scorecard can be run using a cluster service version (CSV), providing a way to test cluster-ready and non-Operator SDK Operators.

Procedure

1. The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy before deploying your Operator with Operator Lifecycle Manager (OLM).

   This step can be performed manually or automated using bash functions. Choose one of the following methods.

   • Manual method:

     1. Create a proxy server secret containing a local kubeconfig file`.

        1. Generate a user name using the namespaced owner reference of the scorecard proxy.

           $ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"'<namespace>'"}' | base64 -w 0 (1)

           1	Replace <namespace> with the namespace your Operator will deploy in.

        2. Write a Config manifest scorecard-config.yaml using the following template, replacing <username> with the base64 user name generated in the previous step:

           apiVersion: v1
           kind: Config
           clusters:
           - cluster:
               insecure-skip-tls-verify: true
               server: http://<username>@localhost:8889
             name: proxy-server
           contexts:
           - context:
               cluster: proxy-server
               user: admin/proxy-server
             name: <namespace>/proxy-server
           current-context: <namespace>/proxy-server
           preferences: {}
           users:
           - name: admin/proxy-server
             user:
               username: <username>
               password: unused

        3. Encode the Config as base64:

           $ cat scorecard-config.yaml | base64 -w 0

        4. Create a Secret manifest scorecard-secret.yaml:

           apiVersion: v1
           kind: Secret
           metadata:
             name: scorecard-kubeconfig
             namespace: <namespace> (1)
           data:
             kubeconfig: <kubeconfig_base64> (2)

           1	Replace <namespace> with the namespace your Operator will deploy in.
           2	Replace <kubeconfig_base64> with the Config encoded as base64.

        5. Apply the secret:

           $ oc apply -f scorecard-secret.yaml

        6. Insert a volume referring to the secret into the deployment for the Operator:

           spec:
             install:
               spec:
                 deployments:
                 - name: memcached-operator
                   spec:
                     ...
                     template:
                       ...
                       spec:
                         containers:
                         ...
                         volumes:
                         - name: scorecard-kubeconfig (1)
                           secret:
                             secretName: scorecard-kubeconfig
                             items:
                             - key: kubeconfig
                               path: config

           1	Scorecard kubeconfig volume.

     2. Insert a volume mount and KUBECONFIG environment variable into each container in the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      - name: container1
                        ...
                        volumeMounts:
                        - name: scorecard-kubeconfig (1)
                          mountPath: /scorecard-secret
                        env:
                        - name: KUBECONFIG (2)
                          value: /scorecard-secret/config
                      - name: container2 (3)
                        ...

        1	Scorecard kubeconfig volume mount.
        2	Scorecard kubeconfig environment variable.
        3	Repeat the same for this and all other containers.

     3. Insert the scorecard proxy container into the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      ...
                      - name: scorecard-proxy (1)
                        command:
                        - scorecard-proxy
                        env:
                        - name: WATCH_NAMESPACE
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.namespace
                        image: quay.io/operator-framework/scorecard-proxy:master
                        imagePullPolicy: Always
                        ports:
                        - name: proxy
                          containerPort: 8889

        1	Scorecard proxy container.

   • Automated method:

     The community-operators repository has several bash functions that can perform the previous steps in the procedure for you.

     1. Run the following curl command:

        $ curl -Lo csv-manifest-modifiers.sh \
            https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file

     2. Source the csv-manifest-modifiers.sh file:

        $ . ./csv-manifest-modifiers.sh

     3. Create the kubeconfig secret file:

        $ create_kubeconfig_secret_file scorecard-secret.yaml "<namespace>" (1)

        1	Replace <namespace> with the namespace your Operator will deploy in.

     4. Apply the secret:

        $ oc apply -f scorecard-secret.yaml

     5. Insert the kubeconfig volume:

        $ insert_kubeconfig_volume "<csv_file>" (1)

        1	Replace <csv_file> with the path to your CSV manifest.

     6. Insert the kubeconfig secret mount:

        $ insert_kubeconfig_secret_mount "<csv_file>"

     7. Insert the proxy container:

        $ insert_proxy_container "<csv_file>" "quay.io/operator-framework/scorecard-proxy:master"

1. After inserting the proxy container, follow the steps in the Getting started with the Operator SDK guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM.

2. After your Operator has been deployed on OLM, define a .osdk-scorecard.yaml configuration file in your Operator project and ensure both the csv-path: <csv_manifest_path> and olm-deployed options are set.

3. Run the scorecard with both the csv-path: <csv_manifest_path> and olm-deployed options set in your scorecard configuration file:

   $ operator-sdk scorecard