OpenShift Ansible Broker Configuration

Overview

When the OpenShift Ansible broker (OAB) is deployed in a cluster, its behavior is largely dictated by the broker’s configuration file loaded on startup. The broker’s configuration is stored as a ConfigMap object in the broker’s namespace (ansible-service-broker by default).

Example OpenShift Ansible Broker Configuration File

registry: (1)
  - type: dockerhub
    name: docker
    url: https://registry.hub.docker.com
    org: <dockerhub_org>
    fail_on_error: false
  - type: rhcc
    name: rhcc
    url: https://registry.redhat.io
    fail_on_error: true
    white_list:
      - "^foo.*-apb$"
      - ".*-apb$"
    black_list:
      - "bar.*-apb$"
      - "^my-apb$"
  - type: local_openshift
    name: lo
    namespaces:
      - openshift
    white_list:
      - ".*-apb$"
dao: (2)
  etcd_host: localhost
  etcd_port: 2379
log: (3)
  logfile: /var/log/ansible-service-broker/asb.log
  stdout: true
  level: debug
  color: true
openshift: (4)
  host: ""
  ca_file: ""
  bearer_token_file: ""
  image_pull_policy: IfNotPresent
  sandbox_role: "edit"
  keep_namespace: false
  keep_namespace_on_error: true
broker: (5)
  bootstrap_on_startup: true
  dev_broker: true
  launch_apb_on_bind: false
  recovery: true
  output_request: true
  ssl_cert_key: /path/to/key
  ssl_cert: /path/to/cert
  refresh_interval: "600s"
  auth:
    - type: basic
      enabled: true
secrets: (6)
  - title: Database credentials
    secret: db_creds
    apb_name: dh-rhscl-postgresql-apb

Modifying the OpenShift Ansible Broker Configuration

To modify the OAB’s default broker configuration after it has been deployed:

1. Edit the broker-config ConfigMap object in the OAB’s namespace as a user with cluster-admin privileges:

   $ oc edit configmap broker-config -n ansible-service-broker

2. After saving any updates, redeploy the OAB’s deployment configuration for the changes to take effect:

   $ oc rollout latest dc/asb -n ansible-service-broker

Registry Configuration

The registry section allows you to define the registries that the broker should look at for APBs.

Production or Development

A production broker configuration is designed to be pointed at a trusted container distribution registry, such as the Red Hat Container Catalog (RHCC):

registry:
  - name: rhcc
    type: rhcc
    url: https://registry.redhat.io
    tag: v3.11
    white_list:
      - ".*-apb$"
  - type: local_openshift
    name: localregistry
    namespaces:
      - openshift
    white_list: []

However, a development broker configuration is primarily used by developers working on the broker. To enable developer settings, set the registry name to dev and the dev_broker field in the broker section to true:

registry:
  name: dev

broker:
  dev_broker: true

Storing Registry Credentials

The broker configuration determines how the broker should read any registry credentials. They can be read from the user and pass values in the registry section, for example:

registry:
  - name: isv
    type: openshift
    url: https://registry.connect.redhat.com
    user: <user>
    pass: <password>

If you want to ensure these credentials are not publicly accessible, the auth_type field in the registry section can be set to the secret or file type. The secret type configures a registry to use a secret from the broker’s namespace, while the file type configures a registry to use a secret that has been mounted as a volume.

To use the secret or file type:

1. The associated secret should have the values username and password defined. When using a secret, you must ensure that the ansible-service-broker namespace exists, as this is where the secret will be read from.

   For example, create a reg-creds.yaml file:

   $ cat reg-creds.yaml
   ---
   username: <user_name>
   password: <password>

2. Create a secret from this file in the ansible-service-broker namespace:

   $ oc create secret generic \
       registry-credentials-secret \
       --from-file reg-creds.yaml \
       -n ansible-service-broker

3. Choose whether you want to use the secret or file type:

   • To use the secret type:

     1. In the broker configuration, set auth_type to secret and auth_name to the name of the secret:

        registry:
          - name: isv
            type: openshift
            url: https://registry.connect.redhat.com
            auth_type: secret
            auth_name: registry-credentials-secret

     2. Set the namespace where the secret is located:

        openshift:
          namespace: ansible-service-broker

-   To use the `file` type:
    1.  Edit the `asb` deployment configuration to mount your file into ***/tmp/registry-credentials/reg-creds.yaml***:
        ```
        $ oc edit dc/asb -n ansible-service-broker
        ```
        In the `containers.volumeMounts` section, add:
        ```
        volumeMounts:
          - mountPath: /tmp/registry-credentials
            name: reg-auth
        ```
        In the `volumes` section, add:
        ```
            volumes:
              - name: reg-auth
                secret:
                  defaultMode: 420
                  secretName: registry-credentials-secret
        ```
    2.  In the broker configuration, set `auth_type` to `file` and `auth_type` to the location of the file:
        ```
        registry:
          - name: isv
            type: openshift
            url: https://registry.connect.redhat.com
            auth_type: file
            auth_name: /tmp/registry-credentials/reg-creds.yaml
        ```

APB Filtering

APBs can be filtered out by their image name using a combination of the white_list or black_list parameters, set on a registry basis inside the broker’s configuration.

Both are optional lists of regular expressions that will be run over the total set of discovered APBs for a given registry to determine matches.

For example:

Whitelist Only

white_list:
  - "foo.*-apb$"
  - "^my-apb$"

Anything matching on foo.*-apb$ and only my-apb will be allowed through in this case. All other APBs will be rejected.

Blacklist Only

black_list:
  - "bar.*-apb$"
  - "^foobar-apb$"

Anything matching on bar.*-apb$ and only foobar-apb will be blocked in this case. All other APBs will be allowed through.

Whitelist and Blacklist

white_list:
  - "foo.*-apb$"
  - "^my-apb$"
black_list:
  - "^foo-rootkit-apb$"

Here, foo-rootkit-apb is specifically blocked by the blacklist despite its match in the whitelist because the whitelist match is overridden.

Otherwise, only those matching on foo.*-apb$ and my-apb will be allowed through.

Example Broker Configuration registry Section:

registry:
  - type: dockerhub
    name: dockerhub
    url: https://registry.hub.docker.com
    user: <user>
    pass: <password>
    org: <org>
    white_list:
      - "foo.*-apb$"
      - "^my-apb$"
    black_list:
      - "bar.*-apb$"
      - "^foobar-apb$"

Mock Registry

A mock registry is useful for reading local APB specs. Instead of going out to a registry to search for image specs, this uses a list of local specs. Set the name of the registry to mock to use the mock registry.

registry:
  - name: mock
    type: mock

Dockerhub Registry

The dockerhub type allows you to load APBs from a specific organization in the DockerHub. For example, the ansibleplaybookbundle organization.

registry:
  - name: dockerhub
    type: dockerhub
    org: ansibleplaybookbundle
    user: <user>
    pass: <password>
    white_list:
      - ".*-apb$"

Ansible Galaxy Registry

The galaxy type allows you to use APB roles from Ansible Galaxy. You can also optionally specify an organization.

registry:
  - name: galaxy
    type: galaxy
    # Optional:
    # org: ansibleplaybookbundle
    runner: docker.io/ansibleplaybookbundle/apb-base:latest
    white_list:
      - ".*$"

Local OpenShift Container Registry

Using the local_openshift type will allow you to load APBs from the OpenShift Container Registry that is internal to the OKD cluster. You can configure the namespaces in which you want to look for published APBs.

registry:
  - type: local_openshift
    name: lo
    namespaces:
      - openshift
    white_list:
      - ".*-apb$"

Red Hat Container Catalog Registry

Using the rhcc type will allow you to load APBs that are published to the Red Hat Container Catalog (RHCC) registry.

registry:
  - name: rhcc
    type: rhcc
    url: https://registry.redhat.io
    white_list:
      - ".*-apb$"

Red Hat Connect Partner Registry

Third-party images in the Red Hat Container Catalog are served from the Red Hat Connect Partner Registry at https://registry.connect.redhat.com. The partner_rhcc type allows the broker to be bootstrapped from the Partner Registry to retrieve a list of APBs and load their specs. The Partner Registry requires authentication for pulling images with a valid Red Hat Customer Portal user name and password.

registry:
  - name: partner_reg
    type: partner_rhcc
    url:  https://registry.connect.redhat.com
    user: <registry_user>
    pass: <registry_password>
    white_list:
      - ".*-apb$"

Because the Partner Registry requires authentication, the following manual step is also required to configure the broker to use the Partner Registry URL:

1. Run the following command on all nodes of a OKD cluster:

   # docker --config=/var/lib/origin/.docker \
       login -u <registry_user> -p <registry_password> \
       registry.connect.redhat.com

Helm Chart Registry

Using the helm type allows you to consume Helm Charts from a Helm Chart Repository.

registry:
  - name: stable
    type: helm
    url: "https://kubernetes-charts.storage.googleapis.com"
    runner: "docker.io/automationbroker/helm-runner:latest"
    white_list:
      - ".*"

API V2 Docker Registry

Using the apiv2 type allows you to consume images from docker registries that implement the Docker Registry HTTP API V2 protocol.

registry:
  - name: <registry_name>
    type: apiv2
    url:  <registry_url>
    user: <registry-user>
    pass: <registry-password>
    white_list:
      - ".*-apb$"

If the registry requires authentication for pulling images, this can be achieved by running the following command on every node in your existing cluster:

$ docker --config=/var/lib/origin/.docker login -u <registry-user> -p <registry-password> <registry_url>

Quay Docker Registry

Using the quay type allows you to load APBs that are published to the CoreOS Quay Registry. If an authentication token is provided, private repositories that the token is configured to access will load. Public repositories in the specified organization do not require a token to load.

registry:
  - name: quay_reg
    type: quay
    url:  https://quay.io
    token: <for_private_repos>
    org: <your_org>
    white_list:
      - ".*-apb$"

If the Quay registry requires authentication for pulling images, this can be achieved by running the following command on every node in your existing cluster:

$ docker --config=/var/lib/origin/.docker login -u <registry-user> -p <registry-password> quay.io

Multiple Registries

You can use more than one registry to separate APBs into logical organizations and be able to manage them from the same broker. The registries must have a unique, non-empty name. If there is no unique name, the service broker will fail to start with an error message alerting you to the problem.

registry:
  - name: dockerhub
    type: dockerhub
    org: ansibleplaybookbundle
    user: <user>
    pass: <password>
    white_list:
      - ".*-apb$"
  - name: rhcc
    type: rhcc
    url: <rhcc_url>
    white_list:
      - ".*-apb$"

Broker Authentication

The broker supports authentication, meaning when connecting to the broker, the caller must supply the Basic Auth or Bearer Auth credentials for each request. Using curl, it is as simple as supplying:

-u <user_name>:<password>

or

-h "Authorization: bearer <token>

to the command. The service catalog must be configured with a secret containing the user name and password combinations or the bearer token.

Basic Auth

To enable Basic Auth usage, set the following in the broker configuration:

broker:
   ...
   auth:
     - type: basic (1)
       enabled: true (2)

Deployment Template and Secrets

Typically the broker is configured using a ConfigMap in a deployment template. You supply the authentication configuration the same way as in the file configuration.

The following is an example of the deployment template:

auth:
  - type: basic
    enabled: ${ENABLE_BASIC_AUTH}

Another part to Basic Auth is the user name and password used to authenticate against the broker. While the Basic Auth implementation can be backed by different back-end services, the currently supported one is backed by a secret. The secret must be injected into the pod via a volume mount at the /var/run/asb_auth location. This is from where the broker will read the user name and password.

In the deployment template, a secret must be specified. For example:

- apiVersion: v1
  kind: Secret
  metadata:
    name: asb-auth-secret
    namespace: ansible-service-broker
  data:
    username: ${BROKER_USER}
    password: ${BROKER_PASS}

The secret must contain a user name and password. The values must be base64 encoded. The easiest way to generate the values for those entries is to use the echo and base64 commands:

$ echo -n admin | base64 (1)
YWRtaW4=

This secret must now be injected to the pod via a volume mount. This is configured in the deployment template as well:

spec:
  serviceAccount: asb
  containers:
  - image: ${BROKER_IMAGE}
    name: asb
    imagePullPolicy: IfNotPresent
    volumeMounts:
      ...
      - name: asb-auth-volume
        mountPath: /var/run/asb-auth

Then, in the volumes section, mount the secret:

volumes:
  ...
  - name: asb-auth-volume
    secret:
      secretName: asb-auth-secret

The above will have created a volume mount located at /var/run/asb-auth. This volume will have two files: a user name and password written by the asb-auth-secret secret.

Configuring Service Catalog and Broker Communication

Now that the broker is configured to use Basic Auth, you must tell the service catalog how to communicate with the broker. This is accomplished by the authInfo section of the broker resource.

The following is an example of creating a broker resource in the service catalog. The spec tells the service catalog what URL the broker is listening at. The authInfo tells it what secret to read to get the authentication information.

apiVersion: servicecatalog.k8s.io/v1alpha1
kind: Broker
metadata:
  name: ansible-service-broker
spec:
  url: https://asb-1338-ansible-service-broker.172.17.0.1.nip.io
  authInfo:
    basicAuthSecret:
      namespace: ansible-service-broker
      name: asb-auth-secret

Starting with v0.0.17 of the service catalog, the broker resource configuration changes:

apiVersion: servicecatalog.k8s.io/v1alpha1
kind: ServiceBroker
metadata:
  name: ansible-service-broker
spec:
  url: https://asb-1338-ansible-service-broker.172.17.0.1.nip.io
  authInfo:
    basic:
      secretRef:
        namespace: ansible-service-broker
        name: asb-auth-secret

Bearer Auth

By default, if no authentication is specified the broker will use bearer token authentication (Bearer Auth). Bearer Auth uses delegated authentication from the Kubernetes apiserver library.

The configuration grants access, through Kubernetes RBAC roles and role bindings, to the URL prefix. The broker has added a configuration option cluster_url to specify the url_prefix. This value defaults to ansible-service-broker.

Example Cluster Role

- apiVersion: authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: access-asb-role
  rules:
  - nonResourceURLs: ["/ansible-service-broker", "/ansible-service-broker/*"]
    verbs: ["get", "post", "put", "patch", "delete"]

Deployment Template and Secrets

The following is an example of creating a secret that the service catalog can use. This example assumes that the role, access-asb-role, has been created already. From the deployment template:

- apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: ansibleservicebroker-client
    namespace: ansible-service-broker
- apiVersion: authorization.openshift.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: ansibleservicebroker-client
  subjects:
  - kind: ServiceAccount
    name: ansibleservicebroker-client
    namespace: ansible-service-broker
  roleRef:
    kind: ClusterRole
    name: access-asb-role
- apiVersion: v1
  kind: Secret
  metadata:
    name: ansibleservicebroker-client
    annotations:
      kubernetes.io/service-account.name: ansibleservicebroker-client
  type: kubernetes.io/service-account-token

The above example creates a service account, granting access to access-asb-role and creating a secret for that service accounts token.

Configuring Service Catalog and Broker Communication

Now that the broker is configured to use Bearer Auth tokens, you must tell the service catalog how to communicate with the broker. This is accomplished by the authInfo section of the broker resource.

The following is an example of creating a broker resource in the service catalog. The spec tells the service catalog what URL the broker is listening at. The authInfo tells it what secret to read to get the authentication information.

apiVersion: servicecatalog.k8s.io/v1alpha1
kind: ServiceBroker
metadata:
  name: ansible-service-broker
spec:
  url: https://asb.ansible-service-broker.svc:1338${BROKER_URL_PREFIX}/
  authInfo:
    bearer:
      secretRef:
        kind: Secret
        namespace: ansible-service-broker
        name: ansibleservicebroker-client

DAO Configuration

Log Configuration

OpenShift Configuration

Broker Configuration

The broker section tells the broker what functionality should be enabled and disabled. It will also tell the broker where to find files on disk that will enable the full functionality.

Secrets Configuration

The secrets section creates associations between secrets in the broker’s namespace and APBs the broker runs. The broker uses these rules to mount secrets into running APBs, allowing the user to use secrets to pass parameters without exposing them to the catalog or users.

The section is a list where each entry has the following structure:

You can download and use the create_broker_secret.py file to create and format this configuration section.

secrets:
- title: Database credentials
  secret: db_creds
  apb_name: dh-rhscl-postgresql-apb

Running Behind a Proxy

When running the OAB inside of a proxied OKD cluster, it is important to understand its core concepts and consider them within the context of a proxy used for external network access.

As an overview, the broker itself runs as a pod within the cluster. It has a requirement for external network access depending on how its registries have been configured.

Registry Adapter Whitelists

The broker’s configured registry adapters must be able to communicate with their external registries in order to bootstrap successfully and load remote APB manifests. These requests can be made via the proxy, however, the proxy must ensure that the required remote hosts are accessible.

Example required whitelisted hosts:

Configuring the Broker Behind a Proxy Using Ansible

If during initial installation you configure your OKD cluster to run behind a proxy (see Configuring Global Proxy Options), when the OAB is deployed it will:

• inherit those cluster-wide proxy settings automatically and

• generate the required NO_PROXY list, including the cidr fields and serviceNetworkCIDR,

and no further configuration is needed.

Configuring the Broker Behind a Proxy Manually

If your cluster’s global proxy options were not configured during initial installation or prior to the broker being deployed, or if you have modified the global proxy settings, you must manually configure the broker for external access via proxy:

1. Before attempting to run the OAB behind a proxy, review Working with HTTP Proxies and ensure your cluster is configured accordingly to run behind a proxy.

   In particular, the cluster must be configured to not proxy internal cluster requests. This is typically configured with a NO_PROXY setting of:

   .cluster.local,.svc,<serviceNetworkCIDR_value>,<master_IP>,<master_domain>,.default

   in addition to any other desired NO_PROXY settings. See Configuring NO_PROXY for more details.

   Brokers deploying unversioned, or v1 APBs must also add 172.30.0.1 to their NO_PROXY list. APBs prior to v2 extracted their credentials from running APB pods via an exec HTTP request, rather than a secret exchange. Unless you are running a broker with experimental proxy support in a cluster prior to OKD 3.9, you probably do not have to worry about this.

2. Edit the broker’s DeploymentConfig as a user with cluster-admin privileges:

   $ oc edit dc/asb -n ansible-service-broker

3. Set the following environment variables:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

<table><tbody><tr><td><i title="Note"></i></td><td><div><p>See <a href="$d137c428e6779a11.md#setting-environment-variables-in-pods">Setting Proxy Environment Variables in Pods</a> for more information.</p></div></td></tr></tbody></table>

1. After saving any updates, redeploy the OAB’s deployment configuration for the changes to take effect:

   $ oc rollout latest dc/asb -n ansible-service-broker

Setting Proxy Environment Variables in Pods

It is common that APB pods themselves may require external access via proxy as well. If the broker recognizes it has a proxy configuration, it will transparently apply these environment variables to the APB pods that it spawns. As long as the modules used within the APB respect proxy configuration via environment variable, the APB will also use these settings to perform its work.

Finally, it is possible the services spawned by the APB may also require external network access via proxy. The APB must be authored to set these environment variables explicitly if recognizes them in its own execution environment, or the cluster operator must manually modify the required services to inject them into their environments.