Using remote health reporting in a restricted network

You can manually gather and upload Insights Operator archives to diagnose issues from a restricted network.

To use the Insights Operator in a restricted network, you must:

  • Create a copy of your Insights Operator archive.

  • Upload the Insights Operator archive to console.redhat.com.

Additionally, you can choose to obfuscate the Insights Operator data before upload.

Running an Insights Operator gather operation

You must run a gather operation to create an Insights Operator archive.

Prerequisites

  • You are logged in to OKD as cluster-admin.

Procedure

  1. Create a file named gather-job.yaml using this template:

    1. apiVersion: batch/v1
    2. kind: Job
    3. metadata:
    4.   name: insights-operator-job
    5.   annotations:
    6.     config.openshift.io/inject-proxy: insights-operator
    7. spec:
    8.   backoffLimit: 6
    9.   ttlSecondsAfterFinished: 600
    10.   template:
    11.     spec:
    12.       restartPolicy: OnFailure
    13.       serviceAccountName: operator
    14.       nodeSelector:
    15.         beta.kubernetes.io/os: linux
    16.         node-role.kubernetes.io/master: ""
    17.       tolerations:
    18.       - effect: NoSchedule
    19.         key: node-role.kubernetes.io/master
    20.         operator: Exists
    21.       - effect: NoExecute
    22.         key: node.kubernetes.io/unreachable
    23.         operator: Exists
    24.         tolerationSeconds: 900
    25.       - effect: NoExecute
    26.         key: node.kubernetes.io/not-ready
    27.         operator: Exists
    28.         tolerationSeconds: 900
    29.       volumes:
    30.       - name: snapshots
    31.         emptyDir: {}
    32.       - name: service-ca-bundle
    33.         configMap:
    34.           name: service-ca-bundle
    35.           optional: true
    36.       initContainers:
    37.       - name: insights-operator
    38.         image: quay.io/openshift/origin-insights-operator:latest
    39.         terminationMessagePolicy: FallbackToLogsOnError
    40.         volumeMounts:
    41.         - name: snapshots
    42.           mountPath: /var/lib/insights-operator
    43.         - name: service-ca-bundle
    44.           mountPath: /var/run/configmaps/service-ca-bundle
    45.           readOnly: true
    46.         ports:
    47.         - containerPort: 8443
    48.           name: https
    49.         resources:
    50.           requests:
    51.             cpu: 10m
    52.             memory: 70Mi
    53.         args:
    54.         - gather
    55.         - -v=4
    56.         - --config=/etc/insights-operator/server.yaml
    57.       containers:
    58.         - name: sleepy
    59.           image: quay.io/openshift/origin-base:latest
    60.           args:
    61.             - /bin/sh
    62.             - -c
    63.             - sleep 10m
    64.           volumeMounts: [{name: snapshots, mountPath: /var/lib/insights-operator}]

  2. Copy your insights-operator image version:

    1. $ oc get -n openshift-insights deployment insights-operator -o yaml

  3. Paste your image version in gather-job.yaml:

    1. initContainers:
    2.       - name: insights-operator
    3.         image: <your_insights_operator_image_version>
    4.         terminationMessagePolicy: FallbackToLogsOnError
    5.         volumeMounts:

  4. Create the gather job:

    1. $ oc apply -n openshift-insights -f gather-job.yaml

  5. Find the name of the job pod:

    1. $ oc describe -n openshift-insights job/insights-operator-job

    Example output

    1. Events:
    2.   Type    Reason            Age    From            Message
    3.   ----    ------            ----   ----            -------
    4.   Normal  SuccessfulCreate  7m18s  job-controller  Created pod: insights-operator-job-<your_job>

    where insights-operator-job-*<your_job>* is the name of the pod.

  6. Verify that the operation has finished:

    1. $ oc logs -n openshift-insights insights-operator-job-<your_job> insights-operator

    Example output

    1. I0407 11:55:38.192084       1 diskrecorder.go:34] Wrote 108 records to disk in 33ms

  7. Save the created archive:

    1. $ oc cp openshift-insights/insights-operator-job-<your_job>:/var/lib/insights-operator ./insights-data

  8. Clean up the job:

    1. $ oc delete -n openshift-insights job insights-operator-job

Uploading an Insights Operator archive

You can manually upload an Insights Operator archive to console.redhat.com to diagnose potential issues.

Prerequisites

  • You are logged in to OKD as cluster-admin.

  • You have a workstation with unrestricted internet access.

  • You have created a copy of the Insights Operator archive.

Procedure

  1. Download the dockerconfig.json file:

    1. $ oc extract secret/pull-secret -n openshift-config --to=.

  2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

    1. {
    2.   "auths": {
    3.     "cloud.openshift.com": {
    4.       "auth": "<your_token>",
    5.       "email": "asd@redhat.com"
    6.     }
    7. }

  3. Upload the archive to console.redhat.com:

    1. $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

    where *<cluster_id>* is your cluster ID, *<your_token>* is the token from your pull secret, and *<path_to_archive>* is the path to the Insights Operator archive.

    If the operation is successful, the command returns a "request_id" and "account_number":

    Example output

    1. * Connection #0 to host console.redhat.com left intact
    2. {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

Verification steps

  1. Log in to https://console.redhat.com/openshift.

  2. Click the Clusters menu in the left pane.

  3. To display the details of the cluster, click the cluster name.

  4. Open the Insights Advisor tab of the cluster.

    If the upload was successful, the tab displays one of the following:

    • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.

    • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).

Enabling Insights Operator data obfuscation

You can enable obfuscation to mask sensitive and identifiable IPv4 addresses and cluster base domains that the Insights Operator sends to console.redhat.com.

Although this feature is available, Red Hat recommends keeping obfuscation disabled for a more effective support experience.

Obfuscation assigns non-identifying values to cluster IPv4 addresses, and uses a translation table that is retained in memory to change IP addresses to their obfuscated versions throughout the Insights Operator archive before uploading the data to console.redhat.com.

For cluster base domains, obfuscation changes the base domain to a hardcoded substring. For example, cluster-api.openshift.example.com becomes cluster-api.<CLUSTER_BASE_DOMAIN>.

The following procedure enables obfuscation using the support secret in the openshift-config namespace.

Prerequisites

  • You are logged in to the OKD web console as cluster-admin.

Procedure

  1. Navigate to Workloads → Secrets.

  2. Select the openshift-config project.

  3. Search for the support secret using the Search by name field. If it does not exist, click Create → Key/value secret to create it.

  4. Click the Options menu kebab, and then click Edit Secret.

  5. Click Add Key/Value.

  6. Create a key named enableGlobalObfuscation with a value of true, and click Save.

  7. Navigate to Workloads → Pods

  8. Select the openshift-insights project.

  9. Find the insights-operator pod.

  10. To restart the insights-operator pod, click the Options menu kebab, and then click Delete Pod.

Verification

  1. Navigate to Workloads → Secrets.

  2. Select the openshift-insights project.

  3. Search for the obfuscation-translation-table secret using the Search by name field.

If the obfuscation-translation-table secret exists, then obfuscation is enabled and working.

Alternatively, you can inspect /insights-operator/gathers.json in your Insights Operator archive for the value "is_global_obfuscation_enabled": true.

Additional resources