Generating Diffs to Ignore Modified GitRepos

Continuous Delivery in Rancher is powered by fleet. When a user adds a GitRepo CR, then Continuous Delivery creates the associated fleet bundles.

You can access these bundles by navigating to the Cluster Explorer (Dashboard UI), and selecting the Bundles section.

The bundled charts may have some objects that are amended at runtime, for example in ValidatingWebhookConfiguration the caBundle is empty and the CA cert is injected by the cluster.

This leads the status of the bundle and associated GitRepo to be reported as “Modified”

Generating Diffs to Ignore Modified GitRepos - 图1

Associated Bundle Generating Diffs to Ignore Modified GitRepos - 图2

Fleet bundles support the ability to specify a custom jsonPointer patch.

With the patch, users can instruct fleet to ignore object modifications.

Simple Example

https://github.com/rancher/fleet-examples/tree/master/bundle-diffs

Gatekeeper Example

In this example, we are trying to deploy opa-gatekeeper using Continuous Delivery to our clusters.

The opa-gatekeeper bundle associated with the opa GitRepo is in modified state.

Each path in the GitRepo CR, has an associated Bundle CR. The user can view the Bundles, and the associated diff needed in the Bundle status.

In our case the differences detected are as follows:

  1. summary:
  2. desiredReady: 1
  3. modified: 1
  4. nonReadyResources:
  5. - bundleState: Modified
  6. modifiedStatus:
  7. - apiVersion: admissionregistration.k8s.io/v1
  8. kind: ValidatingWebhookConfiguration
  9. name: gatekeeper-validating-webhook-configuration
  10. patch: '{"$setElementOrder/webhooks":[{"name":"validation.gatekeeper.sh"},{"name":"check-ignore-label.gatekeeper.sh"}],"webhooks":[{"clientConfig":{"caBundle":"Cg=="},"name":"validation.gatekeeper.sh","rules":[{"apiGroups":["*"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["*"]}]},{"clientConfig":{"caBundle":"Cg=="},"name":"check-ignore-label.gatekeeper.sh","rules":[{"apiGroups":[""],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["namespaces"]}]}]}'
  11. - apiVersion: apps/v1
  12. kind: Deployment
  13. name: gatekeeper-audit
  14. namespace: cattle-gatekeeper-system
  15. patch: '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"manager"}],"containers":[{"name":"manager","resources":{"limits":{"cpu":"1000m"}}}],"tolerations":[]}}}}'
  16. - apiVersion: apps/v1
  17. kind: Deployment
  18. name: gatekeeper-controller-manager
  19. namespace: cattle-gatekeeper-system
  20. patch: '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"manager"}],"containers":[{"name":"manager","resources":{"limits":{"cpu":"1000m"}}}],"tolerations":[]}}}}'

Based on this summary, there are three objects which need to be patched.

We will look at these one at a time.

1. ValidatingWebhookConfiguration:

The gatekeeper-validating-webhook-configuration validating webhook has two ValidatingWebhooks in its spec.

In cases where more than one element in the field requires a patch, that patch will refer these to as $setElementOrder/ELEMENTNAME

From this information, we can see the two ValidatingWebhooks in question are:

  1. "$setElementOrder/webhooks": [
  2. {
  3. "name": "validation.gatekeeper.sh"
  4. },
  5. {
  6. "name": "check-ignore-label.gatekeeper.sh"
  7. }
  8. ],

Within each ValidatingWebhook, the fields that need to be ignore are as follows:

  1. {
  2. "clientConfig": {
  3. "caBundle": "Cg=="
  4. },
  5. "name": "validation.gatekeeper.sh",
  6. "rules": [
  7. {
  8. "apiGroups": [
  9. "*"
  10. ],
  11. "apiVersions": [
  12. "*"
  13. ],
  14. "operations": [
  15. "CREATE",
  16. "UPDATE"
  17. ],
  18. "resources": [
  19. "*"
  20. ]
  21. }
  22. ]
  23. },

and

  1. {
  2. "clientConfig": {
  3. "caBundle": "Cg=="
  4. },
  5. "name": "check-ignore-label.gatekeeper.sh",
  6. "rules": [
  7. {
  8. "apiGroups": [
  9. ""
  10. ],
  11. "apiVersions": [
  12. "*"
  13. ],
  14. "operations": [
  15. "CREATE",
  16. "UPDATE"
  17. ],
  18. "resources": [
  19. "namespaces"
  20. ]
  21. }
  22. ]
  23. }

In summary, we need to ignore the fields rules and clientConfig.caBundle in our patch specification.

The field webhook in the ValidatingWebhookConfiguration spec is an array, so we need to address the elements by their index values.

Generating Diffs to Ignore Modified GitRepos - 图3

Based on this information, our diff patch would look as follows:

  1. - apiVersion: admissionregistration.k8s.io/v1
  2. kind: ValidatingWebhookConfiguration
  3. name: gatekeeper-validating-webhook-configuration
  4. operations:
  5. - {"op": "remove", "path":"/webhooks/0/clientConfig/caBundle"}
  6. - {"op": "remove", "path":"/webhooks/0/rules"}
  7. - {"op": "remove", "path":"/webhooks/1/clientConfig/caBundle"}
  8. - {"op": "remove", "path":"/webhooks/1/rules"}

2. Deployment gatekeeper-controller-manager:

The gatekeeper-controller-manager deployment is modified since there are cpu limits and tolerations applied (which are not in the actual bundle).

  1. {
  2. "spec": {
  3. "template": {
  4. "spec": {
  5. "$setElementOrder/containers": [
  6. {
  7. "name": "manager"
  8. }
  9. ],
  10. "containers": [
  11. {
  12. "name": "manager",
  13. "resources": {
  14. "limits": {
  15. "cpu": "1000m"
  16. }
  17. }
  18. }
  19. ],
  20. "tolerations": []
  21. }
  22. }
  23. }
  24. }

In this case, there is only 1 container in the deployment container spec, and that container has cpu limits and tolerations added.

Based on this information, our diff patch would look as follows:

  1. - apiVersion: apps/v1
  2. kind: Deployment
  3. name: gatekeeper-controller-manager
  4. namespace: cattle-gatekeeper-system
  5. operations:
  6. - {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
  7. - {"op": "remove", "path": "/spec/template/spec/tolerations"}

3. Deployment gatekeeper-audit:

The gatekeeper-audit deployment is modified in a similarly, to the gatekeeper-controller-manager, with additional cpu limits and tolerations applied.

  1. {
  2. "spec": {
  3. "template": {
  4. "spec": {
  5. "$setElementOrder/containers": [
  6. {
  7. "name": "manager"
  8. }
  9. ],
  10. "containers": [
  11. {
  12. "name": "manager",
  13. "resources": {
  14. "limits": {
  15. "cpu": "1000m"
  16. }
  17. }
  18. }
  19. ],
  20. "tolerations": []
  21. }
  22. }
  23. }
  24. }

Similar to gatekeeper-controller-manager, there is only 1 container in the deployments container spec, and that has cpu limits and tolerations added.

Based on this information, our diff patch would look as follows:

  1. - apiVersion: apps/v1
  2. kind: Deployment
  3. name: gatekeeper-audit
  4. namespace: cattle-gatekeeper-system
  5. operations:
  6. - {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
  7. - {"op": "remove", "path": "/spec/template/spec/tolerations"}

Combining It All Together

We can now combine all these patches as follows:

  1. diff:
  2. comparePatches:
  3. - apiVersion: apps/v1
  4. kind: Deployment
  5. name: gatekeeper-audit
  6. namespace: cattle-gatekeeper-system
  7. operations:
  8. - {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
  9. - {"op": "remove", "path": "/spec/template/spec/tolerations"}
  10. - apiVersion: apps/v1
  11. kind: Deployment
  12. name: gatekeeper-controller-manager
  13. namespace: cattle-gatekeeper-system
  14. operations:
  15. - {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
  16. - {"op": "remove", "path": "/spec/template/spec/tolerations"}
  17. - apiVersion: admissionregistration.k8s.io/v1
  18. kind: ValidatingWebhookConfiguration
  19. name: gatekeeper-validating-webhook-configuration
  20. operations:
  21. - {"op": "remove", "path":"/webhooks/0/clientConfig/caBundle"}
  22. - {"op": "remove", "path":"/webhooks/0/rules"}
  23. - {"op": "remove", "path":"/webhooks/1/clientConfig/caBundle"}
  24. - {"op": "remove", "path":"/webhooks/1/rules"}

We can add these now to the bundle directly to test and also commit the same to the fleet.yaml in your GitRepo.

Once these are added, the GitRepo should deploy and be in “Active” status.