GnuPG signature verification

Overview

As of v1.7 it is possible to configure ArgoCD to only sync against commits that are signed in Git using GnuPG. Signature verification is configured on project level.

If a project is configured to enforce signature verification, all applications associated with this project must have the commits in the source repositories signed with a GnuPG public key known to ArgoCD. ArgoCD will refuse to sync to any revision that does not have a valid signature made by one of the configured keys. The controller will emit a ResourceComparison error if it tries to sync to a revision that is either not signed, or is signed by an unknown or not allowed public key.

By default, signature verification is enabled but not enforced. If you wish to completely disable the GnuPG functionality in ArgoCD, you have to set the environment variable ARGOCD_GPG_ENABLED to "false" in the pod templates of the argocd-server, argocd-repo-server and argocd-application-controller deployment manifests.

Verification of GnuPG signatures is only supported with Git repositories. It is not possible using Helm repositories.

A few words about trust

ArgoCD uses a very simple trust model for the keys you import: Once the key is imported, ArgoCD will trust it. ArgoCD does not support more complex trust models, and it is not necessary (nor possible) to sign the public keys you are going to import into ArgoCD.

Signature verification targets

If signature verification is enforced, ArgoCD will verify the signature using following strategy:

  • If target revision is a pointer to a commit object (i.e. a branch name, the name of a reference such as HEAD or a commit SHA), ArgoCD will perform the signature verification on the commit object the name points to, i.e. a commit.

  • If target revision resolves to a tag and the tag is a lightweight tag, the behaviour is same as if target revision would be a pointer to a commit object. However, if the tag is annotated, the target revision will point to a tag object and thus, the signature verification is performed on the tag object, i.e. the tag itself must be signed (using git tag -s).

Enforcing signature verification

To configure enforcing of signature verification, the following steps must be performed:

  • Import the GnuPG public key(s) used for signing commits in ArgoCD
  • Configure a project to enforce signature verification for given keys

Once you have configured one or more keys to be required for verification for a given project, enforcement is active for all applications associated with this project.

Warning

If signature verification is enforced, you will not be able to sync from local sources (i.e. argocd app sync --local) anymore.

RBAC rules for managing GnuPG keys

The appropriate resource notation for Argo CD’s RBAC implementation to allow the managing of GnuPG keys is gpgkeys.

To allow listing of keys for a role named role:myrole, use:

  1. p, role:myrole, gpgkeys, get, *, allow

To allow adding keys for a role named role:myrole, use:

  1. p, role:myrole, gpgkeys, create, *, allow

And finally, to allow deletion of keys for a role named role:myrole, use:

  1. p, role:myrole, gpgkeys, delete, *, allow

Importing GnuPG public keys

You can configure the GnuPG public keys that ArgoCD will use for verification of commit signatures using either the CLI, the web UI or configuring it using declarative setup.

Note

After you have imported a GnuPG key, it may take a while until the key is propagated within the cluster, even if listed as configured. If you still cannot sync to commits signed by the already imported key, please see the troubleshooting section below.

Users wanting to manage the GnuPG public key configuration require the RBAC permissions for gpgkeys resources.

Manage public keys using the CLI

To configure GnuPG public keys using the CLI, use the argocd gpg command.

Listing all configured keys

To list all configured keys known to ArgoCD, use the argocd gpg list sub-command:

  1. argocd gpg list

Show information about a certain key

To get information about a specific key, use the argocd gpg get sub-command:

  1. argocd gpg get <key-id>

Importing a key

To import a new key to ArgoCD, use the argocd gpg add sub-command:

  1. argocd gpg add --from <path-to-key>

The key to be imported can be either in binary or ASCII-armored format.

Removing a key from configuration

To remove a previously configured key from the configuration, use the argocd gpg rm sub-command:

  1. argocd gpg rm <key-id>

Manage public keys using the Web UI

Basic key management functionality for listing, importing and removing GnuPG public keys is implemented in the Web UI. You can find the configuration module from the Settings page in the GnuPG keys module.

Please note that when you configure keys using the Web UI, the key must be imported in ASCII armored format for now.

Manage public keys in declarative setup

ArgoCD stores public keys internally in the argocd-gpg-keys-cm ConfigMap resource, with the public GnuPG key’s ID as its name and the ASCII armored key data as string value, i.e. the entry for the GitHub’s web-flow signing key would look like follows:

  1. 4AEE18F83AFDEB23: |
  2.     -----BEGIN PGP PUBLIC KEY BLOCK-----
  4.     mQENBFmUaEEBCACzXTDt6ZnyaVtueZASBzgnAmK13q9Urgch+sKYeIhdymjuMQta
  5.     x15OklctmrZtqre5kwPUosG3/B2/ikuPYElcHgGPL4uL5Em6S5C/oozfkYzhwRrT
  6.     SQzvYjsE4I34To4UdE9KA97wrQjGoz2Bx72WDLyWwctD3DKQtYeHXswXXtXwKfjQ
  7.     7Fy4+Bf5IPh76dA8NJ6UtjjLIDlKqdxLW4atHe6xWFaJ+XdLUtsAroZcXBeWDCPa
  8.     buXCDscJcLJRKZVc62gOZXXtPfoHqvUPp3nuLA4YjH9bphbrMWMf810Wxz9JTd3v
  9.     yWgGqNY0zbBqeZoGv+TuExlRHT8ASGFS9SVDABEBAAG0NUdpdEh1YiAod2ViLWZs
  10.     b3cgY29tbWl0IHNpZ25pbmcpIDxub3JlcGx5QGdpdGh1Yi5jb20+iQEiBBMBCAAW
  11.     BQJZlGhBCRBK7hj4Ov3rIwIbAwIZAQAAmQEH/iATWFmi2oxlBh3wAsySNCNV4IPf
  12.     DDMeh6j80WT7cgoX7V7xqJOxrfrqPEthQ3hgHIm7b5MPQlUr2q+UPL22t/I+ESF6
  13.     9b0QWLFSMJbMSk+BXkvSjH9q8jAO0986/pShPV5DU2sMxnx4LfLfHNhTzjXKokws
  14.     +8ptJ8uhMNIDXfXuzkZHIxoXk3rNcjDN5c5X+sK8UBRH092BIJWCOfaQt7v7wig5
  15.     4Ra28pM9GbHKXVNxmdLpCFyzvyMuCmINYYADsC848QQFFwnd4EQnupo6QvhEVx1O
  16.     j7wDwvuH5dCrLuLwtwXaQh0onG4583p0LGms2Mf5F+Ick6o/4peOlBoZz48=
  17.     =Bvzs
  18.     -----END PGP PUBLIC KEY BLOCK-----

Configuring a project to enforce signature verification

Once you have imported the GnuPG keys to ArgoCD, you must now configure the project to enforce the verification of commit signatures with the imported keys.

Configuring using the CLI

Adding a key ID to list of allowed keys

To add a key ID to the list of allowed GnuPG keys for a project, you can use the argocd proj add-signature-key command, i.e. the following command would add the key ID 4AEE18F83AFDEB23 to the project named myproj:

  1. argocd proj add-signature-key myproj 4AEE18F83AFDEB23

Removing a key ID from the list of allowed keys

Similarly, you can remove a key ID from the list of allowed GnuPG keys for a project using the argocd proj remove-signature-key command, i.e. to remove the key added above from project myproj, use the command:

  1. argocd proj remove-signature-key myproj 4AEE18F83AFDEB23

Showing allowed key IDs for a project

To see which key IDs are allowed for a given project, you can inspect the output of the argocd proj get command, i.e for a project named gpg:

  1. $ argocd proj get gpg
  2. Name:                        gpg
  3. Description:                 GnuPG verification
  4. Destinations:                *,*
  5. Repositories:                *
  6. Allowed Cluster Resources:   */*
  7. Denied Namespaced Resources: <none>
  8. Signature keys:              4AEE18F83AFDEB23, 07E34825A909B250
  9. Orphaned Resources:          disabled

Override list of key IDs

You can also explicitly set the currently allowed keys with one or more new keys using the argocd proj set command in combination with the --signature-keys flag, which you can use to specify a comma separated list of allowed key IDs:

  1. argocd proj set myproj --signature-keys 4AEE18F83AFDEB23,07E34825A909B250

The --signature-keys flag can also be used on project creation, i.e. the argocd proj create command.

Configure using the Web UI

You can configure the GnuPG key IDs required for signature verification using the web UI, in the Project configuration. Navigate to the Settings page and select the Projects module, then click on the project you want to configure.

From the project’s details page, click Edit and find the Required signature keys section, where you can add or remove the key IDs for signature verification. After you have modified your project, click Update to save the changes.

Configure using declarative setup

You can specify the key IDs required for signature verification in the project manifest within the signatureKeys section, i.e:

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: AppProject
  3. metadata:
  4.   name: gpg
  5.   namespace: argocd
  6. spec:
  7.   clusterResourceWhitelist:
  8.   - group: '*'
  9.     kind: '*'
  10.   description: GnuPG verification
  11.   destinations:
  12.   - namespace: '*'
  13.     server: '*'
  14.   namespaceResourceWhitelist:
  15.   - group: '*'
  16.     kind: '*'
  17.   signatureKeys:
  18.   - keyID: 4AEE18F83AFDEB23
  19.   sourceRepos:
  20.   - '*'

signatureKeys is an array of SignatureKey objects, whose only property is keyID at the moment.

Troubleshooting

Disabling the feature

The GnuPG feature can be completely disabled if desired. In order to disable it, set the environment variable ARGOCD_GPG_ENABLED to false for the pod templates of the argocd-server, argocd-repo-server and argocd-application-controller deployments.

After the pods have been restarted, the GnuPG feature is disabled.

GnuPG key ring

The GnuPG key ring used for signature verification is maintained within the pods of argocd-repo-server. The keys in the keyring are synchronized to the configuration stored in the argocd-gpg-keys-cm ConfigMap resource, which is volume-mounted to the argocd-repo-server pods.

Note

The GnuPG key ring in the pods is transient and gets recreated from the configuration on each restart of the pods. You should never add or remove keys manually to the key ring, because your changes will be lost. Also, any of the private keys found in the key ring are transient and will be regenerated upon each restart. The private key is only used to build the trust DB for the running pod.

To check whether the keys are actually in sync, you can kubectl exec into the repository server’s pods and inspect the key ring, which is located at path /app/config/gpg/keys

  1. $ kubectl exec -it argocd-repo-server-7d6bdfdf6d-hzqkg bash
  2. argocd@argocd-repo-server-7d6bdfdf6d-hzqkg:~$ GNUPGHOME=/app/config/gpg/keys gpg --list-keys
  3. /app/config/gpg/keys/pubring.kbx
  4. --------------------------------
  5. pub   rsa2048 2020-06-15 [SC] [expires: 2020-12-12]
  6.       D48F075D818A813C436914BC9324F0D2144753B1
  7. uid           [ultimate] Anon Ymous (ArgoCD key signing key) <noreply@argoproj.io>
  9. pub   rsa2048 2017-08-16 [SC]
  10.       5DE3E0509C47EA3CF04A42D34AEE18F83AFDEB23
  11. uid           [ultimate] GitHub (web-flow commit signing) <noreply@github.com>
  13. argocd@argocd-repo-server-7d6bdfdf6d-hzqkg:~$

If the key ring stays out of sync with your configuration after you have added or removed keys for a longer period of time, you might want to restart your argocd-repo-server pods. If such a problem persists, please consider raising a bug report.