Using manual mode with STS

Manual mode with STS is available as a Technology Preview for Amazon Web Services (AWS).

Support for AWS Secure Token Service (STS) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

In manual mode with STS, the individual OKD cluster components use AWS Secure Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

Requests for new and refreshed credentials are automated by using an appropriately configured AWS IAM OpenID Connect (OIDC) identity provider, combined with AWS IAM roles. OKD signs service account tokens that are trusted by AWS IAM, and can be projected into a pod and used for authentication. Tokens are refreshed after one hour.

Detailed authentication flow between AWS and the cluster when using AWS STS

Figure 1. STS authentication flow

Using manual mode with STS changes the content of the AWS credentials that are provided to individual OKD components.

AWS secret format using long-lived credentials

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4. namespace: <target-namespace> (1)
  5. name: <target-secret-name> (2)
  6. data:
  7. aws_access_key_id: <base64-encoded-access-key-id>
  8. aws_secret_access_key: <base64-encoded-secret-access-key>
1The namespace for the component.
2The name of the component secret.

AWS secret format with STS

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4. namespace: <target-namespace> (1)
  5. name: <target-secret-name> (2)
  6. data:
  7. role_name: <operator-role-name> (3)
  8. web_identity_token_file: <path-to-token> (4)
1The namespace for the component.
2The name of the component secret.
3The IAM role for the component.
4The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

Installing an OKD cluster configured for manual mode with STS

To install a cluster that is configured to use the CCO in manual mode with STS in OKD version 4.7:

  1. Create the required AWS resources

  2. Run the OKD installer

  3. Verify that the cluster is using short-lived credentials

Creating AWS resources manually

To install an OKD cluster that is configured to use the CCO in manual mode with STS, you must first manually create the required AWS resources.

Procedure

  1. Generate a private key to sign the ServiceAccount object:

    1. $ openssl genrsa -out sa-signer 4096
  2. Generate a ServiceAccount object public key:

    1. $ openssl rsa -in sa-signer -pubout -out sa-signer.pub
  3. Create an S3 bucket to hold the OIDC configuration:

    1. $ aws s3api create-bucket --bucket <oidc_bucket_name> --region <aws_region> --create-bucket-configuration LocationConstraint=<aws_region>

    If the value of <aws_region> is us-east-1, do not specify the LocationConstraint parameter.

  4. Retain the S3 bucket URL:

    1. OPENID_BUCKET_URL="https://<oidc_bucket_name>.s3.<aws_region>.amazonaws.com"
  5. Build an OIDC configuration:

    1. Create a file named keys.json that contains the following information:

      1. {
      2. "keys": [
      3. {
      4. "use": "sig",
      5. "kty": "RSA",
      6. "kid": "<public_signing_key_id>",
      7. "alg": "RS256",
      8. "n": "<public_signing_key_modulus>",
      9. "e": "<public_signing_key_exponent>"
      10. }
      11. ]
      12. }

      Where:

      • <public_signing_key_id> is generated from the public key with:

        1. $ openssl rsa -in sa-signer.pub -pubin --outform DER | openssl dgst -binary -sha256 | openssl base64 | tr '/+' '_-' | tr -d '='

        This command converts the public key to DER format, performs a SHA-256 checksum on the binary representation, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.

      • <public_signing_key_modulus> is generated from the public key with:

        1. $ openssl rsa -pubin -in sa-signer.pub -modulus -noout | sed -e 's/Modulus=//' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='

        This command prints the modulus of the public key, extracts the hex representation of the modulus, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.

      • <public_signing_key_exponent> is generated from the public key with:

        1. $ printf "%016x" $(openssl rsa -pubin -in sa-signer.pub -noout -text | grep Exponent | awk '{ print $2 }') | awk '{ sub(/(00)+/, "", $1); print $1 }' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='

        This command extracts the decimal representation of the public key exponent, prints it as hex with a padded 0 if needed, removes leading 00 pairs, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to use only characters that can be used in a URL.

    2. Create a file named openid-configuration that contains the following information:

      1. {
      2. "issuer": "$OPENID_BUCKET_URL",
      3. "jwks_uri": "${OPENID_BUCKET_URL}/keys.json",
      4. "response_types_supported": [
      5. "id_token"
      6. ],
      7. "subject_types_supported": [
      8. "public"
      9. ],
      10. "id_token_signing_alg_values_supported": [
      11. "RS256"
      12. ],
      13. "claims_supported": [
      14. "aud",
      15. "exp",
      16. "sub",
      17. "iat",
      18. "iss",
      19. "sub"
      20. ]
      21. }
  6. Upload the OIDC configuration:

    1. $ aws s3api put-object --bucket <oidc_bucket_name> --key keys.json --body ./keys.json
    1. $ aws s3api put-object --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --body ./openid-configuration

    Where <oidc_bucket_name> is the S3 bucket that was created to hold the OIDC configuration.

  7. Allow the AWS IAM OpenID Connect (OIDC) identity provider to read these files:

    1. $ aws s3api put-object-acl --bucket <oidc_bucket_name> --key keys.json --acl public-read
    1. $ aws s3api put-object-acl --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --acl public-read
  8. Create an AWS IAM OIDC identity provider:

    1. Get the certificate chain from the server that hosts the OIDC configuration:

      1. $ echo | openssl s_client -servername $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com -connect $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com:443 -showcerts 2>/dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".pem"; print >out}'
    2. Calculate the fingerprint for the certificate at the root of the chain:

      1. $ export BUCKET_FINGERPRINT=$(openssl x509 -in cert<number>.pem -fingerprint -noout | sed -e 's/.*Fingerprint=//' -e 's/://g')

      Where <number> is the highest number in the files that were saved. For example, if 2 is the highest number in the files that were saved, use cert2.pem.

    3. Create the identity provider:

      1. $ aws iam create-open-id-connect-provider --url $OPENID_BUCKET_URL --thumbprint-list $BUCKET_FINGERPRINT --client-id-list openshift sts.amazonaws.com
    4. Retain the returned ARN of the newly created identity provider. This ARN is later referred to as <aws_iam_openid_arn>.

  9. Generate IAM roles:

    1. Locate all CredentialsRequest CRs in this release image that target the cloud you are deploying on:

      1. $ oc adm release extract quay.io/openshift-release-dev/ocp-release:4.<y>.<z>-x86_64 --credentials-requests --cloud=aws

      Where <y> and <z> are the numbers corresponding to the version of OKD you are installing.

    2. For each CredentialsRequest CR, create an IAM role of type Web identity using the previously created IAM Identity Provider that grants the necessary permissions and establishes a trust relationship that trusts the identity provider previously created.

      For example, for the openshift-machine-api-operator CredentialsRequest CR in 0000_30_machine-api-operator_00_credentials-request.yaml, create an IAM role that allows an identity from the created OIDC provider created for the cluster, similar to the following:

      1. {
      2. "Role": {
      3. "Path": "/",
      4. "RoleName": "openshift-machine-api-aws-cloud-credentials",
      5. "RoleId": "ARSOMEROLEID",
      6. "Arn": "arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials",
      7. "CreateDate": "2021-01-06T15:54:13Z",
      8. "AssumeRolePolicyDocument": {
      9. "Version": "2012-10-17",
      10. "Statement": [
      11. {
      12. "Effect": "Allow",
      13. "Principal": {
      14. "Federated": "<aws_iam_openid_arn>"
      15. },
      16. "Action": "sts:AssumeRoleWithWebIdentity",
      17. "Condition": {
      18. "StringEquals": {
      19. "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com/$BUCKET_NAME:aud": "openshift"
      20. }
      21. }
      22. }
      23. ]
      24. },
      25. "Description": "OpenShift role for openshift-machine-api/aws-cloud-credentials",
      26. "MaxSessionDuration": 3600,
      27. "RoleLastUsed": {
      28. "LastUsedDate": "2021-02-03T02:51:24Z",
      29. "Region": "<aws_region>"
      30. }
      31. }
      32. }

      Where <aws_iam_openid_arn> is the returned ARN of the newly created identity provider.

    3. To further restrict the role such that only specific cluster ServiceAccount objects can assume the role, modify the trust relationship of each role by updating the .Role.AssumeRolePolicyDocument.Statement[].Condition field to the specific ServiceAccount objects for each component.

      • Modify the trust relationship of the cluster-image-registry-operator role to have the following condition:

        1. "Condition": {
        2. "StringEquals": {
        3. "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
        4. "system:serviceaccount:openshift-image-registry:registry",
        5. "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator"
        6. ]
        7. }
        8. }
      • Modify the trust relationship of the openshift-ingress-operator to have the following condition:

        1. "Condition": {
        2. "StringEquals": {
        3. "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
        4. "system:serviceaccount:openshift-ingress-operator:ingress-operator"
        5. ]
        6. }
        7. }
      • Modify the trust relationship of the openshift-cluster-csi-drivers to have the following condition:

        1. "Condition": {
        2. "StringEquals": {
        3. "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
        4. "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-operator",
        5. "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-controller-sa"
        6. ]
        7. }
        8. }
      • Modify the trust relationship of the openshift-machine-api to have the following condition:

        1. "Condition": {
        2. "StringEquals": {
        3. "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
        4. "system:serviceaccount:openshift-machine-api:machine-api-controllers"
        5. ]
        6. }
        7. }
  1. For each IAM role, attach an IAM policy to the role that reflects the required permissions from the corresponding CredentialsRequest objects.

    For example, for openshift-machine-api, attach an IAM policy similar to the following:

    1. {
    2. "RoleName": "openshift-machine-api-aws-cloud-credentials",
    3. "PolicyName": "openshift-machine-api-aws-cloud-credentials",
    4. "PolicyDocument": {
    5. "Version": "2012-10-17",
    6. "Statement": [
    7. {
    8. "Effect": "Allow",
    9. "Action": [
    10. "ec2:CreateTags",
    11. "ec2:DescribeAvailabilityZones",
    12. "ec2:DescribeDhcpOptions",
    13. "ec2:DescribeImages",
    14. "ec2:DescribeInstances",
    15. "ec2:DescribeSecurityGroups",
    16. "ec2:DescribeSubnets",
    17. "ec2:DescribeVpcs",
    18. "ec2:RunInstances",
    19. "ec2:TerminateInstances",
    20. "elasticloadbalancing:DescribeLoadBalancers",
    21. "elasticloadbalancing:DescribeTargetGroups",
    22. "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
    23. "elasticloadbalancing:RegisterTargets",
    24. "iam:PassRole",
    25. "iam:CreateServiceLinkedRole"
    26. ],
    27. "Resource": "*"
    28. },
    29. {
    30. "Effect": "Allow",
    31. "Action": [
    32. "kms:Decrypt",
    33. "kms:Encrypt",
    34. "kms:GenerateDataKey",
    35. "kms:GenerateDataKeyWithoutPlainText",
    36. "kms:DescribeKey"
    37. ],
    38. "Resource": "*"
    39. },
    40. {
    41. "Effect": "Allow",
    42. "Action": [
    43. "kms:RevokeGrant",
    44. "kms:CreateGrant",
    45. "kms:ListGrants"
    46. ],
    47. "Resource": "*",
    48. "Condition": {
    49. "Bool": {
    50. "kms:GrantIsForAWSResource": true
    51. }
    52. }
    53. }
    54. ]
    55. }
    56. }
  2. Prepare to run the OKD installer:

    1. Create the install-config.yaml file:

      1. $ ./openshift-install create install-config
    2. Configure the cluster to install with the CCO in manual mode:

      1. $ echo "credentialsMode: Manual" >> install-config.yaml
    3. Create install manifests:

      1. $ ./openshift-install create manifests
    4. Create a tls directory, and copy the private key generated previously there:

      The target file name must be ./tls/bound-service-account-signing-key.key.

      1. $ mkdir tls ; cp <path_to_service_account_signer> ./tls/bound-service-account-signing-key.key
    5. Create a custom Authentication CR with the file name cluster-authentication-02-config.yaml:

      1. $ cat << EOF > manifests/cluster-authentication-02-config.yaml
      2. apiVersion: config.openshift.io/v1
      3. kind: Authentication
      4. metadata:
      5. name: cluster
      6. spec:
      7. serviceAccountIssuer: $OPENID_BUCKET_URL
      8. EOF
    6. For each CredentialsRequest CR that is extracted from the release image, create a secret with the target namespace and target name that is indicated in each CredentialsRequest, substituting the AWS IAM role ARN created previously for each component:

      Example secret manifest for openshift-machine-api:

      1. $ cat manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
      2. apiVersion: v1
      3. stringData:
      4. credentials: |-
      5. [default]
      6. role_arn = arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials
      7. web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
      8. kind: Secret
      9. metadata:
      10. name: aws-cloud-credentials
      11. namespace: openshift-machine-api
      12. type: Opaque

Running the installer

  • Run the OKD installer:

    1. $ ./openshift-install create cluster

Verifying the installation

  1. Connect to the OKD cluster.

  2. Verify that the cluster does not have root credentials:

    1. $ oc get secrets -n kube-system aws-creds

    The output should look similar to:

    1. Error from server (NotFound): secrets "aws-creds" not found
  3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

    Example command with the Image Registry Operator

    1. $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

    The output should show the role and web identity token that are used by the component and look similar to:

    Example output with the Image Registry Operator

    1. [default]
    2. role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
    3. web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token