Customize Calico configuration

Big picture

Perform common customizations of a Calico installation.

Concepts

Calico operator

Calico is installed by an operator which manages the installation, upgrade, and general lifecycle of a Calico cluster. The operator is installed directly on the cluster as a Deployment, and is configured through one or more custom Kubernetes API resources.

Calico manifests

Calico can also be installed using raw manifests as an alternative to the operator. The manifests contain the necessary resources for installing Calico on each node in your Kubernetes cluster. Using manifests is not recommended as they cannot automatically manage the lifecycle of the Calico as the operator does. However, manifests may be useful for clusters that require highly specific modifications to the underlying Kubernetes resources.

How to

  • Operator
  • Manifest

About customizing an operator install

Operator installations read their configuration from a specific set of Kubernetes APIs. These APIs are installed on the cluster as part of tigera-operator.yaml in the operator.tigera.io/v1 API group.

  • Installation: a singleton resource with name “default” that configures common installation parameters for a Calico cluster.
  • APIServer: a singleton resource with name “default” that configures installation of the Calico API server extension.

Configure the pod IP range

For many environments, Calico will auto-detect the correct pod IP range to use, or select an unused range on the cluster.

You can select a specific pod IP range by modifying the spec.calicoNetwork.ipPools array in the Installation API resource.

  1. kind: Installation
  2. apiVersion: operator.tigera.io/v1
  3. metadata:
  4. name: default
  5. spec:
  6. calicoNetwork:
  7. ipPools:
  8. - cidr: 198.51.100.0/24

Customize Calico configuration - 图1note

The the ipPools array can take at most one IPv4 and one IPv6 CIDR, and only takes effect when installing Calico for the first time on a given cluster. To add additional pools, see the IPPool API.

Use VXLAN

You can enable VXLAN in a cluster by setting the option on your IPv4 pool. You can also disable BGP via the spec.calicoNetwork.bgp field.

  1. kind: Installation
  2. apiVersion: operator.tigera.io/v1
  3. metadata:
  4. name: default
  5. spec:
  6. calicoNetwork:
  7. bgp: Disabled
  8. ipPools:
  9. - cidr: 198.51.100.0/24
  10. encapsulation: VXLAN

We provide a number of manifests to make deployment of Calico easy. You can optionally modify the manifests before applying them. Or you can modify the manifest and reapply it to change settings as needed.

About customizing Calico manifests

Each manifest contains all the necessary resources for installing Calico on each node in your Kubernetes cluster.

It installs the following Kubernetes resources:

  • Installs the calico/node container on each host using a DaemonSet.
  • Installs the Calico CNI binaries and network config on each host using a DaemonSet.
  • Runs calico/kube-controllers as a deployment.
  • The calico-etcd-secrets secret, which optionally allows for providing etcd TLS assets.
  • The calico-config ConfigMap, which contains parameters for configuring the install.

The sections that follow discuss the configurable parameters in greater depth.

Configure the pod IP range

Calico IPAM assigns IP addresses from IP pools.

To change the default IP range used for pods, modify the CALICO_IPV4POOL_CIDR section of the calico.yaml manifest. For more information, see Configuring calico/node.

Configuring IP-in-IP

By default, the manifests enable IP-in-IP encapsulation across subnets. Many users may want to disable IP-in-IP encapsulation, such as under the following circumstances.

  • Their cluster is running in a properly configured AWS VPC.
  • All their Kubernetes nodes are connected to the same layer 2 network.
  • They intend to use BGP peering to make their underlying infrastructure aware of pod IP addresses.

To disable IP-in-IP encapsulation, modify the CALICO_IPV4POOL_IPIP section of the manifest. For more information, see Configuring calico/node.

Switching from IP-in-IP to VXLAN

By default, the Calico manifests enable IP-in-IP encapsulation. If you are on a network that blocks IP-in-IP, such as Azure, you may wish to switch to Calico’s VXLAN encapsulation mode. To do this at install time (so that Calico creates the default IP pool with VXLAN and no IP-in-IP configuration has to be undone):

  • Start with one of the Calico for policy and networking manifests.
  • Replace environment variable name CALICO_IPV4POOL_IPIP withCALICO_IPV4POOL_VXLAN. Leave the value of the new variable as “Always”.
  • Optionally, (to save some resources if you’re running a VXLAN-only cluster) completely disable Calico’s BGP-based networking:
    • Replace calico_backend: "bird" with calico_backend: "vxlan". This disables BIRD.
    • Comment out the line - -bird-ready and - -bird-live from the calico/node readiness/liveness check (otherwise disabling BIRD will cause the readiness/liveness check to fail on every node):
  1. livenessProbe:
  2. exec:
  3. command:
  4. - /bin/calico-node
  5. - -felix-live
  6. - -bird-live
  7. readinessProbe:
  8. exec:
  9. command:
  10. - /bin/calico-node
  11. - -bird-ready
  12. - -felix-ready

For more information on calico/node’s configuration variables, including additional VXLAN settings, see Configuring calico/node.

Customize Calico configuration - 图2note

The CALICO_IPV4POOL_VXLAN environment variable only takes effect when the first calico/node to start creates the default IP pool. It has no effect after the pool has already been created. To switch to VXLAN mode after installation time, use calicoctl to modify the IPPool resource.

Configuring etcd

By default, these manifests do not configure secure access to etcd and assume an etcd proxy is running on each host. The following configuration options let you specify custom etcd cluster endpoints as well as TLS.

The following table outlines the supported ConfigMap options for etcd:

OptionDescriptionDefault
etcd_endpointsComma-delimited list of etcd endpoints to connect to.http://127.0.0.1:2379
etcd_caThe file containing the root certificate of the CA that issued the etcd server certificate. Configures calico/node, the CNI plugin, and the Kubernetes controllers to trust the signature on the certificates provided by the etcd server.None
etcd_keyThe file containing the private key of the calico/node, the CNI plugin, and the Kubernetes controllers client certificate. Enables these components to participate in mutual TLS authentication and identify themselves to the etcd server.None
etcd_certThe file containing the client certificate issued to calico/node, the CNI plugin, and the Kubernetes controllers. Enables these components to participate in mutual TLS authentication and identify themselves to the etcd server.None

To use these manifests with a TLS-enabled etcd cluster you must do the following:

  1. Download the v3.24 manifest that corresponds to your installation method.

    Calico for policy and networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico-etcd.yaml -O

    Calico for policy and flannel for networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/canal.yaml -O
  2. Within the ConfigMap section, uncomment the etcd_ca, etcd_key, and etcd_cert lines so that they look as follows.

    1. etcd_ca: '/calico-secrets/etcd-ca'
    2. etcd_cert: '/calico-secrets/etcd-cert'
    3. etcd_key: '/calico-secrets/etcd-key'
  3. Ensure that you have three files, one containing the etcd_ca value, another containing the etcd_key value, and a third containing the etcd_cert value.

  4. Using a command like the following to strip the newlines from the files and base64-encode their contents.

    1. cat <file> | base64 -w 0
  5. In the Secret named calico-etcd-secrets, uncomment etcd_ca, etcd_key, and etcd_cert and paste in the appropriate base64-encoded values.

    1. apiVersion: v1
    2. kind: Secret
    3. type: Opaque
    4. metadata:
    5. name: calico-etcd-secrets
    6. namespace: kube-system
    7. data:
    8. Populate the following files with etcd TLS configuration if desired, but leave blank if
    9. not using TLS for etcd.
    10. This self-hosted install expects three files with the following names. The values
    11. should be base64 encoded strings of the entire contents of each file.
    12. etcd-key: LS0tLS1CRUdJTiB...VZBVEUgS0VZLS0tLS0=
    13. etcd-cert: LS0tLS1...ElGSUNBVEUtLS0tLQ==
    14. etcd-ca: LS0tLS1CRUdJTiBD...JRklDQVRFLS0tLS0=
  6. Apply the manifest.

    Calico for policy and networking

    1. kubectl apply -f calico.yaml

    Calico for policy and flannel for networking

    1. kubectl apply -f canal.yaml

Authorization options

Calico’s manifests assign its components one of two service accounts. Depending on your cluster’s authorization mode, you’ll want to back these service accounts with the necessary permissions.

Other configuration options

The following table outlines the remaining supported ConfigMap options.

OptionDescriptionDefault
calico_backendThe backend to use.bird
cni_network_configThe CNI Network config to install on each node. Supports templating as described below.

CNI network configuration template

The cni_network_config configuration option supports the following template fields, which will be filled in automatically by the calico/cni container:

FieldSubstituted with
KUBERNETES_SERVICE_HOSTThe Kubernetes service Cluster IP, e.g 10.0.0.1
KUBERNETES_SERVICE_PORTThe Kubernetes service port, e.g., 443
SERVICEACCOUNT_TOKENThe service account token for the namespace, if one exists.
ETCD_ENDPOINTSThe etcd endpoints specified in etcdendpoints.
KUBECONFIG_FILEPATHThe path to the automatically generated kubeconfig file in the same directory as the CNI network configuration file.
ETCD_KEY_FILEThe path to the etcd key file installed to the host. Empty if no key is present.
ETCD_CERT_FILEThe path to the etcd certificate file installed to the host, empty if no cert present.
_ETCD_CA_CERT_FILEThe path to the etcd certificate authority file installed to the host. Empty if no certificate authority is present.

About customizing application layer policy manifests

Instead of installing from our pre-modified Istio manifests, you may wish to customize your Istio install or use a different Istio version. This section walks you through the necessary changes to a generic Istio install manifest to allow application layer policy to operate.

The standard Istio manifests for the sidecar injector include a ConfigMap that contains the template used when adding pods to the cluster. The template adds an init container and the Envoy sidecar. Application layer policy requires an additional lightweight sidecar called Dikastes which receives Calico policy from Felix and applies it to incoming connections and requests.

If you haven’t already done so, download an Istio release and untar it to a working directory.

Open the install/kubernetes/istio-demo-auth.yaml file in an editor, and locate the istio-sidecar-injector ConfigMap. In the existing istio-proxy container, add a new volumeMount.

  1. - mountPath: /var/run/dikastes
  2. name: dikastes-sock

Add a new container to the template.

  1. - name: dikastes
  2. image: calico/dikastes:v3.24.5
  3. args: ["server", "-l", "/var/run/dikastes/dikastes.sock", "-d", "/var/run/felix/nodeagent/socket"]
  4. securityContext:
  5. allowPrivilegeEscalation: false
  6. livenessProbe:
  7. exec:
  8. command:
  9. - /healthz
  10. - liveness
  11. initialDelaySeconds: 3
  12. periodSeconds: 3
  13. readinessProbe:
  14. exec:
  15. command:
  16. - /healthz
  17. - readiness
  18. initialDelaySeconds: 3
  19. periodSeconds: 3
  20. volumeMounts:
  21. - mountPath: /var/run/dikastes
  22. name: dikastes-sock
  23. - mountPath: /var/run/felix
  24. name: felix-sync

Add two new volumes.

  1. - name: dikastes-sock
  2. emptyDir:
  3. medium: Memory
  4. - name: felix-sync
  5. csi:
  6. driver: 'csi.tigera.io'

The volumes you added are used to create Unix domain sockets that allow communication between Envoy and Dikastes and between Dikastes and Felix. Once created, a Unix domain socket is an in-memory communications channel. The volumes are not used for any kind of stateful storage on disk.

Refer to the Calico ConfigMap manifest for an example with the above changes.