Secure Calico Prometheus endpoints

About securing access to Calico’s metrics endpoints

When using Calico with Prometheus metrics enabled, we recommend using network policy to limit access to Calico’s metrics endpoints.

Prerequisites

Choosing an approach

This guide provides two example workflows for creating network policies to limit access to Calico’s Prometheus metrics. Choosing an approach depends on your requirements.

  • Using a deny-list approach

    This approach allows all traffic to your hosts by default, but lets you limit access to specific ports using Calico policy. This approach allows you to restrict access to specific ports, while leaving other host traffic unaffected.

  • Using an allow-list approach

    This approach denies traffic to and from your hosts by default, and requires that all desired communication be explicitly allowed by a network policy. This approach is more secure because only explicitly-allowed traffic will get through, but it requires you to know all the ports that should be open on the host.

Using a deny-list approach

Overview

The basic process is as follows:

  1. Create a default network policy that allows traffic to and from your hosts.
  2. Create host endpoints for each node that you’d like to secure.
  3. Create a network policy that denies unwanted traffic to the Calico metrics endpoints.
  4. Apply labels to allow access to the Prometheus metrics.

Example for calico/node

This example shows how to limit access to the calico/node Prometheus metrics endpoints.

  1. Create a default network policy to allow host traffic

    First, create a default-allow policy. Do this first to avoid a drop in connectivity when adding the host endpoints later, since host endpoints with no policy default to deny.

    To do this, create a file named default-host-policy.yaml with the following contents.

    1. apiVersion: projectcalico.org/v3
    2. kind: GlobalNetworkPolicy
    3. metadata:
    4. name: default-host
    5. spec:
    6. # Select all Calico nodes.
    7. selector: running-calico == "true"
    8. order: 5000
    9. ingress:
    10. - action: Allow
    11. egress:
    12. - action: Allow

    Then, use calicoctl to apply this policy.

    1. calicoctl apply -f default-host-policy.yaml
  2. List the nodes on which Calico is running with the following command.

    1. calicoctl get nodes

    In this case, we have two nodes in the cluster.

    1. NAME
    2. kubeadm-master
    3. kubeadm-node-0
  3. Create host endpoints for each Calico node.

    Create a file named host-endpoints.yaml containing a host endpoint for each node listed above. In this example, the contents would look like this.

    1. apiVersion: projectcalico.org/v3
    2. kind: HostEndpoint
    3. metadata:
    4. name: kubeadm-master.eth0
    5. labels:
    6. running-calico: 'true'
    7. spec:
    8. node: kubeadm-master
    9. interfaceName: eth0
    10. expectedIPs:
    11. - 10.100.0.15
    12. ---
    13. apiVersion: projectcalico.org/v3
    14. kind: HostEndpoint
    15. metadata:
    16. name: kubeadm-node-0.eth0
    17. labels:
    18. running-calico: 'true'
    19. spec:
    20. node: kubeadm-node-0
    21. interfaceName: eth0
    22. expectedIPs:
    23. - 10.100.0.16

    In this file, replace eth0 with the desired interface name on each node, and populate the expectedIPs section with the IP addresses on that interface.

    Note the use of a label to indicate that this host endpoint is running Calico. The label matches the selector of the network policy created in step 1.

    Then, use calicoctl to apply the host endpoints with the following command.

    1. calicoctl apply -f host-endpoints.yaml
  4. Create a network policy that restricts access to the calico/node Prometheus metrics port.

    Now let’s create a network policy that limits access to the Prometheus metrics port such that only endpoints with the label calico-prometheus-access: true can access the metrics.

    To do this, create a file named calico-prometheus-policy.yaml with the following contents.

    1. # Allow traffic to Prometheus only from sources that are
    2. # labeled as such, but don't impact any other traffic.
    3. apiVersion: projectcalico.org/v3
    4. kind: GlobalNetworkPolicy
    5. metadata:
    6. name: restrict-calico-node-prometheus
    7. spec:
    8. # Select all Calico nodes.
    9. selector: running-calico == "true"
    10. order: 500
    11. types:
    12. - Ingress
    13. ingress:
    14. # Deny anything that tries to access the Prometheus port
    15. # but that doesn't match the necessary selector.
    16. - action: Deny
    17. protocol: TCP
    18. source:
    19. notSelector: calico-prometheus-access == "true"
    20. destination:
    21. ports:
    22. - 9091

    This policy selects all endpoints that have the label running-calico: true, and enforces a single ingress deny rule. The ingress rule denies traffic to port 9091 unless the source of traffic has the label calico-prometheus-access: true, meaning all Calico workload endpoints, host endpoints, and global network sets that do not have the label, as well as any other network endpoints unknown to Calico.

    Then, use calicoctl to apply this policy.

    1. calicoctl apply -f calico-prometheus-policy.yaml
  5. Apply labels to any endpoints that should have access to the metrics.

    At this point, only endpoints that have the label calico-prometheus-access: true can reach Calico’s Prometheus metrics endpoints on each node. To grant access, simply add this label to the desired endpoints.

    For example, to allow access to a Kubernetes pod you can run the following command.

    1. kubectl label pod my-prometheus-pod calico-prometheus-access=true

    If you would like to grant access to a specific IP network, you can create a global network set using calicoctl.

    For example, you might want to grant access to your management subnets.

    1. apiVersion: projectcalico.org/v3
    2. kind: GlobalNetworkSet
    3. metadata:
    4. name: calico-prometheus-set
    5. labels:
    6. calico-prometheus-access: 'true'
    7. spec:
    8. nets:
    9. - 172.15.0.0/24
    10. - 172.101.0.0/24

Additional steps for Typha deployments

If your Calico installation uses the Kubernetes API datastore and has greater than 50 nodes, it is likely that you have installed Typha. This section shows how to use an additional network policy to secure the Typha Prometheus endpoints.

After following the steps above, create a file named typha-prometheus-policy.yaml with the following contents.

  1. # Allow traffic to Prometheus only from sources that are
  2. # labeled as such, but don't impact any other traffic.
  3. apiVersion: projectcalico.org/v3
  4. kind: GlobalNetworkPolicy
  5. metadata:
  6. name: restrict-calico-node-prometheus
  7. spec:
  8. # Select all Calico nodes.
  9. selector: running-calico == "true"
  10. order: 500
  11. types:
  12. - Ingress
  13. ingress:
  14. # Deny anything that tries to access the Prometheus port
  15. # but that doesn't match the necessary selector.
  16. - action: Deny
  17. protocol: TCP
  18. source:
  19. notSelector: calico-prometheus-access == "true"
  20. destination:
  21. ports:
  22. - 9093

This policy selects all endpoints that have the label running-calico: true, and enforces a single ingress deny rule. The ingress rule denies traffic to port 9093 unless the source of traffic has the label calico-prometheus-access: true, meaning all Calico workload endpoints, host endpoints, and global network sets that do not have the label, as well as any other network endpoints unknown to Calico.

Then, use calicoctl to apply this policy.

  1. calicoctl apply -f typha-prometheus-policy.yaml

Example for kube-controllers

If your Calico installation exposes metrics from kube-controllers, you can limit access to those metrics with the following network policy.

Create a file named kube-controllers-prometheus-policy.yaml with the following contents.

  1. apiVersion: projectcalico.org/v3
  2. kind: NetworkPolicy
  3. metadata:
  4. name: restrict-kube-controllers-prometheus
  5. namespace: calico-system
  6. spec:
  7. # Select kube-controllers.
  8. selector: k8s-app == "calico-kube-controllers"
  9. order: 500
  10. types:
  11. - Ingress
  12. ingress:
  13. # Deny anything that tries to access the Prometheus port
  14. # but that doesn't match the necessary selector.
  15. - action: Deny
  16. protocol: TCP
  17. source:
  18. notSelector: calico-prometheus-access == "true"
  19. destination:
  20. ports:
  21. - 9094

Secure Calico Prometheus endpoints - 图1note

The above policy is installed in the calico-system namespace. If your cluster has Calico installed in the kube-system namespace, you will need to create the policy in that namespace instead.

Then, use calicoctl to apply this policy.

  1. calicoctl apply -f kube-controllers-prometheus-policy.yaml

Using an allow-list approach

Overview

The basic process is as follows:

  1. Create host endpoints for each node that you’d like to secure.
  2. Create a network policy that allows desired traffic to the Calico metrics endpoints.
  3. Apply labels to allow access to the Prometheus metrics.

Example for calico/node

  1. List the nodes on which Calico is running with the following command.

    1. calicoctl get nodes

    In this case, we have two nodes in the cluster.

    1. NAME
    2. kubeadm-master
    3. kubeadm-node-0
  2. Create host endpoints for each Calico node.

    Create a file named host-endpoints.yaml containing a host endpoint for each node listed above. In this example, the contents would look like this.

    1. apiVersion: projectcalico.org/v3
    2. kind: HostEndpoint
    3. metadata:
    4. name: kubeadm-master.eth0
    5. labels:
    6. running-calico: 'true'
    7. spec:
    8. node: kubeadm-master
    9. interfaceName: eth0
    10. expectedIPs:
    11. - 10.100.0.15
    12. ---
    13. apiVersion: projectcalico.org/v3
    14. kind: HostEndpoint
    15. metadata:
    16. name: kubeadm-node-0.eth0
    17. labels:
    18. running-calico: 'true'
    19. spec:
    20. node: kubeadm-node-0
    21. interfaceName: eth0
    22. expectedIPs:
    23. - 10.100.0.16

    In this file, replace eth0 with the desired interface name on each node, and populate the expectedIPs section with the IP addresses on that interface.

    Note the use of a label to indicate that this host endpoint is running Calico. The label matches the selector of the network policy created in step 1.

    Then, use calicoctl to apply the host endpoints with the following command. This will prevent all traffic to and from the host endpoints.

    1. calicoctl apply -f host-endpoints.yaml

    Secure Calico Prometheus endpoints - 图2note

    Calico allows some traffic as a failsafe even after applying this policy. This can be adjusted using the failsafeInboundHostPorts and failsafeOutboundHostPorts options on the FelixConfiguration resource.

  3. Create a network policy that allows access to the calico/node Prometheus metrics port.

    Now let’s create a network policy that allows access to the Prometheus metrics port such that only endpoints with the label calico-prometheus-access: true can access the metrics.

    To do this, create a file named calico-prometheus-policy.yaml with the following contents.

    1. apiVersion: projectcalico.org/v3
    2. kind: GlobalNetworkPolicy
    3. metadata:
    4. name: restrict-calico-node-prometheus
    5. spec:
    6. # Select all Calico nodes.
    7. selector: running-calico == "true"
    8. order: 500
    9. types:
    10. - Ingress
    11. ingress:
    12. # Allow traffic from selected sources to the Prometheus port.
    13. - action: Allow
    14. protocol: TCP
    15. source:
    16. selector: calico-prometheus-access == "true"
    17. destination:
    18. ports:
    19. - 9091

    This policy selects all endpoints that have the label running-calico: true, and enforces a single ingress allow rule. The ingress rule allows traffic to port 9091 from any source with the label calico-prometheus-access: true, meaning all Calico workload endpoints, host endpoints, and global network sets that have the label will be allowed access.

    Then, use calicoctl to apply this policy.

    1. calicoctl apply -f calico-prometheus-policy.yaml
  4. Apply labels to any endpoints that should have access to the metrics.

    At this point, only endpoints that have the label calico-prometheus-access: true can reach Calico’s Prometheus metrics endpoints on each node. To grant access, simply add this label to the desired endpoints.

    For example, to allow access to a Kubernetes pod you can run the following command.

    1. kubectl label pod my-prometheus-pod calico-prometheus-access=true

    If you would like to grant access to a specific IP address in your network, you can create a global network set using calicoctl.

    For example, creating the following network set would grant access to a host with IP 172.15.0.101.

    1. apiVersion: projectcalico.org/v3
    2. kind: GlobalNetworkSet
    3. metadata:
    4. name: calico-prometheus-set
    5. labels:
    6. calico-prometheus-access: 'true'
    7. spec:
    8. nets:
    9. - 172.15.0.101/32

Additional steps for Typha deployments

If your Calico installation uses the Kubernetes API datastore and has greater than 50 nodes, it is likely that you have installed Typha. This section shows how to use an additional network policy to secure the Typha Prometheus endpoints.

After following the steps above, create a file named typha-prometheus-policy.yaml with the following contents.

  1. apiVersion: projectcalico.org/v3
  2. kind: GlobalNetworkPolicy
  3. metadata:
  4. name: restrict-typha-prometheus
  5. spec:
  6. # Select all Calico nodes.
  7. selector: running-calico == "true"
  8. order: 500
  9. types:
  10. - Ingress
  11. ingress:
  12. - action: Allow
  13. protocol: TCP
  14. source:
  15. selector: calico-prometheus-access == "true"
  16. destination:
  17. ports:
  18. - 9093

This policy selects all endpoints that have the label running-calico: true, and enforces a single ingress allow rule. The ingress rule allows traffic to port 9093 from any source with the label calico-prometheus-access: true, meaning all Calico workload endpoints, host endpoints, and global network sets that have the label will be allowed access.

Then, use calicoctl to apply this policy.

  1. calicoctl apply -f typha-prometheus-policy.yaml

Example for kube-controllers

If your Calico installation exposes metrics from kube-controllers, you can limit access to those metrics with the following network policy.

Create a file named kube-controllers-prometheus-policy.yaml with the following contents.

  1. apiVersion: projectcalico.org/v3
  2. kind: NetworkPolicy
  3. metadata:
  4. name: restrict-kube-controllers-prometheus
  5. namespace: calico-system
  6. spec:
  7. selector: k8s-app == "calico-kube-controllers"
  8. order: 500
  9. types:
  10. - Ingress
  11. ingress:
  12. - action: Allow
  13. protocol: TCP
  14. source:
  15. selector: calico-prometheus-access == "true"
  16. destination:
  17. ports:
  18. - 9094

Then, use calicoctl to apply this policy.

  1. calicoctl apply -f kube-controllers-prometheus-policy.yaml