Calico node status

A Calico node status resource (CalicoNodeStatus) represents a collection of status information for a node that Calico reports back to the user for use during troubleshooting.

As of today, status of BGP agents, BGP sessions and routes exposed to BGP agents are collected from Linux nodes only. Windows nodes are not supported at this time. Calico node status resource is only valid when Calico BGP networking is in use.

Notes

The updating of CalicoNodeStatus will have a small performance impact on CPU/Memory usage of the node as well as adding load to kubernetes apiserver.

In our testing on a ten node, full mesh cluster, a CalicoNodeStatus resource was created for each node where the update interval was set to ten seconds. On each node, this resulted in an increase in CPU use of 5% of a vCPU and an increase of 4MB of memory. The control plane node recorded an increase in CPU usage of 5% of a vCPU for these 10 nodes.

Calico node status - 图1caution

The implementation of CalicoNodeStatus is designed to handle a small number of nodes (less than 10 is recommended) reporting back status in the same time. If CalicoNodeStatus are created for a large number of nodes, and with short update interval, the kubernetes apiserver may become slower and less responsive. You should create CalicoNodeStatus for the node you are interested in and for debugging purpose only. CalicoNodeStatus resource should be deleted upon the completion of the debugging process.

Sample YAML

To use this function, the user creates a CalicoNodeStatus object for the node, specifying the information to collect and the interval it should be collected at. This example collects information for node “my-kadm-node-0” with an update interval of 10 seconds.

  1. kubectl apply -f -<<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: CalicoNodeStatus
  4. metadata:
  5. name: my-caliconodestatus-1
  6. spec:
  7. classes:
  8. - Agent
  9. - BGP
  10. - Routes
  11. node: my-kadm-node-0
  12. updatePeriodSeconds: 10
  13. EOF

The user then reads back the same resource using the command kubectl get caliconodestatus my-caliconodestatus-1 -o yaml, and the information collected can be found in the status field. For example:

  1. apiVersion: projectcalico.org/v3
  2. kind: CalicoNodeStatus
  3. metadata:
  4. annotations:
  5. kubectl.kubernetes.io/last-applied-configuration: |
  6. {"apiVersion":"crd.projectcalico.org/v1","kind":"CalicoNodeStatus","metadata":{"annotations":{},"name":"my-caliconodestatus-1"},"spec":{"classes":["Agent","BGP","Routes"],"node":"my-kadm-node-0","updatePeriodSeconds":10}}
  7. projectcalico.org/metadata: '{"uid":"6e3b676b-546a-41db-aeda-ad44f25ff73f","generation":2,"creationTimestamp":"2021-10-18T10:30:43Z","managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"projectcalico.org/v3","time":"2021-10-18T10:30:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:node":{},"f:updatePeriodSeconds":{}}}},{"manager":"kubectl-edit","operation":"Update","apiVersion":"crd.projectcalico.org/v1","time":"2021-10-18T12:59:25Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:classes":{}}}}]}'
  8. creationTimestamp: '2021-10-18T10:30:43Z'
  9. generation: 139
  10. name: my-caliconodestatus-1
  11. resourceVersion: '203255'
  12. uid: 6e3b676b-546a-41db-aeda-ad44f25ff73f
  13. spec:
  14. classes:
  15. - Agent
  16. - BGP
  17. - Routes
  18. node: my-kadm-node-0
  19. updatePeriodSeconds: 10
  20. status:
  21. agent:
  22. birdV4:
  23. lastBootTime: '2021-10-18 09:23:19'
  24. lastReconfigurationTime: '2021-10-18 09:23:19'
  25. routerID: 10.128.0.240
  26. state: Ready
  27. version: v0.3.3+birdv1.6.8
  28. birdV6:
  29. lastBootTime: '2021-10-18 09:23:19'
  30. lastReconfigurationTime: '2021-10-18 09:23:19'
  31. routerID: 10.128.0.240
  32. state: Ready
  33. version: v0.3.3+birdv1.6.8
  34. bgp:
  35. numberEstablishedV4: 2
  36. numberEstablishedV6: 0
  37. numberNotEstablishedV4: 1
  38. numberNotEstablishedV6: 0
  39. peersV4:
  40. - peerIP: 10.128.0.242
  41. since: '09:23:22'
  42. state: Established
  43. type: NodeMesh
  44. - peerIP: 10.128.0.200
  45. since: '09:23:21'
  46. state: Established
  47. type: NodeMesh
  48. - peerIP: 10.128.0.241
  49. since: '09:23:22'
  50. state: OpenSent
  51. type: NodeMesh
  52. lastUpdated: '2021-10-18T13:22:03Z'
  53. routes:
  54. routesV4:
  55. - destination: 0.0.0.0/0
  56. gateway: 172.18.0.1
  57. interface: eth0
  58. learnedFrom:
  59. sourceType: Kernel
  60. type: FIB
  61. - destination: 192.168.82.0/26
  62. gateway: 10.128.0.200
  63. interface: eth0
  64. learnedFrom:
  65. peerIP: 10.128.0.200
  66. sourceType: NodeMesh
  67. type: FIB
  68. - destination: 192.168.36.0/26
  69. gateway: 10.128.0.242
  70. interface: eth0
  71. learnedFrom:
  72. peerIP: 10.128.0.242
  73. sourceType: NodeMesh
  74. type: FIB
  75. - destination: 10.128.0.5/16
  76. gateway: N/A
  77. interface: eth0
  78. learnedFrom:
  79. sourceType: Direct
  80. type: FIB

Calico node status definition

Metadata

FieldDescriptionAccepted ValuesSchema
nameThe name of this CalicoNodeStatus resource. Required.Alphanumeric string with optional ., _, or -.string

Spec

FieldDescriptionAccepted ValuesSchema
classesList of classes that declare the types of information to monitor for this node and allows for selective status reporting about certain subsets of information. Required.List of Classeslist of string
nodeThe node name identifies the Calico node instance to collect node status for. Required.valid Kubernetes node namestring
updatePeriodSecondsPeriod in seconds at which CalicoNodeStatus should be updated. Set to 0 to disable CalicoNodeStatus refresh. Maximum update period is one day. Required.0 - 86400int

Status

FieldDescriptionValues
lastUpdatedTimestamp representing the server time when the CalicoNodeStatus object last updated. It is represented in RFC3339 form and is in UTC.
agentList of BGP daemon status on the node.‘birdV4’ holds status of IPv4 bird daemon, ‘birdV6’ holds status of IPv6 bird daemon
bgpList of BGP status on the node.‘peersV4’ holds status of IPv4 BGP peers, ‘peersV6’ holds status of IPv6 BGP peers
routesList of Route status learned by BGP daemon on the node.‘routesV4’ holds status of IPv4 routes , ‘routesV6’ holds status of IPv6 routes

Classes

Accepted ValuesDescription
AgentStatus of BGP daemon agent
BGPStatus of BGP sessions
RoutesStatus of routes exposed to BGP daemon

BGPDaemonStatus

FieldDescriptionSchemaPossible Values
stateState of BGP daemon.stringReady, NotReady
versionVersion of BGP daemonstring
routerIDRouter ID used by BGP daemonstring
lastBootTimeLast boot time of BGP daemonstring
lastReconfigurationTimeLast reconfiguration time of BGP daemonstring

BGPStatus

FieldDescriptionSchema
numberEstablishedV4The total number of IPv4 established bgp sessions.int
numberNotEstablishedV4The total number of IPv4 non-established bgp sessions.int
numberEstablishedV6The total number of IPv6 established bgp sessions.int
numberNotEstablishedV6The total number of IPv6 non-established bgp sessions.int
peersV4IPv4 BGP peers status on the node.List of BGP peer
peersV6IPv6 BGP peers status on the node.List of BGP peer

RouteStatus

FieldDescriptionSchema
routesV4IPv4 routes learned by BGP daemon on the node.List of route
routesV6IPv6 routes learned by BGP daemon on the node.List of route

BGPPeer

FieldDescriptionSchemaPossible Values
peerIPIP address of the peer whose condition we are reporting.ip
typeType indicates whether this peer is configured via the node-to-node mesh, or via en explicit global or per-node BGPPeer object.stringNodeMesh, NodePeer, GlobalPeer
stateBGP session state. Value of an empty string means BGP session state is unknown.stringIdle, Connect, Active, OpenSent, OpenConfirm, Established, Close
sinceTimestamp of last BGP session state change.time

Route

FieldDescriptionSchemaPossible Values
typeType indicates if the route is being used for forwarding or not.stringFIB, RIB
destinationDestination IP of the route.ip
gatewayGateway IP of the route. If the route does not involve a gateway, for example a direct route, the value is N/A.ip or N/A
interfaceNetwork interface for the destination. For a blackhole route, the value is blackhole. For a unreachable route, the value is unreachable.string
learnedFromInformation regarding where this route originated.learned from

LearnedFrom

FieldDescriptionSchemaPossible Values
sourceTypeType of the source where a route is learned from.stringKernel, Static, Direct, NodeMesh, BGPPeer
peerIPIf sourceType is NodeMesh or BGPPeer, IP address of the router that sent us this route.ip