VCL Socket Interface

Attention

The VCL socket interface extension is experimental and is currently under active development.

This socket interface extension provides Envoy with high speed L2-L7 user space networking by integrating with fd.io VPP through VPP’s Comms Library (VCL).

The VCL socket interface is only included in contrib images

Example configuration

  1. bootstrap_extensions:
  2. - name: envoy.extensions.vcl.vcl_socket_interface
  3. typed_config:
  4. "@type": type.googleapis.com/envoy.extensions.vcl.v3alpha.VclSocketInterface
  5. default_socket_interface: "envoy.extensions.vcl.vcl_socket_interface"

How it works

If enabled, the extension attaches through a VCL interface (vcl_interface.h) to VCL, and consequently to the external VPP process, when it is initialized during Envoy bootstrap. This registers a main VCL worker, while subsequent Envoy workers are registered whenever the socket interface extension detects that its code is being executed by a pthread that has not yet been registered with VCL.

Because both libevent and VCL want to handle the async polling and the dispatching of IoHandles, the VCL interface delegates control to libevent by registering with it, for each Envoy worker, the eventfd associated to the VCL worker’s VPP message queue. These shared memory message queues are used by VPP to convey io/ctrl events to VCL and the eventfds are used to signal message queue transitions from empty to non-empty state. This ultimately means that VPP generated events force libevent to hand over control to the VCL interface which, for each Envoy worker, uses an internally maintained epoll fd to poll/pull events from VCL and subsequently dispatch them. To support all of these indirect interactions, the socket interface makes use of custom IoHandle and FileEvent implementations that convert between Envoy and VCL API calls.

Installing and running VPP/VCL

For information on how to build and/or install VPP see the getting started guide here. Assuming the use of DPDK interfaces, a minimal startup.conf file that also configures the host stack would consist of:

  1. unix {
  2. # Run in interactive mode and not as a daemon
  3. nodaemon
  4. interactive
  5. # Cli socket to be used by vppctl
  6. cli-listen /run/vpp/cli.sock
  7. # Group id is an example
  8. gid vpp
  9. }
  10. cpu {
  11. # Avoid using core 0 and run vpp's main thread on core 1
  12. skip-cores 0
  13. main-core 1
  14. # Set logical CPU core(s) where worker threads are running. For performance testing make
  15. # sure the cores are on the same numa as the NIC(s). Use lscpu to determine the numa of
  16. # a cpu and "sh hardware" in vpp cli to determine the numa of a NIC. To configure multiple
  17. # workers lists are also possible, e.g., corelist-workers 2-4,6
  18. corelist-workers 2
  19. }
  20. buffers {
  21. # Default is 16384 (8192 if running unpriviledged)
  22. buffers-per-numa 16384
  23. }
  24. dpdk {
  25. # Notes:
  26. # - Assuming only one NIC is used
  27. # - The PCI address is an example, the actual one should be found using something like dpdk_devbind
  28. # https://github.com/DPDK/dpdk/blob/main/usertools/dpdk-devbind.py
  29. # - Number of rx queues (num-rx-queus) should be number of workers
  30. dev 0000:18:00.0 {
  31. num-tx-desc 256
  32. num-rx-desc 256
  33. num-rx-queues 1
  34. }
  35. }
  36. session {
  37. # Use session layer socket api for VCL attachments
  38. use-app-socket-api
  39. # VPP worker's message queues lengths
  40. event-queue-length 100000
  41. }

Manually start VPP, once a binary is obtained: ./vpp -c startup.conf

VCL can be configured by either adding a configuration file to /etc/vpp/vcl.conf or by pointing the VCL_CONFIG environment variable to a configuration file. A minimal example that can be used for RPS load testing can be found lower:

  1. vcl {
  2. # Max rx/tx session buffers sizes in bytes. Increase for high throughput traffic.
  3. rx-fifo-size 400000
  4. tx-fifo-size 400000
  5. # Size of shared memory segments between VPP and VCL in bytes
  6. segment-size 1000000000
  7. # App has access to global routing table
  8. app-scope-global
  9. # Allow inter-app shared-memory cut-through sessions
  10. app-scope-local
  11. # Pointer to session layer's socket api socket
  12. app-socket-api /var/run/vpp/app_ns_sockets/default
  13. # Message queues use eventfds for notifications
  14. use-mq-eventfd
  15. # VCL worker incoming message queue size
  16. event-queue-size 40000
  17. }