The way how Pan-Net is improving the throughput of Compute Nodes

Oct 13, 2020 · 457 words · 3 minute read ovs sr-iov offload packet flow

Pan-Net is using Openstack CMS (Cloud Management System) with OVS (Open vSwitch) as SDN (Software Defined Network) solution. OVS is configured and managed by Neutron (OS project used for networking).

Three main OVS components are:

  • ovs-switchd: the main process of OVS, used to make a decision about packet/frame forwarding, deployed in userspace.
  • kernel module (datapath): forwarding packets/frames, deployed in the kernel.
  • ovsdb-server: DB used to store OVS configuration, deployed in userspace.

OVS supports many features provided by traditional switches:

  • VLAN (802.1Q)
  • LACP and static bonding
  • tunneling protocols (GRE, VxLAN, Geneve, IPSec, etc.)
  • QoS
  • monitoring functions (NetFlow, SPAN, RSPAN, etc.)
  • multicast snooping
  • and so on …

The biggest advantage of OVS is automated control using Openflow.

The OVS solution is CPU intensive, affecting system performance and preventing fully utilizing available bandwidth. To improve switching capability many new technologies are invented (here we will note the most important and well known on the market):

technology speed up(comparing to OVS) deployment standard VM access dependencies comment maturity
OVS DPDK 2-4 Host user space YES virtIO OVS OVS DPDK must use NIC driver very mature
XVIO 20 APP in host user space requires additional vCPUs NO Netronome virtIO OVS offload SR-IOV on NIC OVS kernel vendor lock DPDK based application uses 8 vCPUs not very agile in development
SR-IOV 30 VM and host NIC YES NIC driver NIC vendors lack of vSwitch policies only for static environment very mature
vSwitch offload N/A host NIC not yet N/A another acceleration technology always should come as support (e.g. to SR-IOV) moderate proprietary
vDPA DPDK N/A similar as OVS-DPDK NIC and host should be virtIO pushed by RH and Mellanox CPU use early stage
vDPA kernel 30 similar to SR-IOV NIC control part in host should be virtIO OVS offload pushed by RH and Mellanox preferred vDPA solution early stage

Pan-Net is, as of today, is using Netronome XVIO technology with a strategy to move to Mellanox supported SR-IOV with ASAP. ASAP (Accelerated Switch and Packet Processing) is the way how Mellanox is doing OVS offload.

OVS offload is a way to move forwarding plane from CPU intensive OVS kernel module to hardware resources on the NIC (Network Interface Card). Usually, NICs that supports OVS offloading are called SmartNICs.

Simplified OVS architecture with offloading components is depicted in the figure below.

Figure 1: Simplified OVS offload architecture

Simplified OVS offload architecture

The following figures presents frames flow when SR-IOV with ASAP is used:

Figure 2: Frame flow from VM to NIC

Datapath used when frame is initiated from VM

Figure 3: Frame flow from NIC to VM

Datapath used when frame is initiated from outside of CN

The Pan-Net long term strategy is to use vDPA in the kernel which, from the current perspective, seems to be the best future proof solution. For now, vDPA is not mature enough to be seriously considered for production deployment.

Ivica Čale
Cloud architecture