PFC enabled network to non PFC net

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Related to ESX but in the end its more a network topic I think.

With ESX 6.7 now officially supporting RDMA/iSer I thought I'd have some fun with that (did not).

So I upgraded my boxes, set up my switch with PFC, set up the MLX cards up with PFC according to HowTo Run RoCE over L2 Enabled with PFC (ESXi) | Mellanox Interconnect Community and off it went.
No immediate issues on ESX side and no immediate positive effects either (but did not have a high speed storage set up since I am in the middle of reorganizing).
But then I decided that 2 node/witness vsan setup sucks and I wanted to move back to 3 host setup.

I started moving all my VMs off vsan and noticed abysmal speed - low single digit MBs when moving off the vsan drive to either local or remote datastores. Not knowing what the issue was I swallowed the bitter pill and slowly moved (almost) everything off.

Today the last VMs were moving off after a few days of transfers when I thought I'd investigate some more.
Long story short, I found that when I deactivated PFC on the switch the network speed jumped UP to the usually expected 100+ MB/s.

So my network setup looks currently as follows - I have a dvSwitch in ESX with separate vLANs for the various ESX services (vsan, FT, Provisioning etc). All my hosts have MLX CX3 cards whose 2 ports are uplinks to the dvSwitch; those are connected to a SX6012 in Ethernet mode. That switch has PFC set up.

Now I think that would work fine (did not see any issues initially). Now the interesting part is - the MLX is connected to a Cisco SG350X via SFP+ - o/c the small business switch does not have PFC capability only flow control. Auto neg does not work on SFP+ so I left it disabled (after having some weird STP issues last week).

That switch is connected via 10G Ethernet to another Cisco Switch which then is connected to my backup ESX box where the same dvSwitch is running (MLX without pfc config o/c)


Now, after a long prelude to my question - what happens if a PFC enabled Card sends out packages via a PFC enabled switch via a non PFC enabled other switch to a non PFC enabled client?

I mean I know what happens - nothing good, but how is that supposed to be setup?

Does anyone have resources/info on mixing PFC with Flow Control or non PFC environments? How does the switch behave if the next switch is not PFC capable?

Do I need to separate PFC enabled traffic ? That would make having the same VLAN on both sides quite impossible...

O/c I could just go back to non PFC, but where would be the fun in that ;)




Edit:
I'll dump a few links in here I find with further info

Network Considerations for Global Pause, PFC an... | Mellanox Interconnect Community
Recommended Network Configuration Examples for ... | Mellanox Interconnect Community

PFC Background
Priority Flow Control: Build Reliable Layer 2 Infrastructure
 
Last edited: