PFC enabled network to non PFC net

Discussion in 'Networking' started by Rand__, May 12, 2018.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    Related to ESX but in the end its more a network topic I think.

    With ESX 6.7 now officially supporting RDMA/iSer I thought I'd have some fun with that (did not).

    So I upgraded my boxes, set up my switch with PFC, set up the MLX cards up with PFC according to HowTo Run RoCE over L2 Enabled with PFC (ESXi) | Mellanox Interconnect Community and off it went.
    No immediate issues on ESX side and no immediate positive effects either (but did not have a high speed storage set up since I am in the middle of reorganizing).
    But then I decided that 2 node/witness vsan setup sucks and I wanted to move back to 3 host setup.

    I started moving all my VMs off vsan and noticed abysmal speed - low single digit MBs when moving off the vsan drive to either local or remote datastores. Not knowing what the issue was I swallowed the bitter pill and slowly moved (almost) everything off.

    Today the last VMs were moving off after a few days of transfers when I thought I'd investigate some more.
    Long story short, I found that when I deactivated PFC on the switch the network speed jumped UP to the usually expected 100+ MB/s.

    So my network setup looks currently as follows - I have a dvSwitch in ESX with separate vLANs for the various ESX services (vsan, FT, Provisioning etc). All my hosts have MLX CX3 cards whose 2 ports are uplinks to the dvSwitch; those are connected to a SX6012 in Ethernet mode. That switch has PFC set up.

    Now I think that would work fine (did not see any issues initially). Now the interesting part is - the MLX is connected to a Cisco SG350X via SFP+ - o/c the small business switch does not have PFC capability only flow control. Auto neg does not work on SFP+ so I left it disabled (after having some weird STP issues last week).

    That switch is connected via 10G Ethernet to another Cisco Switch which then is connected to my backup ESX box where the same dvSwitch is running (MLX without pfc config o/c)


    Now, after a long prelude to my question - what happens if a PFC enabled Card sends out packages via a PFC enabled switch via a non PFC enabled other switch to a non PFC enabled client?

    I mean I know what happens - nothing good, but how is that supposed to be setup?

    Does anyone have resources/info on mixing PFC with Flow Control or non PFC environments? How does the switch behave if the next switch is not PFC capable?

    Do I need to separate PFC enabled traffic ? That would make having the same VLAN on both sides quite impossible...

    O/c I could just go back to non PFC, but where would be the fun in that ;)




    Edit:
    I'll dump a few links in here I find with further info

    Network Considerations for Global Pause, PFC an... | Mellanox Interconnect Community
    Recommended Network Configuration Examples for ... | Mellanox Interconnect Community

    PFC Background
    Priority Flow Control: Build Reliable Layer 2 Infrastructure
     
    #1
    Last edited: May 13, 2018
  2. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    823
    Likes Received:
    82
    how/what type of pfc have you enabled ?
     
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    #3
  4. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    #4
Similar Threads: enabled network
Forum Title Date
Networking Switch Advice for Mellanox IB QDR networking May 3, 2018
Networking Advice: Myricom 10G-PCIE-8AL-S Network Adapter(s) Apr 16, 2018
Networking Getting rid of old network lab gear - Las Vegas Apr 6, 2018
Networking Highly Redundant, High Client Count, Low Bandwidth Network Mar 29, 2018
Networking Extend Network to multiple Buildings Mar 23, 2018

Share This Page