Decreased performance when adding switch

Discussion in 'Networking' started by Cape, Feb 11, 2019 at 1:01 PM.

  1. Cape

    Cape New Member

    Joined:
    Oct 28, 2015
    Messages:
    29
    Likes Received:
    1
    Hey,
    I'm rebuilding the lab, and one part will be going from ESXi to Proxmox. Before moving my pfSense setup, I wanted to establish a baseline for what performance I get today. Current setup is ETTH -> pfSense on ESXi -> internal network. I have a 1G/1G uplink.
    Doing a speed test (from a client on the wired LAN) on this yielded ~880 Mbps down/930 Mbps up. Decent real-world performance.

    The new setup will have multiple nodes with WAN access, so next step is moving the ETTH to my switch on a VLAN segment and cabling in the ESXi port that previously had the WAN cable there. New speed test. And now I got 450 Mbps down. Wat.

    The switch is a HP 1810G, and checking the data sheet it should be able to do 48 Gbps switching, so it doesn't seem like I should be hitting some cap there.

    Any ideas? While I basically only use the full bandwidth a few times per month or so when downloading something from Steam, I don't really like the idea of dumping half the performance.
     
    #1
  2. itronin

    itronin Member

    Joined:
    Nov 24, 2018
    Messages:
    30
    Likes Received:
    14
    Your performance number looks suspiciously like a switch port either auto-negotiated poorly or was set to half-duplex.
    Since your direct wired connection performance was good, suspect the switch ports first.
    Check the involved switch ports to see their current status and check their configuration to make sure they are ideally auto duplex or you could configure the duplex to FULL.

    if the switch ports look fine then I have occasionally seen a port set to auto but an interface card negotiate to half so you'll have to check the ETTH device's interface status and settings as well as the pfSense ESXI mapped interface.

    itr
     
    #2
  3. Cape

    Cape New Member

    Joined:
    Oct 28, 2015
    Messages:
    29
    Likes Received:
    1
    That is a very good point! Not sure why it didn't cross my mind... However, the switch reports both ports being full duplex and 1G autoneg, so at least it the switch believes we're doing full speed. There isn't really a "ETTH device", I just get an ethernet port by my apartment door :) (Well, there's a big switch somewhere in the basement, but nothing I have access to).
    pfSense also says full duplex.

    EDIT: Doing a new speed test, I now get 530 Mbps/680 Mbps. So while it sounded plausible with the initial numbers, it doesn't seem to be it. Thanks for the idea, though!
     
    #3
  4. itronin

    itronin Member

    Joined:
    Nov 24, 2018
    Messages:
    30
    Likes Received:
    14
    >=pfSense also says full duplex.
    >EDIT: Doing a new speed test, I now get 530 Mbps/680 Mbps. So while it sounded plausible with the >initial numbers, it doesn't seem to be it. Thanks for the idea, though!

    Your VLAN is untagged at both ends or just on the carrier connection?
    Is your VLAN tagged coming into ESXI or untagged?

    re. pfSense says full duplex: is that the interface from pfSense's view or ESXI's view? What does ESXI say about the actual interfaace?

    If you have a small gig "dumb switch" you might try that instead of using a VLAN just to test and see what the perf numbers look like... That will also force a disconnect and the various switch ports to auto negotiate.

    ETTH-> dumb switch <- ESXI phys interface <- pfSense virt interface

    FWIW the times I have seen a carrier switch auto negotiate poorly the carrier was using Bay networks or older Cisco Cat gear, 35xx, 37xx on metro ethernet. I doubt there is a Bay networks switch in your basement...
     
    #4
  5. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,421
    Likes Received:
    342
    Switch isn’t being forced to do some layer 3 type work instead of a basic layer 2 config is it ?
     
    #5
  6. Cape

    Cape New Member

    Joined:
    Oct 28, 2015
    Messages:
    29
    Likes Received:
    1
    It is untagged coming into ESXi.

    Both ESXi and pfsense, sorry for being unclear.

    No idea what is used, but the ISP's CTO moved in here a few years ago (some years after we signed up with them, so no funny business), so I don't think the equipment is "known bad" at least.
     
    #6
  7. Cape

    Cape New Member

    Joined:
    Oct 28, 2015
    Messages:
    29
    Likes Received:
    1
    Nope, no support for it even.
     
    #7
  8. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    972
    Likes Received:
    437
    Switch / router performance numbers are generally calculated "downhill, with a tail wind" - in other words, optimally sized packets. Normally, "optimally" means 9K jumbos. If one side of the switch is connected to your ISP, you don't have control over what they're willing to send / accept. What is the MTU setting on your host (hypervisor and application)?

    I'm not familiar with HP switches (and from what I've heard, they're OEM'd from several suppliers), but on Cisco switches you can monitor the CPU load. Which can be misleading as a lot of the processing happens in hardware - on the ancient 2900XL switches, 80 to 90 percent of the CPU was used to flash the pretty LEDs on the front panel, but since switching happened in hardware, that didn't matter. Some things tend to punt packets to the processor, though, so it can be useful to check, even on the latest switches.
    Since you reported 3 very different sets of numbers, make sure it isn't actually something outside your LAN - try the same configuration at multiple times of the day / night and see how much things change. I run a 40Gbit/sec speedtest.net instance and looking at the reports sorted by client IP address, there's quite a bit of variation, presumably all from their ISP since I have 40GbE links to multiple upstreams and several peering points. Some of the differences may be several customers with different connection speeds behind a carrier grade NAT, but I've seen the differences with providers who I know don't use CGN.
     
    #8
Similar Threads: Decreased performance
Forum Title Date
Networking 10gb performance issues Jan 18, 2019
Networking Paltry EIPOIB Performance Dec 9, 2018
Networking PFSense VMware High Performance 10Gbe Networking Oct 19, 2018
Networking Soekris net6501 and Openbsd, network performance trouble Jul 24, 2018
Networking QDR Mellanox Performance Test Results with Varying BAR-space Size May 15, 2018

Share This Page