Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Snorf

New Member
Nov 12, 2018
24
8
3
BC, Canada
I just have to say that this thread is awesome.

A big thank you to all who have contributed and a huge thank you to fohdeesha who started this all. I had never heard of TFTP or Putty, but with that killer step by step guide and a little google searching and reading I have flashed my first ICX switch. Now I just need to buy a couple more switches, build a pfSense box, dig a conduit out to my shed, by some transceivers and fiber, pull some fiber and more Cat 6, rebuild my server..............OMG what have I done!

Sheesh, but after all of that I bet I will have the best network on my street! :)

Thanks all, I am sure I will be asking for help sooner or later but so far I have been lucky that everything I have needed has been answered on this forum a few times. Hopefully I can contribute something someday.

Snorf
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,095
642
113
This is still bugging me...

I've switched over to a 6610-24 at home (from a 6610-48, didn't need all those ports). btw, this is the same 6610-24 that I got for $50! (with no power supplies or fan trays) :p.

As such, I love the 24 much better. With no ports active, a single Rev B power supply and a single fan tray, it idles at 70w. That is better than I expected.

The problem - Even with the Rev B power supply, and no ports active, with max temps <50 deg C, the switch will cycle the fans up and down every minute or so. I can't seem to make it settle down, no matter what. This switch no longer has my hacky mod of 3x 120mm fans blowing in, it's a regular, normal switch. I've tried adding a second power supply, second fan tray, didn't stop it.

Think a Rev C PSU will help? Or just resign myself to it?
 

nev_neo

Active Member
Jul 31, 2013
157
44
28
This is still bugging me...

I've switched over to a 6610-24 at home (from a 6610-48, didn't need all those ports). btw, this is the same 6610-24 that I got for $50! (with no power supplies or fan trays) :p.

As such, I love the 24 much better. With no ports active, a single Rev B power supply and a single fan tray, it idles at 70w. That is better than I expected.

The problem - Even with the Rev B power supply, and no ports active, with max temps <50 deg C, the switch will cycle the fans up and down every minute or so. I can't seem to make it settle down, no matter what. This switch no longer has my hacky mod of 3x 120mm fans blowing in, it's a regular, normal switch. I've tried adding a second power supply, second fan tray, didn't stop it.

Think a Rev C PSU will help? Or just resign myself to it?
What I have noticed is that the switch is significantly quieter with all fans ants power supplies plugged in and working. I noticed this only after everything was plugged in powered off and then booted back on again.
Previously I had both the fans and only 1 psu plugged in and in use, I was trying to see if running with 1 psu and 1 fan would make it run quiter, it did not. It was loud enough to be heard upstairs in the living room.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
What I have noticed is that the switch is significantly quieter with all fans ants power supplies plugged in and working. I noticed this only after everything was plugged in powered off and then booted back on again.
Previously I had both the fans and only 1 psu plugged in and in use, I was trying to see if running with 1 psu and 1 fan would make it run quiter, it did not. It was loud enough to be heard upstairs in the living room.
Already tried that.
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
Question regarding jumbo frames ...

I have only two vlans ATM, poorly named "WAN" (uplink to ISP + pfSense WAN port) and "LAN" (pfSense LAN port + everything else) / in place simply to allow pfSense to do it's job / and need to figure out how to allow the switch ports connecting ESXi hosts to support jumbo while everything else = NOT. TVs are not loving jumbo frames.

So I've poured through the documentation and don't get (I suck at networking) it as it says it can be set globally or by port. In layer 3 mode, but doing no routing, it doesnt seem as though adjusment by port is possible (ie all jumbo or no jumbo). Globally is easy enough "jumbo" via CLI, but any port in The "LAN' vlan with the router-interface can't be set to "ip mtu 1500".

Can any of you offer a kind solution please?
 

Juggie

Member
Nov 3, 2018
41
9
8
Question regarding jumbo frames ...

I have only two vlans ATM, poorly named "WAN" (uplink to ISP + pfSense WAN port) and "LAN" (pfSense LAN port + everything else) / in place simply to allow pfSense to do it's job / and need to figure out how to allow the switch ports connecting ESXi hosts to support jumbo while everything else = NOT. TVs are not loving jumbo frames.

So I've poured through the documentation and don't get (I suck at networking) it as it says it can be set globally or by port. In layer 3 mode, but doing no routing, it doesnt seem as though adjusment by port is possible (ie all jumbo or no jumbo). Globally is easy enough "jumbo" via CLI, but any port in The "LAN' vlan with the router-interface can't be set to "ip mtu 1500".

Can any of you offer a kind solution please?
Did you need jumbo frames for ESXi for vm storage? Or for the vm's themselves?
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
Did you need jumbo frames for ESXi for vm storage? Or for the vm's themselves?
All of the above I would think from the storage layer to the switch (inclusive of FreeNAS, setting MTU across ESXi, and at the switch, as is the subject here).

Networking is very difficult for me and this is ultimately my goal (or something like it at least). My point being that it is stupidly simple with just two hosts and 2 uplink ports (1 per host @ 10G, other NICs used for pfSense, and then there is a separate dedicated 40G link between FreeNAS VMs just for segregated replication. So in a way, I need jumbo frames for that too (but that is as easy as appending "mtu 900" to the interface options).

 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Question regarding jumbo frames ...

I have only two vlans ATM, poorly named "WAN" (uplink to ISP + pfSense WAN port) and "LAN" (pfSense LAN port + everything else) / in place simply to allow pfSense to do it's job / and need to figure out how to allow the switch ports connecting ESXi hosts to support jumbo while everything else = NOT. TVs are not loving jumbo frames.

So I've poured through the documentation and don't get (I suck at networking) it as it says it can be set globally or by port. In layer 3 mode, but doing no routing, it doesnt seem as though adjusment by port is possible (ie all jumbo or no jumbo). Globally is easy enough "jumbo" via CLI, but any port in The "LAN' vlan with the router-interface can't be set to "ip mtu 1500".

Can any of you offer a kind solution please?
Did you read the documentation?

Changing the MTU on an individual port
By default, the maximum Ethernet MTU sizes are as follows:

  • 1500 bytes - The maximum for Ethernet II encapsulation
  • 1492 bytes - The maximum for SNAP encapsulation
When jumbo mode is enabled, the maximum Ethernet MTU sizes are as follows:

  • For ICX 6610 devices
    • 10,200 bytes - The maximum for Ethernet II encapsulation (Default MTU: 9216)
    • 10,174 bytes - The maximum for SNAP encapsulation (Default MTU: 9216)
  • For ICX 6430, ICX 6430-C12, and ICX 6450 devices
    • 10,178 bytes - The maximum for Ethernet II encapsulation (Default MTU: 9216)
    • 10,174 bytes - The maximum for SNAP encapsulation (Default MTU: 9216)
  • For other devices
    • 10,218 bytes - The maximum for Ethernet II encapsulation (Default MTU: 9216)
    • 10,214 bytes - The maximum for SNAP encapsulation (Default MTU: 9216)
NOTE
If you set the MTU of a port to a value lower than the global MTU and from 576 through 1499, the port fragments the packets. However, if the port MTU is exactly 1500 and this is larger than the global MTU, the port drops the packets. For ICX 7250, ICX 7450, and ICX 7750 devices, the minimum IPv4 and IPv6 MTU values for both physical and virtual interfaces are 1280.
NOTE
You must save the configuration change and then reload the software to enable jumbo support.
To change the MTU for interface 1/1/5 to 1000, enter the following commands.

device(config)# interface ethernet 1/1/5
device(config-if-1/1/5)# ip mtu 1000
device(config-if-1/1/5)# write memory
device(config-if-1/1/5)# end
device# reload


Syntax: [no] ip mtu num

The num variable specifies the MTU. Ethernet II packets can hold IP packets from 576 through 1500 bytes long. If jumbo mode is enabled, Ethernet II packets can hold IP packets up to 10,218 bytes long. Ethernet SNAP packets can hold IP packets from 576 through 1492 bytes long. If jumbo mode is enabled, SNAP packets can hold IP packets up to 10,214 bytes long. The default MTU for Ethernet II packets is 1500. The default MTU for SNAP packets is 1492.
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
Did you read the documentation?
Shamefully, I did several times, over the course of a couple hours trying to figure this out. Personnally, I think it contradicts itself, but won't debate that point. I believe the issue is just that I need a real working "example" (switch) to compare against the documentation. And then maybe I find it makes sense.

I even reset my entire config thinking I set some setting wrong ... "jumbo" = super easy to turn on, but then I can't set mtu back to 1500, conversely without it on, as is the case, now, I can't change a port to jumbo.

Code:
SSH@ICX6450(config)#interface ethernet 1/2/1
SSH@ICX6450(config-if-e10000-1/2/1)#ip mtu 9000
Invalid input -> mtu 9000
Type ? for a list
 

Juggie

Member
Nov 3, 2018
41
9
8
Ultimately, you probably do not want to run your normal lan at jumbo frames. Lots of consumer grade stuff as you've discovered will not play nice.

The solution is to create a storage vlan for esxi and any other devices you may want to have higher MTU fast access to freenas.

I don't have specific instructions as my 9000mtu storage is entirely virtual (both the vms and freenas itself is a vm with hardware passthrough) but it should be something along the lines of creating a VLAN, adding the VLAN to the ports going to ESXI and FreeNAS. Add the port group with 9000 MTU with the proper vlan in ESXi and attach a vmkernel nic to it for storage. Add FreeNas optional network with a VLAN. Finally add the storage in ESXi using the 9000 MTU vlan ips.

There is prob better instructions online if you google but this may get you started.

Edit: I know little about the brocade (still hunting for one) but it likely can break up the higher MTU for a lower MTU'ed port based on what someone just posted. I'd think that's prob less efficient though than just keeping a 9000MTU VLAN for whatever you need to run in Jumbo Frames which is likely, very little.
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
Ultimately, you probably do not want to run your normal lan at jumbo frames. Lots of consumer grade stuff as you've discovered will not play nice.
  • Agreed, there isn't a single device that would benefit.
The solution is to create a storage vlan for esxi and any other devices you may want to have higher MTU fast access to freenas.
  • Understood ... and ultimately part of my final objective ... slow going for me at first.
  • There are only 4 physical ports that need jumbo on them (2 uplinks ... 1 per host @ switch + virtualized 40G direct link via ConnectX-3, so really only 2 physical.
  • And perhaps to your point, nearly all of the needed jumbo frames is virtualized networking, but would like to use it as it of course makes a huge difference when running iperf and I bet that functionally makes vmotion, replication, et al much faster too.
I don't have specific instructions as my 9000mtu storage is entirely virtual (both the vms and freenas itself is a vm with hardware passthrough) but it should be something along the lines of creating a VLAN, adding the VLAN to the ports going to ESXI and FreeNAS. Add the port group with 9000 MTU with the proper vlan in ESXi and attach a vmkernel nic to it for storage. Add FreeNas optional network with a VLAN. Finally add the storage in ESXi using the 9000 MTU vlan ips.
  • I'm good with all of it (and it has been configured, but I had to roll back) ... heck, if you told me to turn jumbo on for everything I could knock it out in just a few moments, the problem was the non-server components that jumbo was of course having a negative impact on, thus the question, how can it be enabled in one segment and not another at the ICX6450 level only. (sorry if I was less than clear there)
  • The manual comments were posted above, I just wasn't able to follow them (your point was understood about you being 100% virtual).
There is prob better instructions online if you google but this may get you started.
  • I've look at good bit trust ... I could have taught my schnauzer a good bit of Mandarin in the same amount of time I spent looking. ;)
 

Juggie

Member
Nov 3, 2018
41
9
8
  • Agreed, there isn't a single device that would benefit.
  • Understood ... and ultimately part of my final objective ... slow going for me at first.
  • There are only 4 physical ports that need jumbo on them (2 uplinks ... 1 per host @ switch + virtualized 40G direct link via ConnectX-3, so really only 2 physical.
  • And perhaps to your point, nearly all of the needed jumbo frames is virtualized networking, but would like to use it as it of course makes a huge difference when running iperf and I bet that functionally makes vmotion, replication, et al much faster too.
  • I'm good with all of it (and it has been configured, but I had to roll back) ... heck, if you told me to turn jumbo on for everything I could knock it out in just a few moments, the problem was the non-server components that jumbo was of course having a negative impact on, thus the question, how can it be enabled in one segment and not another at the ICX6450 level only. (sorry if I was less than clear there)
  • The manual comments were posted above, I just wasn't able to follow them (your point was understood about you being 100% virtual).
  • I've look at good bit trust ... I could have taught my schnauzer a good bit of Mandarin in the same amount of time I spent looking. ;)
I don't know enough about the brocade stuff to tell you if the MTU is set per port only, or can be set per port, per vlan.

Keep in mind though, what that is really controlling is the max MTU that the switch will allow to pass through without fragmentation. Then there is the MTU you configure on the devices that are attached to the switch. If every nic on the lan is set to the normal MTU then that's all your getting no matter what the switch is configured for. Presumably somewhere you can also set the MTU of the management interface and which vlans it appears on, etc.

If it is indeed per port only, you can safely set the port to jumbo frames as long as you configure it properly on the devices on that port. Eg, devices on VLAN 0 (untagged) will be configured with normal MTU, while the storage VLAN which most devices wont see will be configured with 9000. If it can be configured per port, per vlan than you can surely do that as well.

Sorry I can't be of more help on the brocade stuff still waiting to get mine. I would be happy to answer any VMWare ESXi questions though.

Edit: In quickly scanning the docs what you likely want is the defaults. Enable jumbo mode and leave the default Jumbo MTU on all the ports.

Then on your untagged interface, ensure everything, storage, esxi, pfsense, is all configured for normal MTU.

Once this works, then you can layer on your 9000MTU Vlan where the Nic's will be configured to use 9000 MTU. Trying to run multiple MTU's on the same lan will result in trouble which is why its best to keep that on a VLAN in my 2 cents.
 
Last edited:

svtkobra7

Active Member
Jan 2, 2017
362
87
28
I don't know enough about the brocade stuff to tell you if the MTU is set per port only, or can be set per port, per vlan.
  • I bet it is per virtual interface, VLAN, I just haven't figured out how to do it yet.
  • Your reply is much appreciated, nevertheless.
Keep in mind though, what that is really controlling is the max MTU that the switch will allow to pass through without fragmentation. Then there is the MTU you configure on the devices that are attached to the switch. If every nic on the lan is set to the normal MTU then that's all your getting no matter what the switch is configured for. Presumably somewhere you can also set the MTU of the management interface and which vlans it appears on, etc.
  • Right, that much I do get, and correct me if I'm wrong, but the user has some say here as at least with the Brocade there is the option to drop those frames which are larger than the mtu, or fragment them.
If it is indeed per port only, you can safely set the port to jumbo frames as long as you configure it properly on the devices on that port. Eg, devices on VLAN 0 will be configured with normal MTU, while the storage VLAN which most devices wont see will be configured with 9000. If it can be configured per port, per vlan than you can surely do that as well.
  • Maybe we are saying the same thing, or maybe just a long day for me, but per port only would be fine by me (me thinks - eek) as I can ensure my 100M tvs are set to mtu = 1500 and those two ports on the switch @ 9000.
  • Maybe I entirely misunderstand the architecture; however, I was under the assumption that with a dvs, considering all is virtual (including a switch) everything should be set to 9000 ... it is just where the dvs meets the physical world, that may take some tweaking ...
Sorry I can't be of more help on the brocade stuff still waiting to get mine. I would be happy to answer any VMWare ESXi questions though.
 

Juggie

Member
Nov 3, 2018
41
9
8
  • I bet it is per virtual interface, VLAN, I just haven't figured out how to do it yet.
  • Your reply is much appreciated, nevertheless.
  • Right, that much I do get, and correct me if I'm wrong, but the user has some say here as at least with the Brocade there is the option to drop those frames which are larger than the mtu, or fragment them.
  • Maybe we are saying the same thing, or maybe just a long day for me, but per port only would be fine by me (me thinks - eek) as I can ensure my 100M tvs are set to mtu = 1500 and those two ports on the switch @ 9000.
  • Maybe I entirely misunderstand the architecture; however, I was under the assumption that with a dvs, considering all is virtual (including a switch) everything should be set to 9000 ... it is just where the dvs meets the physical world, that may take some tweaking ...

The MTU isn't detected typically, its set by device. Where you likely ran into trouble was enabling jumbo frames, and then enabling 9000MTU on your pfsense interface. Is that accurate was pfsense lan nic set to jumbo frames?

Edit: If you think of your typical $30 8 port unmanaged switch from newegg. These can, these days, support Jumbo Frames. The mere existence of Jumbo frame support meaning they will pass packets up to and above 9000 MTU does not in, itself cause any harm. Nor should enabling JUMBO mode on the brocade.

Where you run into trouble is when some devices on the network are set to jumbo frames, and some are not. This is a mismatch, causes fragmentation and on some devices where the network stack is crappy, or not tested in this situation you'll get problems like your TV had of not being able to deal with fragmentation.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
sorry been crazy busy

for l3 firmware you set it per port using the VE interface, it sets the MTU for that whole vlan and any ports in it. it can only go up to what the global limit is set at, so you have to do a global jumbo first

something like

enable
conf t
jumbo
#itll tell you to restart
write mem
exit
reload

when it comes back, increase mtu on the vlan you want:

enable
conf t
int ve 10
ip mtu 9000
write mem

I think all the other vlans stay at default MTU with global jumbo enabled but I would check by running "sh ip int ve 10 | inc mtu"

if they all got pushed to 9k as well just use the above guide to go and set them back to 1500


jumbo frames are a giant pain in the ass, especially in mixed environments, as you'll find out not a whole lot of IOT devices support PMTUD to figure out the total path MTU and end up not working correctly in a mixed environment. I don't even bother with it on storage networks anymore, in real world usage on modern CPU's I rarely see more than a 5% or 6% performance gain and it's not worth the hassle
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
The MTU isn't detected typically, its set by device. Where you likely ran into trouble was enabling jumbo frames, and then enabling 9000MTU on your pfsense interface. Is that accurate was pfsense lan nic set to jumbo frames?
  • Maybe I erred but I didn't think a MTU change in pfSense was necessary (and due to all of the non-Jumbo devices it serves). LAN is mtu 1500 in below tests, but while it may not impact FN <=> FN something was dropping hella packets (and indeed the switch was "pre-Jumbo" below).
  • As to the "detection" bit ... point taken indeed, but to add some detail ... as to how jail (Plex) networking works in FreeNAS: (a) lets say you have "FreeNAS Interface vmx0" that is acquiring DHCP and is primary, (b) the jail generates a virtualized NIC, via bridge, (c) so if parent = 9000, then jail NIC = 9000.
  • ... so I guess you could say that it does detect, more correctly, emulates MTU (not what you meant with your assertion o/c).
Edit: If you think of your typical $30 8 port unmanaged switch from newegg. These can, these days, support Jumbo Frames. The mere existence of Jumbo frame support meaning they will pass packets up to and above 9000 MTU does not in, itself cause any harm. Nor should enabling JUMBO mode on the brocade.
  • So the Brocade has this option => mtu-exceed { forward | hard-drop }:
  1. [forward] - Fragments and forwards a packet from a port with a larger MTU to a port with a smaller MTU.
  2. [hard-drop] - Resets to default and removes the forward function.
  • I set it to forward, but my assumption is that it is there because fragmentation = bad and dropping eliminates it. Perhaps another error on my part.
  • Agreed, with your point o/c.
Where you run into trouble is when some devices on the network are set to jumbo frames, and some are not. This is a mismatch, causes fragmentation and on some devices where the network stack is crappy, or not tested in this situation you'll get problems like your TV had of not being able to deal with fragmentation.
  • EXACTLY.
  • b/c bencmarks are fun, some follow ... (just performed)
root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -P4
[ ID] Interval Transfer Bitrate Retr
[SUM] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec 6766 sender
[SUM] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec receiver

set FN vmx0 to 9000, shutdown, change mtu in ESXi, retest
root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -w 1M -P4
[SUM] 0.00-10.06 sec 4.03 MBytes 3.36 Mbits/sec 32 sender
[SUM] 0.00-10.06 sec 0.00 Bytes 0.00 bits/sec receiver

root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -w 1M -P4
[SUM] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec 4139 sender
[SUM] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec receiver
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
sorry been crazy busy
  • Please - no apology needed - good to hear from ya!
for l3 firmware you set it per port using the VE interface, it sets the MTU for that whole vlan and any ports in it. it can only go up to what the global limit is set at, so you have to do a global jumbo first

something like

enable
conf t
jumbo
#itll tell you to restart
write mem
exit
reload

when it comes back, increase mtu on the vlan you want:

enable
conf t
int ve 10
ip mtu 9000
write mem

I think all the other vlans stay at default MTU with global jumbo enabled but I would check by running "sh ip int ve 10 | inc mtu"

if they all got pushed to 9k as well just use the above guide to go and set them back to 1500
SSH@ICX6450#sh int e 1/2/1
MTU 1500 bytes, encapsulation ethernet

SSH@ICX6450#sh int ve200
Internet address is 10.0.0.10/24, IP MTU 1500 bytes, encapsulation ethernet

SSH@ICX6450(config)#int ve200
SSH@ICX6450(config-vif-200)#ip mtu 9000
Error - Invalid input 9000. Valid range is between 576 and 1500

SSH@ICX6450(config-vif-200)#int e 1/2/1
SSH@ICX6450(config-if-e10000-1/2/1)#ip mtu 9000
Invalid input -> mtu 9000

Code:
jumbo | write mem | reload
SSH@ICX6450#sh int e 1/2/1
MTU 10200 bytes, encapsulation ethernet

SSH@ICX6450#sh int ve200
MTU 10200 bytes, encapsulation ethernet

SSH@ICX6450(config)#int ve200
SSH@ICX6450(config-vif-200)#ip mtu 1500

GUIDE =>"You can change the MTU globally or on individual ports."

OK OK - I just misunderstood the guide. When I think port I'm thinking int e not int ve. Perhaps it could be worded differently or better yet, perhaps I need to learn more.

jumbo frames are a giant pain in the ass, especially in mixed environments, as you'll find out not a whole lot of IOT devices support PMTUD to figure out the total path MTU and end up not working correctly in a mixed environment. I don't even bother with it on storage networks anymore, in real world usage on modern CPU's I rarely see more than a 5% or 6% performance gain and it's not worth the hassle
  • Interesting - it has a freaking massive impact on my perf.
SO - Basically, I need to do one of two things, cut the (a) dingleberry (Plex) off of ZFS (preferred), or (b) set up a "LAN-nojumbo" VLAN chillin next to "LAN-jumbo," which will be a good exercise as I haven't had to deal with multiple int ve's yet. Enable jumbo, pull it back down to 1500 for the "LAN-nojumbo" int ve. Thank god I actually have a "vision" for an end game as my network is stupid in the interim. LOL

(I'm glad my suggestion that it was set at the int ve level / suggested remedy relayed offline = yours, it gives me some hope for the future)

Thanks! :)
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
also if you're getting 1.2gbps from freenas at mtu 1500 something is crazy broken. I'm getting 32.4gbps at mtu 1500 on older 26xx v1's
  • I'm glad you agree now that FN <=> FN perf is subpar (I probably didn't represent it so well initially).
  • I'm sure you know, but that is on 10G, which is does max out @9k, so hopefully not broken beyond repair.
  • Where should I look? There are only a handful of config edits made for my hosts, so I don't doubt that I did screw something up, it is just a matter of where.
  • I get those #s @ 9k ESXi <=> ESXi (over 40G), which makes me wonder something ...
Where is the buffer coming from for iperf? I can't go above 1M @ 4P without an error. Is it swap (which I don't really know what that is anyway), if so, that is my issue as I only have 4GB allocated, since I have 200GB of RAM allocated to the FN instances. And never see swap touched.