Brocade ICX6450 / ICX6610 / ETC

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. Snorf

    Snorf New Member

    Joined:
    Nov 12, 2018
    Messages:
    3
    Likes Received:
    5
    I just have to say that this thread is awesome.

    A big thank you to all who have contributed and a huge thank you to fohdeesha who started this all. I had never heard of TFTP or Putty, but with that killer step by step guide and a little google searching and reading I have flashed my first ICX switch. Now I just need to buy a couple more switches, build a pfSense box, dig a conduit out to my shed, by some transceivers and fiber, pull some fiber and more Cat 6, rebuild my server..............OMG what have I done!

    Sheesh, but after all of that I bet I will have the best network on my street! :)

    Thanks all, I am sure I will be asking for help sooner or later but so far I have been lucky that everything I have needed has been answered on this forum a few times. Hopefully I can contribute something someday.

    Snorf
     
    #1121
    Last edited: Dec 5, 2018 at 7:21 AM
  2. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    457
    Likes Received:
    159
    This is still bugging me...

    I've switched over to a 6610-24 at home (from a 6610-48, didn't need all those ports). btw, this is the same 6610-24 that I got for $50! (with no power supplies or fan trays) :p.

    As such, I love the 24 much better. With no ports active, a single Rev B power supply and a single fan tray, it idles at 70w. That is better than I expected.

    The problem - Even with the Rev B power supply, and no ports active, with max temps <50 deg C, the switch will cycle the fans up and down every minute or so. I can't seem to make it settle down, no matter what. This switch no longer has my hacky mod of 3x 120mm fans blowing in, it's a regular, normal switch. I've tried adding a second power supply, second fan tray, didn't stop it.

    Think a Rev C PSU will help? Or just resign myself to it?
     
    #1122
  3. nev_neo

    nev_neo Member

    Joined:
    Jul 31, 2013
    Messages:
    109
    Likes Received:
    17
    What I have noticed is that the switch is significantly quieter with all fans ants power supplies plugged in and working. I noticed this only after everything was plugged in powered off and then booted back on again.
    Previously I had both the fans and only 1 psu plugged in and in use, I was trying to see if running with 1 psu and 1 fan would make it run quiter, it did not. It was loud enough to be heard upstairs in the living room.
     
    #1123
  4. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    457
    Likes Received:
    159
    Already tried that.
     
    #1124
  5. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    Question regarding jumbo frames ...

    I have only two vlans ATM, poorly named "WAN" (uplink to ISP + pfSense WAN port) and "LAN" (pfSense LAN port + everything else) / in place simply to allow pfSense to do it's job / and need to figure out how to allow the switch ports connecting ESXi hosts to support jumbo while everything else = NOT. TVs are not loving jumbo frames.

    So I've poured through the documentation and don't get (I suck at networking) it as it says it can be set globally or by port. In layer 3 mode, but doing no routing, it doesnt seem as though adjusment by port is possible (ie all jumbo or no jumbo). Globally is easy enough "jumbo" via CLI, but any port in The "LAN' vlan with the router-interface can't be set to "ip mtu 1500".

    Can any of you offer a kind solution please?
     
    #1125
  6. Juggie

    Juggie New Member

    Joined:
    Nov 3, 2018
    Messages:
    16
    Likes Received:
    4
    Did you need jumbo frames for ESXi for vm storage? Or for the vm's themselves?
     
    #1126
  7. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    All of the above I would think from the storage layer to the switch (inclusive of FreeNAS, setting MTU across ESXi, and at the switch, as is the subject here).

    Networking is very difficult for me and this is ultimately my goal (or something like it at least). My point being that it is stupidly simple with just two hosts and 2 uplink ports (1 per host @ 10G, other NICs used for pfSense, and then there is a separate dedicated 40G link between FreeNAS VMs just for segregated replication. So in a way, I need jumbo frames for that too (but that is as easy as appending "mtu 900" to the interface options).

    [​IMG]
     
    #1127
  8. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    457
    Likes Received:
    159
    Did you read the documentation?

     
    #1128
  9. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    Shamefully, I did several times, over the course of a couple hours trying to figure this out. Personnally, I think it contradicts itself, but won't debate that point. I believe the issue is just that I need a real working "example" (switch) to compare against the documentation. And then maybe I find it makes sense.

    I even reset my entire config thinking I set some setting wrong ... "jumbo" = super easy to turn on, but then I can't set mtu back to 1500, conversely without it on, as is the case, now, I can't change a port to jumbo.

    Code:
    SSH@ICX6450(config)#interface ethernet 1/2/1
    SSH@ICX6450(config-if-e10000-1/2/1)#ip mtu 9000
    Invalid input -> mtu 9000
    Type ? for a list
    
     
    #1129
  10. Juggie

    Juggie New Member

    Joined:
    Nov 3, 2018
    Messages:
    16
    Likes Received:
    4
    Ultimately, you probably do not want to run your normal lan at jumbo frames. Lots of consumer grade stuff as you've discovered will not play nice.

    The solution is to create a storage vlan for esxi and any other devices you may want to have higher MTU fast access to freenas.

    I don't have specific instructions as my 9000mtu storage is entirely virtual (both the vms and freenas itself is a vm with hardware passthrough) but it should be something along the lines of creating a VLAN, adding the VLAN to the ports going to ESXI and FreeNAS. Add the port group with 9000 MTU with the proper vlan in ESXi and attach a vmkernel nic to it for storage. Add FreeNas optional network with a VLAN. Finally add the storage in ESXi using the 9000 MTU vlan ips.

    There is prob better instructions online if you google but this may get you started.

    Edit: I know little about the brocade (still hunting for one) but it likely can break up the higher MTU for a lower MTU'ed port based on what someone just posted. I'd think that's prob less efficient though than just keeping a 9000MTU VLAN for whatever you need to run in Jumbo Frames which is likely, very little.
     
    #1130
  11. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    • Agreed, there isn't a single device that would benefit.
    • Understood ... and ultimately part of my final objective ... slow going for me at first.
    • There are only 4 physical ports that need jumbo on them (2 uplinks ... 1 per host @ switch + virtualized 40G direct link via ConnectX-3, so really only 2 physical.
    • And perhaps to your point, nearly all of the needed jumbo frames is virtualized networking, but would like to use it as it of course makes a huge difference when running iperf and I bet that functionally makes vmotion, replication, et al much faster too.
    • I'm good with all of it (and it has been configured, but I had to roll back) ... heck, if you told me to turn jumbo on for everything I could knock it out in just a few moments, the problem was the non-server components that jumbo was of course having a negative impact on, thus the question, how can it be enabled in one segment and not another at the ICX6450 level only. (sorry if I was less than clear there)
    • The manual comments were posted above, I just wasn't able to follow them (your point was understood about you being 100% virtual).
    • I've look at good bit trust ... I could have taught my schnauzer a good bit of Mandarin in the same amount of time I spent looking. ;)
     
    #1131
  12. Juggie

    Juggie New Member

    Joined:
    Nov 3, 2018
    Messages:
    16
    Likes Received:
    4
    I don't know enough about the brocade stuff to tell you if the MTU is set per port only, or can be set per port, per vlan.

    Keep in mind though, what that is really controlling is the max MTU that the switch will allow to pass through without fragmentation. Then there is the MTU you configure on the devices that are attached to the switch. If every nic on the lan is set to the normal MTU then that's all your getting no matter what the switch is configured for. Presumably somewhere you can also set the MTU of the management interface and which vlans it appears on, etc.

    If it is indeed per port only, you can safely set the port to jumbo frames as long as you configure it properly on the devices on that port. Eg, devices on VLAN 0 (untagged) will be configured with normal MTU, while the storage VLAN which most devices wont see will be configured with 9000. If it can be configured per port, per vlan than you can surely do that as well.

    Sorry I can't be of more help on the brocade stuff still waiting to get mine. I would be happy to answer any VMWare ESXi questions though.

    Edit: In quickly scanning the docs what you likely want is the defaults. Enable jumbo mode and leave the default Jumbo MTU on all the ports.

    Then on your untagged interface, ensure everything, storage, esxi, pfsense, is all configured for normal MTU.

    Once this works, then you can layer on your 9000MTU Vlan where the Nic's will be configured to use 9000 MTU. Trying to run multiple MTU's on the same lan will result in trouble which is why its best to keep that on a VLAN in my 2 cents.
     
    #1132
    Last edited: Dec 6, 2018 at 8:25 PM
  13. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    • I bet it is per virtual interface, VLAN, I just haven't figured out how to do it yet.
    • Your reply is much appreciated, nevertheless.
    • Right, that much I do get, and correct me if I'm wrong, but the user has some say here as at least with the Brocade there is the option to drop those frames which are larger than the mtu, or fragment them.
    • Maybe we are saying the same thing, or maybe just a long day for me, but per port only would be fine by me (me thinks - eek) as I can ensure my 100M tvs are set to mtu = 1500 and those two ports on the switch @ 9000.
    • Maybe I entirely misunderstand the architecture; however, I was under the assumption that with a dvs, considering all is virtual (including a switch) everything should be set to 9000 ... it is just where the dvs meets the physical world, that may take some tweaking ...
     
    #1133
  14. Juggie

    Juggie New Member

    Joined:
    Nov 3, 2018
    Messages:
    16
    Likes Received:
    4
    The MTU isn't detected typically, its set by device. Where you likely ran into trouble was enabling jumbo frames, and then enabling 9000MTU on your pfsense interface. Is that accurate was pfsense lan nic set to jumbo frames?

    Edit: If you think of your typical $30 8 port unmanaged switch from newegg. These can, these days, support Jumbo Frames. The mere existence of Jumbo frame support meaning they will pass packets up to and above 9000 MTU does not in, itself cause any harm. Nor should enabling JUMBO mode on the brocade.

    Where you run into trouble is when some devices on the network are set to jumbo frames, and some are not. This is a mismatch, causes fragmentation and on some devices where the network stack is crappy, or not tested in this situation you'll get problems like your TV had of not being able to deal with fragmentation.
     
    #1134
    Last edited: Dec 6, 2018 at 8:42 PM
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    934
    Likes Received:
    680
    sorry been crazy busy

    for l3 firmware you set it per port using the VE interface, it sets the MTU for that whole vlan and any ports in it. it can only go up to what the global limit is set at, so you have to do a global jumbo first

    something like

    enable
    conf t
    jumbo
    #itll tell you to restart
    write mem
    exit
    reload

    when it comes back, increase mtu on the vlan you want:

    enable
    conf t
    int ve 10
    ip mtu 9000
    write mem

    I think all the other vlans stay at default MTU with global jumbo enabled but I would check by running "sh ip int ve 10 | inc mtu"

    if they all got pushed to 9k as well just use the above guide to go and set them back to 1500


    jumbo frames are a giant pain in the ass, especially in mixed environments, as you'll find out not a whole lot of IOT devices support PMTUD to figure out the total path MTU and end up not working correctly in a mixed environment. I don't even bother with it on storage networks anymore, in real world usage on modern CPU's I rarely see more than a 5% or 6% performance gain and it's not worth the hassle
     
    #1135
    tommybackeast and lambda like this.
  16. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    • Maybe I erred but I didn't think a MTU change in pfSense was necessary (and due to all of the non-Jumbo devices it serves). LAN is mtu 1500 in below tests, but while it may not impact FN <=> FN something was dropping hella packets (and indeed the switch was "pre-Jumbo" below).
    • As to the "detection" bit ... point taken indeed, but to add some detail ... as to how jail (Plex) networking works in FreeNAS: (a) lets say you have "FreeNAS Interface vmx0" that is acquiring DHCP and is primary, (b) the jail generates a virtualized NIC, via bridge, (c) so if parent = 9000, then jail NIC = 9000.
    • ... so I guess you could say that it does detect, more correctly, emulates MTU (not what you meant with your assertion o/c).
    • So the Brocade has this option => mtu-exceed { forward | hard-drop }:
    1. [forward] - Fragments and forwards a packet from a port with a larger MTU to a port with a smaller MTU.
    2. [hard-drop] - Resets to default and removes the forward function.
    • I set it to forward, but my assumption is that it is there because fragmentation = bad and dropping eliminates it. Perhaps another error on my part.
    • Agreed, with your point o/c.
    • EXACTLY.
    • b/c bencmarks are fun, some follow ... (just performed)
    root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -P4
    [ ID] Interval Transfer Bitrate Retr
    [SUM] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec 6766 sender
    [SUM] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec receiver

    set FN vmx0 to 9000, shutdown, change mtu in ESXi, retest
    root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -w 1M -P4
    [SUM] 0.00-10.06 sec 4.03 MBytes 3.36 Mbits/sec 32 sender
    [SUM] 0.00-10.06 sec 0.00 Bytes 0.00 bits/sec receiver

    root@FreeNAS-02[~]# iperf3 -c 10.0.0.51 -w 1M -P4
    [SUM] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec 4139 sender
    [SUM] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec receiver
     
    #1136
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    934
    Likes Received:
    680
    also if you're getting 1.2gbps from freenas at mtu 1500 something is crazy broken. I'm getting 32.4gbps at mtu 1500 on older 26xx v1's
     
    #1137
    lambda likes this.
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    934
    Likes Received:
    680
  19. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    • Please - no apology needed - good to hear from ya!
    SSH@ICX6450#sh int e 1/2/1
    MTU 1500 bytes, encapsulation ethernet

    SSH@ICX6450#sh int ve200
    Internet address is 10.0.0.10/24, IP MTU 1500 bytes, encapsulation ethernet

    SSH@ICX6450(config)#int ve200
    SSH@ICX6450(config-vif-200)#ip mtu 9000
    Error - Invalid input 9000. Valid range is between 576 and 1500

    SSH@ICX6450(config-vif-200)#int e 1/2/1
    SSH@ICX6450(config-if-e10000-1/2/1)#ip mtu 9000
    Invalid input -> mtu 9000

    Code:
    jumbo | write mem | reload
    SSH@ICX6450#sh int e 1/2/1
    MTU 10200 bytes, encapsulation ethernet

    SSH@ICX6450#sh int ve200
    MTU 10200 bytes, encapsulation ethernet

    SSH@ICX6450(config)#int ve200
    SSH@ICX6450(config-vif-200)#ip mtu 1500

    GUIDE =>"You can change the MTU globally or on individual ports."

    OK OK - I just misunderstood the guide. When I think port I'm thinking int e not int ve. Perhaps it could be worded differently or better yet, perhaps I need to learn more.

    • Interesting - it has a freaking massive impact on my perf.
    SO - Basically, I need to do one of two things, cut the (a) dingleberry (Plex) off of ZFS (preferred), or (b) set up a "LAN-nojumbo" VLAN chillin next to "LAN-jumbo," which will be a good exercise as I haven't had to deal with multiple int ve's yet. Enable jumbo, pull it back down to 1500 for the "LAN-nojumbo" int ve. Thank god I actually have a "vision" for an end game as my network is stupid in the interim. LOL

    (I'm glad my suggestion that it was set at the int ve level / suggested remedy relayed offline = yours, it gives me some hope for the future)

    Thanks! :)
     
    #1139
  20. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    188
    Likes Received:
    36
    • I'm glad you agree now that FN <=> FN perf is subpar (I probably didn't represent it so well initially).
    • I'm sure you know, but that is on 10G, which is does max out @9k, so hopefully not broken beyond repair.
    • Where should I look? There are only a handful of config edits made for my hosts, so I don't doubt that I did screw something up, it is just a matter of where.
    • I get those #s @ 9k ESXi <=> ESXi (over 40G), which makes me wonder something ...
    Where is the buffer coming from for iperf? I can't go above 1M @ 4P without an error. Is it swap (which I don't really know what that is anyway), if so, that is my issue as I only have 4GB allocated, since I have 200GB of RAM allocated to the FN instances. And never see swap touched.
     
    #1140
Similar Threads: Brocade ICX6450
Forum Title Date
Networking Brocade ICX6450 - Help with VLAN Config/SFP Licensing Question Oct 3, 2018
Networking Brocade ICX-6450 DHCP problems with certain IoT devices Dec 2, 2018
Networking LB6M (brocade firmware) Trouble Routing Vlans Jul 13, 2018
Networking Brocade ICX-6610 with Bell Canada Fibe FTTH Jul 1, 2018
Networking Having Issues With Brocade Switch - Not Booting May 5, 2018

Share This Page