Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. kroy

    kroy New Member

    Joined:
    May 17, 2019
    Messages:
    23
    Likes Received:
    7
    So I’ve finally navigated the Brocade desert and have found the promised land.

    • 6450-48p was the first switch that I ordered. It worked perfectly but I suffered from a bit of a layer 8 issue here. I didn’t do my homework and discovered this switch doesn’t support dynamic IPv6
    • 6650. I ordered this for a bit of a different project. This switch was DOA and immediately RMAd.
    • 7250-24p #1. Since I was looking for dynamic routing on v6, this seemed like the obvious choice. This one arrived with obviously fried ports (lights stuck on), and no working POE. It took me a while to figure out because I had to hack out the cable. RMAd.
    • 7250-24p #2. Despite being “fully tested”, this switch had a bad POE board. To make things worse, it tried to start a fire while I was trying to troubleshoot with the seller. If not for the fire attempt, I might have kept it because POE wasn’t a deal-killer for me. RMAd.
    • 7250-24p #3. The eBay seller had listed this wrong. It was actually a 7250-24G. No 10Gb, no routing, I didn’t even take this one fully out of the box before RMAing.
    • 6610-24pe. And this point I was sick of the 7250 problems, and tried this switch. Despite mentioning POE all over the listing, and “all ports POE fully working”, what was delivered was a 6610-24-PE, not a 6610-24P-PE. I’ll definitely cop a bit to being a bit of a layer 8 issue here. I believed the listing, but I also didn’t know the difference between P-PE and just -PE. Went right back into the box once someone here pointed this out. I was a bit thankful because with one PSU, this bastard was LOUD. RMAd.
    • 7250-24p #4. FINALLY. A fully working switch. I paid a bit more, but this is a super clean switch.
    It’s definitely been a journey, but I really do love these switches. The 7250s are especially nice as they eliminate dumb things like the `dual-mode` ports.

    Thanks to the people here that have helped me along the way, especially through the layer8 problems, and especially to @fohdeesha for all the work he’s done on these things.
     
    #2141
    fohdeesha likes this.
  2. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,389
    Likes Received:
    1,117
    damn! you might have had the worst luck out of anyone in this 100 page thread. props for sticking with it
     
    #2142
  3. Wolfstar

    Wolfstar Member

    Joined:
    Nov 28, 2015
    Messages:
    60
    Likes Received:
    28
    Okay, I gotta ask. Which dumb thing with dual-mode ports? We use dual-mode on the FCXs and ICX6450s at work, haven't had a chance to really dig on the 7450s we're using yet. We use dual-mode as a way of assigning what Cisco calls a native VLAN, which makes tons of sense to my (admittedly network-oriented) brain.
     
    #2143
  4. kroy

    kroy New Member

    Joined:
    May 17, 2019
    Messages:
    23
    Likes Received:
    7
    Just with the way the rest of the VLANs work, it’s clunky and disconnected. I ran into some goofy circular issues trying to change the native VLAN on a port to like 9, then tag in some other ones.

    It’s just weird to have all your VLANs: you tag your trunk ports, untag your access ports, but then you have to go to a different section to have the untagged VLAN for a trunk port.

    In the later Brocade versions that run on the 7250, you just add the port as untagged on the VLAN. It just makes more sense from a configuration standpoint.
     
    #2144
  5. Wolfstar

    Wolfstar Member

    Joined:
    Nov 28, 2015
    Messages:
    60
    Likes Received:
    28
    Yeah, my major complaint about... well, just about EVERYONE that isn't Cisco last I looked (and maybe Juniper, haven't used their stuff), but Brocade/Ruckus in particular is the way they handle VLANs and tagged vs. untagged.

    What actually is nice for anything running 08.0.30 code is that if you set up per-port authentication, you can actually define access port VLANs on the interface - you just use `auth-default-vlan XXX` and as long as the port's authenticated it will work on that VLAN.

    I will admit, dual-mode command on the port after you just did `no tagged ethe 1/1/1` for 15 VLANs is really freaking irritating.
     
    #2145
  6. kroy

    kroy New Member

    Joined:
    May 17, 2019
    Messages:
    23
    Likes Received:
    7
    I definitely hear you there. I’ve been tearing through switches in the last year and have seen much of the good/bad/ugly/etc.

    Cisco pretty much simplifies it down on a per interface basis to:
    • Choose native vlan
    • Choose tagged VLANs to allow/deny
    • Choose trunk vs access
    The fs.com switches are pretty Cisco-like. They have a nice convenience trick where when you allow a VLAN on an interface, it’s automatically added to the VLAN database. Which is nice for a quick 2-4094.

    Arubas go a bit in reverse and make you create profile and then assign that profile to a port. It works well but it’s obnoxious when you have a port that’s a bit of an outlier as you end up creating a profile specifically for that port.

    In comparison to those, Brocade definitely isn’t terrible. They go a bit in reverse like the Arubas, and make you assign ports to VLANs instead of VLANs to ports. That would probably be super obnoxious, but the way Brocade accepts port and vlan ranges is definitely helpful.
     
    #2146
  7. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Apr 9, 2018
    Messages:
    62
    Likes Received:
    4
    Long time reader, first time poster! (in this thread)

    I’m looking at getting 2 x 6610-48Ps and stacking them.

    I may have missed posts about this topic, but I just wanted to check that 2 units can still be stacked using the non-breakout 40Gb ports?

    I’m ideally wanting to use the 8 x 10Gb front ports, 2 x 40Gb breakout ports as 8 x 10Gb, and still use the 2 x 40Gb non-breakout ports to stack together.

    Failing that, can the breakout ports be used for stacking instead?

    Thanks!
     
    #2147
  8. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,389
    Likes Received:
    1,117
    yes they can be stacked like that, that's how the main core stack at our annual lan parties are configured Fohdeesha/reboot-lan
     
    #2148
  9. Roelf Zomerman

    Joined:
    Jan 10, 2019
    Messages:
    34
    Likes Received:
    4
    Sunon MagLev KDE1204PKV3-MS 3 pin 12V 5200RPM 18dB-A

    and swap the two wires.. done..
     
    #2149
    gustav9797 likes this.
  10. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    67
    Anyone ever have just a single port PoE die suddenly? I rearranged some power cables today, taking the switch down, and now the port I had my AP on no longer comes up. Plugged it into a different one and it works. show inline power says 30000 allocated for a few seconds befoing going back to 0, but the AP doesn't power up.
     
    #2150
  11. gustav9797

    gustav9797 New Member

    Joined:
    Dec 30, 2016
    Messages:
    13
    Likes Received:
    5
    Checked the power draw of the ICX6450-24P for fun. Looks like it draws 50W at startup and then goes up to a stable 78W after booting, this is with nothing connected but the power cord. A bit far from 21W, is there anything wrong with it?
     
    #2151
  12. BackupProphet

    BackupProphet Well-Known Member

    Joined:
    Jul 2, 2014
    Messages:
    785
    Likes Received:
    278
    The PoE unit draws a lot more power.
     
    #2152
  13. gustav9797

    gustav9797 New Member

    Joined:
    Dec 30, 2016
    Messages:
    13
    Likes Received:
    5
    Not according to the main post and the specifications here: Ruckus ICX 6450-24P
    :rolleyes:
     
    #2153
  14. infoMatt

    infoMatt Member

    Joined:
    Apr 16, 2019
    Messages:
    43
    Likes Received:
    19
    No, it shouldn't... during normal operating conditions, the difference should be negligeable (say 1-2W), other than the load required for the PoE devices themselves, of course.
    The "partial load" consumption found on the datasheet is measured as 5% ethernet traffic, full load of PoE PD ;), that's why it's not much different than the 100% load figure.

    Personally, I don't have (at the moment?) a power meter, but just by feeling, to me the 75W sounds way off the tracks... I mean, it was pretty hot here in Italy in the last few days, and the exhaust of my ICX 6450-24P felt "by hand" barely warmer than the surrounding environment, even after an entire afternoon in a not-so-well-ventilated room with an ambient temperature of 28-29°C (no aircon of course)... Inline power was enabled on all ports, but there were just two access points connected.
     
    #2154
    Last edited: Aug 17, 2019
  15. pyroguysf

    pyroguysf New Member

    Joined:
    Aug 17, 2019
    Messages:
    3
    Likes Received:
    0
    I'm looking to upgrade my wireless router + unmanaged switch, and the 6450 seems perfect for future expansion. There's a lot of great resources here, but I had a few questions.
    1. What would I need to set up the 10gb connections to my FreeNAS box and Windows workstation for faster transfers? I was looking at the Intel X520-DA1, but would the Mellanox cards mentioned here work in FreeNAS? It looks like I'd need some transceivers in addtion to specific cables, but I'm having trouble grasping the compatibility. A QSFP breakout seems a little cluttered and unnecessary for my usage.
    2. Would I be able to add an unmanaged switch downstream, or would that cause issues/functionality loss?
     
    #2155
  16. infoMatt

    infoMatt Member

    Joined:
    Apr 16, 2019
    Messages:
    43
    Likes Received:
    19
    Yep, both the Intel and the ConnectX would work, pick the one that you can find at the lowest price.
    Other than the NICs you'll need the cabes, and you have basically two choices:
    • SFP+ DAC (direct attach cable), if you need to cover a short distance and they don't need to be routed through the walls/conduits, or
    • SFP+ optics and fiber optic: these switches don't care about the brand of the optics, but if it's not Brocade you wouldn't see the monitoring details (ie. optical power received and temperature, among the others...). If you choose this route you'll have to match fiber types and optics at both ends (single mode vs multimode, for in-house purpose basically consider the shortest range optics, ie. SR for MM and LR for single mode). Don't quote me on that, but from my memory the Intel NICs by default are picky about the optics, and require Intel ones, but I've not tried myself (this is what I've been told at $dayjob, but can't confirm or deny).
    EDIT: QSFP breakouts are used at the switch side for the 40G ports, and there are none on the 6450; if you find a 40G QSFP NIC you'll need a QSFP-to-SFP+ converter (look at the last few pages).

    No problems; watch for any loops in the network, because if you make a loop ad activate the spanning tree, the switch might ERR-DISABLE the downlink port ;).
     
    #2156
    Last edited: Aug 18, 2019
  17. pyroguysf

    pyroguysf New Member

    Joined:
    Aug 17, 2019
    Messages:
    3
    Likes Received:
    0
    Thanks Matt.
    So you don't need optics with a DAC? I guess the direct attach part makes sense now. When I was looking at FS, their DACs had a different option for each manufacturer - does that matter if optics aren't involved, or is it just more of a marketing thing? Should I just get one that matches the NIC, or would I need a dual compatible one?

    Similarly, I read about Intel NICs being optics locked, so if I went that route, I'd want an Intel transceiver at each end with an LC-LC OM1 cable? If I went Mellanox, would the Brocade transceiver work on that end for the extra details?

    That makes much more sense, I don't know how I missed that. I swear I was reading it as people using breakouts on the NIC side which made no sense.

    Edit: I've read conflicting reports - if I connect my NAS and workstation to the SFP+, will I be able to transfer at 10Gb?
     
    #2157
    Last edited: Aug 18, 2019
  18. French Chamallow

    French Chamallow New Member

    Joined:
    Mar 15, 2019
    Messages:
    17
    Likes Received:
    3
    can other users with a powermeter confirm that the ICX6450-24P consumes so much watt in idle ?
     
    #2158
  19. gustav9797

    gustav9797 New Member

    Joined:
    Dec 30, 2016
    Messages:
    13
    Likes Received:
    5
    I have a feeling that I must verify this myself, this was measured with a cheap meter, it feels a bit far off.
     
    #2159
    French Chamallow likes this.
  20. klui

    klui Member

    Joined:
    Feb 3, 2019
    Messages:
    37
    Likes Received:
    16
    Check your meter's other measurements. I have a brand new ICX7150-C12P and my Kill-a-Watt measures 36W with only the console cable connected.

    This Kill-a-Watt is used intermittently, but I have shut off the circuit while it was plugged in before. Its LCD is currently slow to update and you can see the LCD segments' refresh rate is slow even when values aren't changing. The voltage reading is on the outlet with nothing plugged in is 180V and that isn't right. It displays that voltage for every other single-phase circuit in my house. A CatII multimeter shows normal voltage at 120V. So the measurement for wattage will be off due to voltage, at least. If I convert using 120 instead of 180, I get roughly 24W or so. Shutting off PoE on all ports don't change the wattage value from the Kill-a-Watt.
     
    #2160
Similar Threads: Brocade Series
Forum Title Date
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019

Share This Page