Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kpfleming]

Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?

 

[crackelf]

Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?

Sorry hall monitor, didn't know I needed my hall pass around here. Will update my other thread and dip. Enjoy STH!

 

[i386]

take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.

 

[crackelf]

take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.

http://imgur.com/ILu9nwK

http://imgur.com/ekRSs3R

Look legit? I don't know how to spot fakes.

 

[i386]

I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"

 

[crackelf]

I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

I agree. I was fully expecting HP flashed NICs when I bought these and got these instead.

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"

Thank you very very much for the insight and references. I would never have known this. I appreciate your help. I've flopped between it being the cards, the fiber, the transceivers and the switch about 100 times, but something is definitely up here that doesn't seem to be software related. I have some ConnectX-3's coming in along with a few other cables / transcievers and will update my post over here if I ever get to the bottom of this.

 

[NablaSquaredG]

BTW, I have just ordered a bunch of transceivers from the USA, including the recommended XQX2502 because I really want to see how they behave.

 

[itronin]

after reading through all of "that" - I find myself using the ignore member function for the first time.

 

[crackelf]

after reading through all of "that" - I find myself using the ignore member function for the first time.

XDDDDD LAWL ROFLCOPTER don't worry I'll let you get back to your beloved men's support forum.
Mod edited for crimes against vision.

 

[RobstarUSA]

Just had my first real weirdness with my 6610. I had a unifi access point go offline. Weird. I power cycled the POE port on my 6610, and while the port has 2 lights on, the cli says the port is down. I try the port next to it (19)...same issue. I grab a new spare out of the box, same issue. I test the RJ45 cable with a tester (all pairs good). I replace the cable anyway, same issue.

I reload the POE stack member these are attached into and everything is perfect again. ARGHH

 

[fohdeesha]

Just had my first real weirdness with my 6610. I had a unifi access point go offline. Weird. I power cycled the POE port on my 6610, and while the port has 2 lights on, the cli says the port is down. I try the port next to it (19)...same issue. I grab a new spare out of the box, same issue. I test the RJ45 cable with a tester (all pairs good). I replace the cable anyway, same issue.

I reload the POE stack member these are attached into and everything is perfect again. ARGHH

the spare you grabbed and tested, was it a spare AP, or a space ICX? if you had the problem even on a separate newly booted ICX, that def sounds like the unifi device was having issues. if it was a spare AP, and the same icx with the prob was still having the issue on the different AP, then yeah it sounds like a strange icx bug. what OS and PoE fw version are you on?

 

[RobstarUSA]

the spare you grabbed and tested, was it a spare AP, or a space ICX? if you had the problem even on a separate newly booted ICX, that def sounds like the unifi device was having issues. if it was a spare AP, and the same icx with the prob was still having the issue on the different AP, then yeah it sounds like a strange icx bug. what OS and PoE fw version are you on?

Sorry for the confusion. I tried spare ap, cable & port in the same stack member. I have a 2 stack setup using the 40G QSFP connectors on back. I don't have another ICX6610 that has POE, so as a last resort I kicked just the POE stackmember which has my access points (only) in it.

I did everything according to the guide you setup (much thanks btw!).

SSH@BCORE#sh ver | i Version
SW: Version 08.0.30uT7f3
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)

Not sure offhand how to get the poe firmware version, but I remember upgrading it as per your guide.

I kinda wish I had waited for a deal, as I found the 48 port POE model with 2 PS for about $150 shipped on eBay which I thought was superb.
For my first one, non-poe, 48 port, It was $90 but only with one power supply.

To be fair I use Cisco all day at my day job and have had, at one time, 3 Cisco TAC Teams working for 24 hours straight on a multicast over GRE over ipsec issue that was not solvable by them. I found a workaround as I was on a tight time schedule & we eventually ended up removing that equipment as it was older (although still supported). I"m kinda famous @ work for finding issues There is an HP G6 or G7 BIOS out there that has a bios memory setting bug I found. HP Later released an updated bios and the only fix was the bug I found. So maybe it's just me & not the equpment

 

[crackelf]

@fohdeesha The Infallible, confirmed fallible - ConnectX-4 Lx DO NOT work with KAIAM modules:

Another user is saying the fiber I have & transceivers won't work with the card...but I was convinced otherwise:

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core

show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209

Settings for enp1s0np0:
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no

Proof that Mellanox programmed transceivers work:

show media ethernet 1/4/1
Port   1/4/1: Type  : 40GBASE-Passive Copper  2m (QSFP+)
         Vendor: Mellanox             Version: A3
         Part# : MC2207130-002        Serial#: MT1433VS02614

Settings for enp1s0np0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: yes

Proof that KAIAM transceivers work properly with ConnectX-3 cards:

02:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
    Kernel driver in use: mlx4_core
    Kernel modules: mlx4_core

show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209

Settings for enp2s0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes

but as you've noticed, you're the only one in this thread having these issues. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows.

I haven't noticed that. In fact two hours ago:

I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"

He's wrong like most people on the internet

So you're infallible? To me everyone here is just text on a screen. It's all equal in my eyes until I see something functional.

http://imgur.com/mnXqk9t

 

[dswartz]

after reading through all of "that" - I find myself using the ignore member function for the first time.

same here...

 

[fohdeesha]

same here...

yeah I think that last one might be the cringiest thing I've seen on here in years lmao

 

[Zackey_TNT]

yeah I think that last one might be the cringiest thing I've seen on here in years lmao

Could not agree more.... best part is, its showing on google results. I only just recently got here and this is what I found in relation to Brocade + Mellonox
That guy is crinnnnnnngggggeeee

 

[klui]

You got a nice new avatar for it!

 

[crackelf]

yeah I think that last one might be the cringiest thing I've seen on here in years lmao

lol you guys should untuck your shirts every once in a while.

Still don't want to admit you were wrong though huh? It's not so hard, big man.

Could not agree more.... best part is, its showing on google results. I only just recently got here and this is what I found in relation to Brocade + Mellonox
That guy is crinnnnnnngggggeeee

This is the top SEO because there were no resources on this before me. The algorithm doesn't lie.

I'm simply out here contributing to the knowledge base through documentation. Keep up the "crinnnnnnngggggeeee" posts though buddy. Big lols from ur internet friends!

Nobody has brought any hard proof to this thread prior to my post. Just empty promises of "it works trust me!". Prior to my posts people would be lead to believe big @fohdeesha was right. I brought the receipts:

@fohdeesha The Infallible, confirmed fallible - ConnectX-4 Lx DO NOT work with KAIAM modules:

You got a nice new avatar for it!

My last artwork was removed for "crimes against vision" so I had to bring my A game I'm happy that it can be appreciated by our one true savior


edit: Don't worry I'm actually leaving this time. I'll miss you guys, and the fun we had here, but will keep people updated on any alternative modules and fiber in my other thread.

 

[frogtech]

Damn boys, ya'll popping off in here.

@fohdeesha

thanks for the setup guide. when I got my switches I tried following ruckus' update documentation which for the most part is probably fine but it seems to leave out some steps in their examples which caused me to somehow get a ruckus wireless bootloader flashed? don't ask how that's possible lol. very thankful that they have the preboot environment that lets you flash over TFTP

 

[DaSnipe]

Just wanted to thank you @fohdeesha for the massive guide, allowed me to purchase one of these ICX6450's on ebay for super cheap. Going to replace another tp-link POE switch and get my main sections of the network on 10gbpe. Got my TrueNAS and workstation running at 10gbpe. I got a FS.com RJ45 80m transceiver running 41m of Cat5e to the brocade and getting 10gbps connections and close to 6gbps on iperf/ 550 MB/s transfers.

I was going to ask a question but I figured it out before I finished this message haha. Now to see if this thing can power my Rb5009g via poe