Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[crackelf]

Where did you buy the nics? From mellanox/nvidia directly?

1 for 1 these exact links

Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex

OS2 9/125 LC-LC Singlemode Fiber Optic Cable Duplex 7m(23ft) - Yellow - FS.com

Buy 7m (23ft) OS2 9/125 LC-LC Singlemode Fiber Patch Cable Duplex 2.0mm PVC(OFNR) at reliable fiber Jumper manufacturer FS.COM. In stock, Custom Service on-line.

www.fs.com

KAIAM QSFP+40G-LR4 lite

XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

Find many great new & used options and get the best deals for XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mellanox MCX4131A-GCAT_C05

Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware | eBay

Find many great new & used options and get the best deals for Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[crackelf]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

If it's so easy surely someone would have taken me up on the offer to pay to resolve this, right? Just quickly pocket the reward! So easy!

He's wrong like most people on the internet

He's gotten me further than anyone else has...

 

[fohdeesha]

I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues? Like I said previously, I'm browsing this forum over that exact combination. I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows. but as you've noticed, you're the only one in this thread having these issues. Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something

 

[i386]

that's an optic cable, not the nic (mellanox connectx-4 lx)

1 for 1 these exact links

Ebay seller mentions that they modified the firmware, maybe something went wrong and you got "broken" nics

 

[fohdeesha]

yeah the ebay link you posted for the NICs states "These cards are equivalent to Mellanox part number MCX4131A-GCAT and have had the PSID changed to allow use of Mellanox OEM firmware." so who knows what the cards actually were vendor and rev wise before the crossflash

 

[sko]

Stock from the box they're 1G.

on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..

So why can none of the more than 100 people figure out what's wrong?

Because there's nothing to figure out if it just works for everyone...

 

[crackelf]

I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues?

Let's read together:

So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.

The contradiction is that 100's have this working yet nobody can figure out what's wrong. Where did I say that it doesn't work for you? I can't see that.

I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue.

Just so I know moving forward: you recommend things to others that you've never troubleshot before? If it plugs in and works then you've never seen the inevitable litany of issues that exist in all things because literally nothing in observable science is 100%.

but as you've noticed, you're the only one in this thread having these issues.

I haven't noticed that. In fact two hours ago:

I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"

you're the only one in this thread having these issues.

Do you truly think these are 100% infallible devices? Maybe you telling everyone they should just work makes them second guess the issues they have and you just haven't heard from them.

Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something

Plenty of those around. What do you want? Tried Dell PowerEdge R710 R720 R620 R630 R640, Optiplex boxes, the whole Dell product offering sheet. I'll head to the back of the row and boot up the HP servers from 2008 if that helps.

 

[crackelf]

on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..

Windows, FreeBSD, Debian. Link me an ISO I'll install it.

Because there's nothing to figure out if it just works for everyone...

Oh except for the $2k worth of 40G equipment I'm staring at that doesn't work, but I guess that's not *your* problem though so it doesn't matter, does it? Really lovely forum we have here guys.

 

[fohdeesha]

I haven't noticed that. In fact two hours ago:

I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Since we're reading together (I love group activities), you'll note the part where he's talking about mellanox switches, and infiniband - you have neither. You left out the part where even he himself said:

In cards in Ethernet everything works fine

I'll help you out: you're using a card, and ethernet. We have a match!

because literally nothing in observable science is 100%.

Other than the chances of you no longer getting any help in this thread after you've blamed everyone for your purchase of unknown vendor origin cards lmao

 

[fohdeesha]

Windows, FreeBSD, Debian. Link me an ISO I'll install it

Downloads

 

[crackelf]

If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)

That's a nice chip on your shoulder that you spend your spare time debugging NICs. I prefer spending time with loved ones, but to each their own.

 

[crackelf]

Since we're reading together

You've literally ducked all my points from earlier, but whatever man. I guess you just want to hide behind memes and call it a day. Have a good life dawg. Hope those NICs keep you warm at night.

 

[crackelf]

Downloads

You're a poser for not posting the real Cryus

 

[kpfleming]

Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?

 

[crackelf]

Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?

Sorry hall monitor, didn't know I needed my hall pass around here. Will update my other thread and dip. Enjoy STH!

 

[i386]

take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.

 

[crackelf]

take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.

http://imgur.com/ILu9nwK

http://imgur.com/ekRSs3R

Look legit? I don't know how to spot fakes.

 

[i386]

I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"

 

[crackelf]

I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

I agree. I was fully expecting HP flashed NICs when I bought these and got these instead.

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"

Thank you very very much for the insight and references. I would never have known this. I appreciate your help. I've flopped between it being the cards, the fiber, the transceivers and the switch about 100 times, but something is definitely up here that doesn't seem to be software related. I have some ConnectX-3's coming in along with a few other cables / transcievers and will update my post over here if I ever get to the bottom of this.

 

[NablaSquaredG]

BTW, I have just ordered a bunch of transceivers from the USA, including the recommended XQX2502 because I really want to see how they behave.