Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crackelf

Member
Apr 11, 2021
74
6
8
Where did you buy the nics? From mellanox/nvidia directly?
1 for 1 these exact links
Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex
KAIAM QSFP+40G-LR4 lite
Mellanox MCX4131A-GCAT_C05
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,911
3,437
113
34
fohdeesha.com
I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues? Like I said previously, I'm browsing this forum over that exact combination. I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows. but as you've noticed, you're the only one in this thread having these issues. Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something
 
  • Like
Reactions: itronin

fohdeesha

Kaini Industries
Nov 20, 2016
2,911
3,437
113
34
fohdeesha.com
yeah the ebay link you posted for the NICs states "These cards are equivalent to Mellanox part number MCX4131A-GCAT and have had the PSID changed to allow use of Mellanox OEM firmware." so who knows what the cards actually were vendor and rev wise before the crossflash
 

crackelf

Member
Apr 11, 2021
74
6
8
I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues?
Let's read together:
So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.
The contradiction is that 100's have this working yet nobody can figure out what's wrong. Where did I say that it doesn't work for you? I can't see that.
I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue.
Just so I know moving forward: you recommend things to others that you've never troubleshot before? If it plugs in and works then you've never seen the inevitable litany of issues that exist in all things because literally nothing in observable science is 100%.
but as you've noticed, you're the only one in this thread having these issues.
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
you're the only one in this thread having these issues.
Do you truly think these are 100% infallible devices? Maybe you telling everyone they should just work makes them second guess the issues they have and you just haven't heard from them.

Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something
Plenty of those around. What do you want? Tried Dell PowerEdge R710 R720 R620 R630 R640, Optiplex boxes, the whole Dell product offering sheet. I'll head to the back of the row and boot up the HP servers from 2008 if that helps.
 

crackelf

Member
Apr 11, 2021
74
6
8
on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..
Windows, FreeBSD, Debian. Link me an ISO I'll install it.
Because there's nothing to figure out if it just works for everyone...
Oh except for the $2k worth of 40G equipment I'm staring at that doesn't work, but I guess that's not *your* problem though so it doesn't matter, does it? Really lovely forum we have here guys.
 
  • Sad
Reactions: AndroidCat

fohdeesha

Kaini Industries
Nov 20, 2016
2,911
3,437
113
34
fohdeesha.com
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
Since we're reading together (I love group activities), you'll note the part where he's talking about mellanox switches, and infiniband - you have neither. You left out the part where even he himself said:

In cards in Ethernet everything works fine
I'll help you out: you're using a card, and ethernet. We have a match!

because literally nothing in observable science is 100%.
Other than the chances of you no longer getting any help in this thread after you've blamed everyone for your purchase of unknown vendor origin cards lmao
 

crackelf

Member
Apr 11, 2021
74
6
8
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
That's a nice chip on your shoulder that you spend your spare time debugging NICs. I prefer spending time with loved ones, but to each their own.
 

crackelf

Member
Apr 11, 2021
74
6
8
Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?
Sorry hall monitor, didn't know I needed my hall pass around here. Will update my other thread and dip. Enjoy STH!
 
Last edited:
  • Sad
Reactions: AndroidCat

i386

Well-Known Member
Mar 18, 2016
4,588
1,737
113
35
Germany
take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.
 

i386

Well-Known Member
Mar 18, 2016
4,588
1,737
113
35
Germany
I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"
 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)
I agree. I was fully expecting HP flashed NICs when I bought these and got these instead.
Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"
Thank you very very much for the insight and references. I would never have known this. I appreciate your help. I've flopped between it being the cards, the fiber, the transceivers and the switch about 100 times, but something is definitely up here that doesn't seem to be software related. I have some ConnectX-3's coming in along with a few other cables / transcievers and will update my post over here if I ever get to the bottom of this.
 
  • Like
Reactions: kpfleming

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,803
1,194
113
BTW, I have just ordered a bunch of transceivers from the USA, including the recommended XQX2502 because I really want to see how they behave.