Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crackelf

Member
Apr 11, 2021
74
6
8
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
I updated the fw through the Mellanox MLNX_EN software on their website, but still nothing. It was one fw version out of date, so I didn't expect it to fix much. Another user is saying the fiber I have & transceivers won't work with the card:
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
Which was my original plan:
Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay)
but was convinced otherwise:
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
 
Last edited:

crackelf

Member
Apr 11, 2021
74
6
8
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
My advice to the thread: stay very far away from Mellanox if you value your time.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Another user is saying the fiber I have & transceivers won't work with the card
but was convinced otherwise
He's wrong like most people on the internet, they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon


There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.
The card was plug and play as described, you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor. Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread. Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers. Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue. If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
 

sko

Active Member
Jun 11, 2021
227
121
43
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
 

crackelf

Member
Apr 11, 2021
74
6
8
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
I'm happy they work for you. They don't work for me. It's almost as if life is subjective.
 

crackelf

Member
Apr 11, 2021
74
6
8
He's wrong like most people on the internet
So you're infallible? To me everyone here is just text on a screen. It's all equal in my eyes until I see something functional.
they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon
Happy to be proven otherwise, but I've tried 3 different fiber cables and 20+ different KAIAM transceivers. I'm getting some DAC's and some OS2 soon to test.
The card was plug and play as described
Are they though? I literally plugged two random ones out of the 12 in front of me fresh out of the box and they connect to each other at 1G.
you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor.
With or without SR-IOV enabled they still don't connect... Stock from the box they're 1G.
Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread.
First time with Mellanox. 100% of the Mellanox documentation I found was either incorrect, out of date, or for the wrong card entirely. Their driver page for this card literally links out incorrectly to an entirely different ConnectX. I had to HTML crawl through their directories. Can you imagine if Intel Ark did that?
Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers.
Exactly why I don't go anywhere near Qlogic or broadcom.
Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue.
So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
I'll leave that to the pros. I'm not saying my strife is greater than anybody else's.
 

crackelf

Member
Apr 11, 2021
74
6
8
Where did you buy the nics? From mellanox/nvidia directly?
1 for 1 these exact links
Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex
KAIAM QSFP+40G-LR4 lite
Mellanox MCX4131A-GCAT_C05
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues? Like I said previously, I'm browsing this forum over that exact combination. I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows. but as you've noticed, you're the only one in this thread having these issues. Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something
 
  • Like
Reactions: itronin

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
yeah the ebay link you posted for the NICs states "These cards are equivalent to Mellanox part number MCX4131A-GCAT and have had the PSID changed to allow use of Mellanox OEM firmware." so who knows what the cards actually were vendor and rev wise before the crossflash
 

crackelf

Member
Apr 11, 2021
74
6
8
I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues?
Let's read together:
So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.
The contradiction is that 100's have this working yet nobody can figure out what's wrong. Where did I say that it doesn't work for you? I can't see that.
I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue.
Just so I know moving forward: you recommend things to others that you've never troubleshot before? If it plugs in and works then you've never seen the inevitable litany of issues that exist in all things because literally nothing in observable science is 100%.
but as you've noticed, you're the only one in this thread having these issues.
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
you're the only one in this thread having these issues.
Do you truly think these are 100% infallible devices? Maybe you telling everyone they should just work makes them second guess the issues they have and you just haven't heard from them.

Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something
Plenty of those around. What do you want? Tried Dell PowerEdge R710 R720 R620 R630 R640, Optiplex boxes, the whole Dell product offering sheet. I'll head to the back of the row and boot up the HP servers from 2008 if that helps.
 

crackelf

Member
Apr 11, 2021
74
6
8
on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..
Windows, FreeBSD, Debian. Link me an ISO I'll install it.
Because there's nothing to figure out if it just works for everyone...
Oh except for the $2k worth of 40G equipment I'm staring at that doesn't work, but I guess that's not *your* problem though so it doesn't matter, does it? Really lovely forum we have here guys.
 
  • Sad
Reactions: AndroidCat

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
Since we're reading together (I love group activities), you'll note the part where he's talking about mellanox switches, and infiniband - you have neither. You left out the part where even he himself said:

In cards in Ethernet everything works fine
I'll help you out: you're using a card, and ethernet. We have a match!

because literally nothing in observable science is 100%.
Other than the chances of you no longer getting any help in this thread after you've blamed everyone for your purchase of unknown vendor origin cards lmao
 

crackelf

Member
Apr 11, 2021
74
6
8
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
That's a nice chip on your shoulder that you spend your spare time debugging NICs. I prefer spending time with loved ones, but to each their own.