Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[crackelf]

I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed

I updated the fw through the Mellanox MLNX_EN software on their website, but still nothing. It was one fw version out of date, so I didn't expect it to fix much. Another user is saying the fiber I have & transceivers won't work with the card:

LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.

Which was my original plan:

Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay)

but was convinced otherwise:

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

 

[crackelf]

I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed

Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.

My advice to the thread: stay very far away from Mellanox if you value your time.

 

[fohdeesha]

Another user is saying the fiber I have & transceivers won't work with the card
but was convinced otherwise

He's wrong like most people on the internet, they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon

There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

The card was plug and play as described, you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor. Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread. Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers. Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue. If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)

 

[sko]

Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.

We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.

 

[crackelf]

We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.

I'm happy they work for you. They don't work for me. It's almost as if life is subjective.

 

[crackelf]

He's wrong like most people on the internet

So you're infallible? To me everyone here is just text on a screen. It's all equal in my eyes until I see something functional.

they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon

Happy to be proven otherwise, but I've tried 3 different fiber cables and 20+ different KAIAM transceivers. I'm getting some DAC's and some OS2 soon to test.

The card was plug and play as described

Are they though? I literally plugged two random ones out of the 12 in front of me fresh out of the box and they connect to each other at 1G.

you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor.

With or without SR-IOV enabled they still don't connect... Stock from the box they're 1G.

Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread.

First time with Mellanox. 100% of the Mellanox documentation I found was either incorrect, out of date, or for the wrong card entirely. Their driver page for this card literally links out incorrectly to an entirely different ConnectX. I had to HTML crawl through their directories. Can you imagine if Intel Ark did that?

Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers.

Exactly why I don't go anywhere near Qlogic or broadcom.

Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue.

So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.

If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)

I'll leave that to the pros. I'm not saying my strife is greater than anybody else's.

 

[i386]

Where did you buy the nics? From mellanox/nvidia directly?

 

[crackelf]

Where did you buy the nics? From mellanox/nvidia directly?

1 for 1 these exact links

Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex

OS2 9/125 LC-LC Singlemode Fiber Optic Cable Duplex 7m(23ft) - Yellow - FS.com

Buy 7m (23ft) OS2 9/125 LC-LC Singlemode Fiber Patch Cable Duplex 2.0mm PVC(OFNR) at reliable fiber Jumper manufacturer FS.COM. In stock, Custom Service on-line.

www.fs.com

KAIAM QSFP+40G-LR4 lite

XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

Find many great new & used options and get the best deals for XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mellanox MCX4131A-GCAT_C05

Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware | eBay

Find many great new & used options and get the best deals for Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[crackelf]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

If it's so easy surely someone would have taken me up on the offer to pay to resolve this, right? Just quickly pocket the reward! So easy!

He's wrong like most people on the internet

He's gotten me further than anyone else has...

 

[fohdeesha]

I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues? Like I said previously, I'm browsing this forum over that exact combination. I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows. but as you've noticed, you're the only one in this thread having these issues. Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something

 

[i386]

that's an optic cable, not the nic (mellanox connectx-4 lx)

1 for 1 these exact links

Ebay seller mentions that they modified the firmware, maybe something went wrong and you got "broken" nics

 

[fohdeesha]

yeah the ebay link you posted for the NICs states "These cards are equivalent to Mellanox part number MCX4131A-GCAT and have had the PSID changed to allow use of Mellanox OEM firmware." so who knows what the cards actually were vendor and rev wise before the crossflash

 

[sko]

Stock from the box they're 1G.

on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..

So why can none of the more than 100 people figure out what's wrong?

Because there's nothing to figure out if it just works for everyone...

 

[crackelf]

I'm confused at this point, are you saying myself and @jasonwc along with others in this thread are lying when we say we're using these exact optics with mellanox NICs with no issues?

Let's read together:

So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.

The contradiction is that 100's have this working yet nobody can figure out what's wrong. Where did I say that it doesn't work for you? I can't see that.

I wouldn't recommend something for the fun of it with a vague idea that it may work, I recommended it because myself and others here are using the exact same setup without issue.

Just so I know moving forward: you recommend things to others that you've never troubleshot before? If it plugs in and works then you've never seen the inevitable litany of issues that exist in all things because literally nothing in observable science is 100%.

but as you've noticed, you're the only one in this thread having these issues.

I haven't noticed that. In fact two hours ago:

I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"

you're the only one in this thread having these issues.

Do you truly think these are 100% infallible devices? Maybe you telling everyone they should just work makes them second guess the issues they have and you just haven't heard from them.

Hopefully the DACs you ordered work as that will at least narrow it down, although it will be hard for me to recreate here to test because as I've said previously, I put the KAIAM in, and it worked, I haven't been able to coax a failure out of it (especially not one as odd as a 4x10gb SerDes transceiver somehow linking up at 1gbps). At that point I would try your NIC in another server or something

Plenty of those around. What do you want? Tried Dell PowerEdge R710 R720 R620 R630 R640, Optiplex boxes, the whole Dell product offering sheet. I'll head to the back of the row and boot up the HP servers from 2008 if that helps.

 

[crackelf]

on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..

Windows, FreeBSD, Debian. Link me an ISO I'll install it.

Because there's nothing to figure out if it just works for everyone...

Oh except for the $2k worth of 40G equipment I'm staring at that doesn't work, but I guess that's not *your* problem though so it doesn't matter, does it? Really lovely forum we have here guys.

 

[fohdeesha]

I haven't noticed that. In fact two hours ago:

I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Since we're reading together (I love group activities), you'll note the part where he's talking about mellanox switches, and infiniband - you have neither. You left out the part where even he himself said:

In cards in Ethernet everything works fine

I'll help you out: you're using a card, and ethernet. We have a match!

because literally nothing in observable science is 100%.

Other than the chances of you no longer getting any help in this thread after you've blamed everyone for your purchase of unknown vendor origin cards lmao

 

[fohdeesha]

Windows, FreeBSD, Debian. Link me an ISO I'll install it

Downloads

 

[crackelf]

If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)

That's a nice chip on your shoulder that you spend your spare time debugging NICs. I prefer spending time with loved ones, but to each their own.

 

[crackelf]

Since we're reading together

You've literally ducked all my points from earlier, but whatever man. I guess you just want to hide behind memes and call it a day. Have a good life dawg. Hope those NICs keep you warm at night.

 

[crackelf]

Downloads

You're a poser for not posting the real Cryus