ICX6610, 40Gb, 100Gb, LC - LC

[Rootless]

Ok. I've seen clearly that spending time and money trying to get 10Gb+ running over a ~20M CAT6 cable run is just $$$ and not nearly as good as just running a fiber.

It looks like I can get 40Gb+ for the same price as 10Gb, using fiber.

My main switch is a Brocade ICX6610-48P. (Thanks to ServeTheHome, I can use the 2x 40Gb ports on the back, along with the 10Gb SPF+ ports on the front).
To do 40Gb from the 6610 means QSFP+ (not a significant price difference between QSFP and QSFP+).

Right now my plan is to run LC - LC OM4 (or OM5) fiber between the 6610 and my PC. The reason for going LC - LC is to enable future upgrades to 100Gb over the run when the cost comes down. Or maybe it is worth it to just put the money into the 100Gb now and get a new switch when that part comes down?

The questions are:

Try for 40Gb using QSFP+?

What NIC/HCA for my PC? (Yes, I know and can calculate the bandwidth limitations of the PCIE bus. This is both for immediate use AND for future use)

What QSFP+ transceiver do I need for the NIC/HCA that supports the LC - LC? OR do I just need to go MP?

Thank you, thank you in advance. I've been going round and round and just can't get the certainty I need to pull the trigger.

 

[Dave Corder]

I use these QSFP transceivers in my ICX6610 on one end and Mellanox ConnectX-3 cards on the other (one TrueNAS, one Proxmox): XQX2502 KAIAM QSFP+40G-LR4 and then plain old SMF LC-LC cables. They work great for me.

 

[Rootless]

Thank you. The prices on those transceivers is great. I was hoping to only do the fiber pull once. Is there an LC-LC SMF that will support both 40G and 100G (obviously not at the same time)? Thank you.

 

[Rttg]

If you get OS2 SMF cables, you should be fine.

FS.com: Comparison Between OS1 and OS2 SMF Cables

edit: I’ll also add, I have no use for these (nor a way of testing them, my kit maxes out at 40G), but there are some surprisingly cheap 100G QSFP28 optics out there: SPTSBP2CLCKS INTEL 100G-CWDM4 1271-1331NM OPTICAL TRANSCEIVER MODULE NEW | eBay

 

[Rootless]

If you get OS2 SMF cables, you should be fine.

FS.com: Comparison Between OS1 and OS2 SMF Cables

edit: I’ll also add, I have no use for these (nor a way of testing them, my kit maxes out at 40G), but there are some surprisingly cheap 100G QSFP28 optics out there: SPTSBP2CLCKS INTEL 100G-CWDM4 1271-1331NM OPTICAL TRANSCEIVER MODULE NEW | eBay

Thank you for the OS2.

Right! I wish there was a way for me to use these QSFP28.

 

[Rttg]

Right! I wish there was a way for me to use these QSFP28.

That’s as simple as just buying a 100g NIC. ConnectX-4 prices don’t look horrible, but then again, they’re largely overkill IMO

 

[Rootless]

The ConnectX-4 @ $200+ vs ConnectX-3 @ $50 is a big enough difference. Since my switch won't support it, it doesn't seem worth it until I'm ready to get a new switch.

Should this cable be fine: OS2 LC to LC Fiber Optical Patch Cable Single mode Duplex LSZH, 30 meters | eBay ?

With the shipping delay, better to be certain. Thank you.

(and no, my storage array won't handle the data coming across a 100G nic anyway).

 

[Rootless]

As for the ConnectX-3 there are so many different versions.

Is it better to get a CX353A, a CB194A, or something different?

Thank you, thank you.

 

[Sean Ho]

pretty sure MCB194A is an IB-only card and not a ConnectX-3, that listing's description notwithstanding. 353 is fine, that's a single-port QSFP CX-3. 354 would be the dual-port one. The suffixes indicate IB speed with stock firmware, but all variants of CX353/354 can be flashed to FDR. There's also a FLR version for HP mez. And you know the CX311/312 with SFP+.

 

[Sean Ho]

oh there were some QCAT (rev A) cards that, while still flashable to FDR/40GbE, cannot use the same PSID as the rest.

 

[Rootless]

Thank you all. I just pulled the trigger on:
Mellanox CX354A MCX354A-FCBT ConnectX-3 40GbE Dual-Port Server HBA High Profile Mellanox CX354A MCX354A-FCBT ConnectX-3 40GbE Dual-Port Server HBA High Profile | eBay
OS2 LC to LC Fiber Optical Patch Cable Single mode Duplex LSZH, 30 meters OS2 LC to LC Fiber Optical Patch Cable Single mode Duplex LSZH, 30 meters | eBay
XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

I won't have to pull more fiber when I'm ready to upgrade to 100G and the transceivers were an awesome price (Go Rttg!).
And thank you again.

 

[mach3.2]

Thank you all. I just pulled the trigger on:
Mellanox CX354A MCX354A-FCBT ConnectX-3 40GbE Dual-Port Server HBA High Profile Mellanox CX354A MCX354A-FCBT ConnectX-3 40GbE Dual-Port Server HBA High Profile | eBay
OS2 LC to LC Fiber Optical Patch Cable Single mode Duplex LSZH, 30 meters OS2 LC to LC Fiber Optical Patch Cable Single mode Duplex LSZH, 30 meters | eBay
XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

I won't have to pull more fiber when I'm ready to upgrade to 100G and the transceivers were an awesome price (Go Rttg!).
And thank you again.

Eh btw anything older than a ConnectX4 won't have driver support on ESXi 8, so just a heads up in case you use ESXi.

 

[i386]

As for the ConnectX-3 there are so many different versions.

https://forums.servethehome.com/index.php?resources/mellanox-nics.47/

 

[Rootless]

Thank you for the Mellanox nics link. I knew it had to be here on the forum somewhere, but couldn't come up with the right key words.

And thank you for the ESXi 8 heads up. I was playing around with VMware at one point, but the hardware support was so picky it lost its use-case. Though I would still love to afford ConnectX-4 hardware, it isn't in the budget yet.