Help me understand LC module compatibility

[Atlantic]

Hello STH. I'm just getting started with a home server and hit a stumbling block that I could do some help with.

I bought some mismatched 10G nics for peer to peer over DAC between my r720 server and clients. Works fine. Now I need more distance so want to try fibre so bought another chelsio card that came with LC transceivers, and some fibre cable. But I couldn't get that to work. Transceivers are Brocade and I can see Freenas saying 'unknown PNY inserted' when I slot them in the card.

I assumed the fiber port was an open standard and that any transceiver would work, as long as it was the same speed. Is that not the case? Are they specific to a brand of card?

NICs are chelsio and Solar flare. What modules do I need to get, or will mismatched cards not work at all with fiber?

Thx

 

[fohdeesha]

some NIC vendors like chelsio and intel hate freedom, and lock their cards so they will only work with their own brand of optics transceivers inserted. The chelsio card wants a chelsio optic. These vendors are going to burn in the afterlife, and it's one of the numerous reasons almost everyone here prefers mellanox for most things as they're the same price and have no such arbitrary limitations. Mellanox support in freenas has been flawless for a while now, 40gbE connect x3's are down to around ~$35

 

[fohdeesha]

(there's also the issue of matching optic standard type, there are different optics standards for different lengths and speeds, these need to be matched as well. For short distance (under ~700ft), that would be 10gbase-SR, for longer stuff there's 10gbase-LR, for even longer there's others, but I can almost promise all your transceivers are 10gbase-SR so that's not the issue)

 

[Atlantic]

some NIC vendors like chelsio and intel hate freedom, and lock their cards so they will only work with their own brand of optics transceivers inserted. The chelsio card wants a chelsio optic. These vendors are going to burn in the afterlife, and it's one of the numerous reasons almost everyone here prefers mellanox for most things as they're the same price and have no such arbitrary limitations. Mellanox support in freenas has been flawless for a while now, 40gbE connect x3's are down to around ~$35

Thanks for the reply, that makes sense. Also probably why there are so few Mellanox cards on ebay (in the UK anyway). Why does that Lawrence Systems guy recommend chelsio so much, that's the main reason I sought them out.

So if I got chelsio optics for the card in the r720, am I right in assuming the optical signal its sending is just data and there's nothing proprietary and it would connect on the other end with any other brand of NIC?

EDIT: just saw your other post, the above is also assuming I have the right type of cable. Its turqoise with this sepc from amazon, and says its compatible with 10BASE-SR SFP+.
10m 10G Gigabit Fiber Patch Cables with LC to LC Multimode OM3 Duplex 50/125 OFNP'
.

 

[fohdeesha]

Why does that Lawrence Systems guy recommend chelsio so much, that's the main reason I sought them out.

Like a lot of other people it's probably a holdover in opinion from a few years back - 3 or 4 years or so ago mellanox did not have very good (if any) freebsd support, so freenas and pfsense etc users avoided them like the plague. Their opinions have not adapted as freebsd and freebsd based distros have gotten very mature mellanox drivers in the meantime. Aside from that I quite like Tom from lawrence systems (he constantly pimps a project I contribute to, XCP-ng, which helps)

regarding low mellanox inventory on ebay - are you sure? what are you searching? for a given search I have about 10x mellanox results than I do chelsio, try searching "connectx-3" for instance

as for the optics, there's indeed nothing proprietary about them, but each side still needs to be the same standard, for instance 10gbase-sr and 10gbase-lr both are SFP+ modules that use the same LC connector type but will not talk to each other as they use different wavelengths for starters. As long as you match the proper standard on each side though then of course any vendor will work with any vendor. Multimode fiber is correct for 10gbase-SR which is what you have

 

[Atlantic]

Like a lot of other people it's probably a holdover in opinion from a few years back - 3 or 4 years or so ago mellanox did not have very good (if any) freebsd support, so freenas and pfsense etc users avoided them like the plague. Their opinions have not adapted as freebsd and freebsd based distros have gotten very mature mellanox drivers in the meantime. Aside from that I quite like Tom from lawrence systems (he constantly pimps a project I contribute to, XCP-ng, which helps)

regarding low mellanox inventory on ebay - are you sure? what are you searching? for a given search I have about 10x mellanox results than I do chelsio, try searching "connectx-3" for instance

as for the optics, there's indeed nothing proprietary about them, but each side still needs to be the same standard, for instance 10gbase-sr and 10gbase-lr both are SFP+ modules that use the same LC connector type but will not talk to each other as they use different wavelengths for starters. As long as you match the proper standard on each side though then of course any vendor will work with any vendor. Multimode fiber is correct for 10gbase-SR which is what you have

Ah yes I've dabbled with XCP-ng... I was going to use it to run multiple GPUs on the r720 with Freenas on a vm, alas I couldn't get XCP-ng to see my GPUs, but that's another story. XCP-ng seemed a lot easier to use than ESXi, but then again I'm a noob.

I search 'Mellanox SFP 10G' or just Mellanox, I get 2 hits and they're very expensive (£50 and £83). Whereas the SF and Chelsio I picked up around £20 or less.

connectx-3 does produce much more results, but I see lots of QSFP starting at £40 (and I dont know what that interface is). connectx-3 +SFP gives me two hits, £108, £120. Too spendy :0

 

[Deslok]

QSFP is 40Gb(basically bonded 4x10Gb if I recall correctly which there's ways to use as 4 10gig ports for some cards) if you're after 10Gb you're probably looking for a listing like this one, it looks like a lot of the 10gig cards are creeping up in price a little likely due to the increase in 10Gb gear that's affordable in general.

Mellanox MCX312A-XCBT CX312A ConnectX-3 EN 10GbE SFP+ Dual-Port PCI-E NIC | eBay

 

[Atlantic]

Yes, prices for those do seem higher. Much more of them about in the US too, hardly any in the UK (lots of other brands, SF and Chelsio, HP etc). Yes 10GB SFP is all I need, mainly for the distance, the DAC cables I'm using work great, no compatibility issues there, they're just a bit too short.

 

[Deslok]

I don't know their cost in the UK(they're usually 20 or less here in the us) but i've also seen what's called a QSA, they let you use a SFP+ module in a QSFP port, that would let you get 40gb cards and possibly save money while having 40gb as an option in the future without new network cards.

 

[Atlantic]

Ah ok that's a good tip. I'll do some searches on those.