40G Network Setup Advice Needed

[pete22]

Hi everybody,

I need to setup a 40G network, I am connecting a single workstation with a server. As of right now, I only need point-to-point between these two machines. Distance is app. 15m.

Been reading up on 40G required hardware, but am a little bit confused, so I have some questions - please correct where wrong:

(1) As far as I understand it, I need these items:

> 2 x network controller cards capable of 40G
> 2 x transceivers that are compatible with that network controller
> cable that is compatible with the the transceivers

Anything else needed for point-to-point ?

(2) I'm looking at the Mellanox MCX353A-FCBT ConnectX-3 card. It appears it can do 40G.

What is the procedure to enable single 40G on that card, instead of 4 x 10G ?
Any suggestions for a different network card ?
Does the mellanox card have any compatibility issues regarding certain motherboards or OS ?

(3) For transceivers, I found this one Mellanox 40G QSFP+ MC2210411-SR4 Transceiver Module | FS.COM

seems to be Mellanox compatible...
any suggestions for different transceivers ?

(4) Cable

Since I'm over 10m distance, I will have to go with MTP/MPO cables. I found this one: Customized 12-Fiber OM4 12 Strands MTP Trunk Cable 3.0mm | FS.COM

Can anybody confirm this actually works with the Mellanox transceivers from above ?


Thanks for your help !

pete

 

[darkconz]

I can't comment on the cable as in my setups I've always used DAC.

However, FS.com will program the transceiver to your requirement so just drop them a note to remind them those are going into the Mellanox card direct.

The cards will do 40GBe by default, no special setup is required.


Sent from my iPhone using Tapatalk

 

[pete22]

oh wow, interesting.

So, do the transceivers always have to be programmed for the specific card in use ?

Or am I misunderstanding this and I don't need transceivers for point-to-point ?

Thanks.

 

[i386]

(Optical) transceivers are required when you want to use optic fibers for longer distances (> 10meter).
For shorter distances you could use qsfp+ dac cables. They come in passive (up to 7meter) and active (7to 15meter) versions.

Passive dac cables should work with all 40gbe capable devices (per specification), but active cables and optical transcceivers (also count as active "cables") can cause problems.

 

[pete22]

(Optical) transceivers are required when you want to use optic fibers for longer distances (> 10meter).
For shorter distances you could use qsfp+ dac cables. They come in passive (up to 7meter) and active (7to 15meter) versions.

Passive dac cables should work with all 40gbe capable devices (per specification), but active cables and optical transcceivers (also count as active "cables") can cause problems.

ok, are u saying that when using passive qsfp+ dac cables, one does not need a transceiver and directly plugs the dac into the network card ?

"but active cables and optical transcceivers (also count as active "cables") can cause problems"

what does this relate to and how do we avoid the problems ?

Thanks.

 

[Rand__]

ok, are u saying that when using passive qsfp+ dac cables, one does not need a transceiver and directly plugs the dac into the network card ?

Correct

"but active cables and optical transcceivers (also count as active "cables") can cause problems"
what does this relate to and how do we avoid the problems ?
Thanks.

Problem: Its not working
Solution: Use known compatible cables (type), Transceivers<->Cards (ie the above solution to program to MLX). Programming usually is done by the manufacturer for their specific cards (ie Intel, Cisco, HP etc), FS.com has blanks they can configure (or maybe reconfigure, no idea).

 

[i386]

Active cables/optical transceivers have magic numbers programmed in the eprom. When you plug them into devices (network card, switch or router) the devices could read that number and compare it to known numbers (and refuse to work with unknown numbers).
Some shops can reprogram these magic numbers (this is what @darkconz said about fs.com) and make them compatible for your devices.

 

[pete22]

very informative, thank you everybody.

So maybe if I can keep distance below 10m I can simply start off with 2 x 40G cards + 1 DAC....

any other suggestion for a 40G card other than the Mellanox MCX353A-FCBT ?

 

[Rand__]

354A ?
Kidding
Connectx3 based cards are more modern and support up to 56GB/s but are more expensive.
Depending on funding you can also get connect X2 based ones for really cheap (seen them at 20 bucks for a single port), but you might need to check target OS support (out of the box or at all), eg you need manual driver installation for x2 in ESX 6.5 but it works fine then.

You looking for EN or IB? X2 will not do 40 EN iirc

 

[pete22]

354A ?
Kidding
Connectx3 based cards are more modern and support up to 56GB/s but are more expensive.
Depending on funding you can also get connect X2 based ones for really cheap (seen them at 20 bucks for a single port), but you might need to check target OS support (out of the box or at all), eg you need manual driver installation for x2 in ESX 6.5 but it works fine then.

You looking for EN or IB? X2 will not do 40 EN iirc

target OS will be Windows 10.

Regarding "EN or IB", could u elaborate on that ? Is that Ethernet or Infiniband ? (I thought Infiniband was simply a specific ethernet standard)

I need the transfer speed of 40G, but I've never installed or used 40G. I'm not sure what the difference is in real world application to networking via standard ethernet controller cards...

Apologize, but complete noob ;-)

 

[Rand__]

Yes Ethernet and Infiniband.
No, IB is not a fancy Ethernet type, different protocol/architecture, specs. You'll need Subnet Manager (software possible) for IB.
Win10 should support both types.
Suggest to read up on what you need/want Sorry got no real good links.

 

[i386]

I need the transfer speed of 40G

Now I'm curious...
All flash storage? Some sort of hpc or cluster rendering/computing?

I got 40gbe only becuase I found cheap fdr ib/40gbe hp branded mellanox cards when I searched for infiniband nics

 

[pete22]

Yes Ethernet and Infiniband.
No, IB is not a fancy Ethernet type, different protocol/architecture, specs. You'll need Subnet Manager (software possible) for IB.
Win10 should support both types.
Suggest to read up on what you need/want Sorry got no real good links.

I tried to read up on it, but not found a single good resource that explains all the differences and details (like the stuff with the encoding and the transceivers that I learned here) and why I would prefer one over the other... so, since I need 40G (or at least way more than 10G), is there an ethernet variant that does 40G ?

if Win 10 supports both types natively, and hypothetically I got the IB installed to an external server, is it like simply accessing a network drive (like it is with EN) ?

Thanks !

 

[pete22]

Now I'm curious...
All flash storage? Some sort of hpc or cluster rendering/computing?

I got 40gbe only becuase I found cheap fdr ib/40gbe hp branded mellanox cards when I searched for infiniband nics

will be used to access a render cache and since the files are very high resolution and we need real time playback we need more than 10G...

 

[Rand__]

Older but a ton of info (not for the faint hearted) http://www.ics.uci.edu/~ccgrid11/files/ccgrid11-ib-hse_last.pdf

For your use case IB might be beneficial due to reduced latency and direct memory access options. I am not too deep into IB since I went the easy way with EN mode.
I assume both IB and EN will work at 40GBe (+), not sure whether IB can do it in direct connect but I assume so.
Somebody with more info on this should be able to help out