Which 40GbE/10GbE NICs and cable for PC to PC file transfers?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

SeanSSD

New Member
Oct 10, 2018
6
0
1
I want to connect my desktop, another workstation and NAS for faster file transfers for rather cheap (mainly video/photo content - 300-500GB transfers suck on 1GbE right now). I don't want to spend the $ on a switch yet because I'm on a rather tight budget, so it would be all directly connected to one another. All systems have Windows 10 Pro 64-bit installed. The distance between the desktop to the workstation is about 15ft (5m) and from both to the NAS is about 65ft (20m) and I have a bunch of NVMe PCIe SSDs that can take advantage of faster speeds than 10GbE.

After looking around a bit, I was about to pull the trigger on three of these: Dell branded Intel X540-T2 cards or these Supermicro AOC-STG-I2T cards, or possibly save a few bucks and just get the Intel X520-T2 instead...but I'm not really sure what is the best value and least heat/throttling issues.

I started to research other options and multiple threads here and found the Mellanox adapters like this Mellanox ConnectX-3 VPI or this HP 554 QSFP Host Bus Adapter card or maybe this HP[URL='http://rover.ebay.com/rover/1/711-53200-19255-0/1?pub=5574933291&toolid=10001&campid=5337060739&customid=&mpt=499277&mpre=https%253A%252F%252Fwww.ebay.com%252Fitm%252FHP-649281-B21-Infiniband-10-40GB-Dual-Port-Adapter-Low-Profile-Bracket%252F392018428744']-649281-B21 card[/URL] that support 40GbE. This would be sick, but, I am a noob when it comes to the cables and connectors and don't yet know much about these cards or other options that may be cheaper/better value.

Does anyone have any suggestions or recommendations on getting 10GbE or even 40GbE at a rather low cost for 2-3 systems with Windows 10 support?
 
Last edited:

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
HP branded mellanox connect-x3 nics (about $30 each) + qsfp+transceiver (starting at $20 for used ones,$35 for new from da.com, $120 new from mellanox) + mtp/mpo fibers (you could buy used from eBay or new from FS.com)
 
  • Like
Reactions: SeanSSD

SeanSSD

New Member
Oct 10, 2018
6
0
1
Is this the HP card you are referring to? The HP-649281-B21? Any drivers needed/firmware reflashing, or all plug n play?

Would this cable work? Anything cheaper? It seems it has the QSFP+ connections and is 20m for ~$125. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC A 5m cable is ~$100 with both connectors. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC

I'm not too good with fiber cabling terms yet. And how flexible are these fiber cables? I was planning on running the cable along some walls and around two door frames, which is about a 65ft-70ft (20m) run. I have a spool of Cat6a and terminals already. Can I utilize Cat6 for 10GbE now with QSFP+ tranceivers and these cards to save some $ now and I can upgrade later/rearrange things closer? Or should I just use 10GbE RJ45 nics instead? 10GbE is still plenty fast.
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Is this the HP card you are referring to? The HP-649281-B21?
Yes.
Any drivers needed/firmware reflashing, or all plug n play?
WIndows 10/Server 2016 have driver support, but I would recommend to use the latest mellanox drivers.
You can use the hp firmware or flash to the stock mellanox firmware (I would recommend that: the mellanox driver package can then detect the firmware version and update it when necessary).
Would this cable work? Anything cheaper? It seems it has the QSFP+ connections and is 20m for ~$125. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC A 5m cable is ~$100 with both connectors. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC
These cables should work, but on everything that has to be longer than 3m (15feet) I would recommend getting optical transcivers (2 per connection) + fiber cable.
And how flexible are these fiber cables?
You can bend them, but I don't know how much.

Would this be enough? :
 
  • Like
Reactions: SeanSSD

SeanSSD

New Member
Oct 10, 2018
6
0
1
Yes.

Windows 10/Server 2016 have driver support, but I would recommend to use the latest Mellanox drivers.
You can use the hp firmware or flash to the stock Mellanox firmware (I would recommend that: the Mellanox driver package can then detect the firmware version and update it when necessary).

These cables should work, but on everything that has to be longer than 3m (15feet) I would recommend getting optical transceivers (2 per connection) + fiber cable.

You can bend them, but I don't know how much.

Would this be enough?
Is this the guide I would follow? -> https://forums.servethehome.com/ind...ctx-3-en-with-stock-firmware.8689/#post-80356

Also, do you, or anyone else, know where to get the best-priced fiber cable for this? Does fiber vs copper matter at 20m? I'm having trouble finding anything rather cheap.

For transceivers, would these work? I'm just not sure what specs matter for these with compatibility with the cable and card. I know QSFP+ is the form factor, so it should work, but there are a bunch of other specs I'm unknowledgeable in. -> **LOT OF 2** AVAGO QSFP+ 40Gbe TRANSCEIVER AFBR-79EIDZ | eBay

Also, reading into things more, I would need to utilize multiple transfers to saturate the link, correct? Single thread file transfers (aka drag n drop a big folder) would be limited to just 10GbE speeds in Windows 10. I would need to utilize a program to transfer with multiple threads to saturate it?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,082
113
33
fohdeesha.com
those tranceivers will work, the spec you want is 40gbase-sr4 - I'm a fan of the avago's, mostly for the nice removal handles - Avago 40GB QSFP+ AFBR-79EQPZ Transceiver Parallel Ethernet InfiniBand 850nm 40G 654323094729 | eBay

but keep in mind if you run fiber between two QSFP transceivers, it needs to be key up - key up fiber. 90% of the MPO/MTP fiber you'll find on ebay and elsewhere is key up-key down, which will not work. using an AOC like you linked (which is just a run of fiber and two transceivers, but preassembled, usually permanently) gets rid of that issue for you, but tend to be a little pricey, however you can find deals occasionally like 20m AOC-Q-Q-40G-20M Arista Networks Compatible 40G QSFP+ Active Optical Cable | eBay

regarding single threads being limited to 10gbe - no. 40gbE is only 4x10gbE at the PHY level - the OS, pci-e card interface, etc everything after the port just sees a 40gbE channel. However you're still not going to do much better than 10gbe (if that) with a single threaded file copy, due to CIFS/SAMBA etc being awful performance wise without RDMA etc and you'll probably end up pegging a cpu core at 10gbps blahblahblah
 
  • Like
Reactions: SeanSSD