Which 40GbE/10GbE NICs and cable for PC to PC file transfers?

Discussion in 'Networking' started by SeanSSD, Oct 10, 2018.

  1. SeanSSD

    SeanSSD New Member

    Joined:
    Oct 10, 2018
    Messages:
    6
    Likes Received:
    0
    I want to connect my desktop, another workstation and NAS for faster file transfers for rather cheap (mainly video/photo content - 300-500GB transfers suck on 1GbE right now). I don't want to spend the $ on a switch yet because I'm on a rather tight budget, so it would be all directly connected to one another. All systems have Windows 10 Pro 64-bit installed. The distance between the desktop to the workstation is about 15ft (5m) and from both to the NAS is about 65ft (20m) and I have a bunch of NVMe PCIe SSDs that can take advantage of faster speeds than 10GbE.

    After looking around a bit, I was about to pull the trigger on three of these: Dell branded Intel X540-T2 cards or these Supermicro AOC-STG-I2T cards, or possibly save a few bucks and just get the Intel X520-T2 instead...but I'm not really sure what is the best value and least heat/throttling issues.

    I started to research other options and multiple threads here and found the Mellanox adapters like this Mellanox ConnectX-3 VPI or this HP 554 QSFP Host Bus Adapter card or maybe this HP-649281-B21 card that support 40GbE. This would be sick, but, I am a noob when it comes to the cables and connectors and don't yet know much about these cards or other options that may be cheaper/better value.

    Does anyone have any suggestions or recommendations on getting 10GbE or even 40GbE at a rather low cost for 2-3 systems with Windows 10 support?
     
    #1
    Last edited: Oct 10, 2018
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,450
    Likes Received:
    331
    HP branded mellanox connect-x3 nics (about $30 each) + qsfp+transceiver (starting at $20 for used ones,$35 for new from da.com, $120 new from mellanox) + mtp/mpo fibers (you could buy used from eBay or new from FS.com)
     
    #2
    SeanSSD likes this.
  3. SeanSSD

    SeanSSD New Member

    Joined:
    Oct 10, 2018
    Messages:
    6
    Likes Received:
    0
    Is this the HP card you are referring to? The HP-649281-B21? Any drivers needed/firmware reflashing, or all plug n play?

    Would this cable work? Anything cheaper? It seems it has the QSFP+ connections and is 20m for ~$125. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC A 5m cable is ~$100 with both connectors. -> 40GBASE QSFP+ AOC Cable, 20M | QSFP+ AOC | AOC Cable | QSFP AOC | AOC

    I'm not too good with fiber cabling terms yet. And how flexible are these fiber cables? I was planning on running the cable along some walls and around two door frames, which is about a 65ft-70ft (20m) run. I have a spool of Cat6a and terminals already. Can I utilize Cat6 for 10GbE now with QSFP+ tranceivers and these cards to save some $ now and I can upgrade later/rearrange things closer? Or should I just use 10GbE RJ45 nics instead? 10GbE is still plenty fast.
     
    #3
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,450
    Likes Received:
    331
    Yes.
    WIndows 10/Server 2016 have driver support, but I would recommend to use the latest mellanox drivers.
    You can use the hp firmware or flash to the stock mellanox firmware (I would recommend that: the mellanox driver package can then detect the firmware version and update it when necessary).
    These cables should work, but on everything that has to be longer than 3m (15feet) I would recommend getting optical transcivers (2 per connection) + fiber cable.
    You can bend them, but I don't know how much.

    Would this be enough? :
    [​IMG]
     
    #4
    SeanSSD likes this.
  5. SeanSSD

    SeanSSD New Member

    Joined:
    Oct 10, 2018
    Messages:
    6
    Likes Received:
    0
    Is this the guide I would follow? -> https://forums.servethehome.com/ind...ctx-3-en-with-stock-firmware.8689/#post-80356

    Also, do you, or anyone else, know where to get the best-priced fiber cable for this? Does fiber vs copper matter at 20m? I'm having trouble finding anything rather cheap.

    For transceivers, would these work? I'm just not sure what specs matter for these with compatibility with the cable and card. I know QSFP+ is the form factor, so it should work, but there are a bunch of other specs I'm unknowledgeable in. -> **LOT OF 2** AVAGO QSFP+ 40Gbe TRANSCEIVER AFBR-79EIDZ | eBay

    Also, reading into things more, I would need to utilize multiple transfers to saturate the link, correct? Single thread file transfers (aka drag n drop a big folder) would be limited to just 10GbE speeds in Windows 10. I would need to utilize a program to transfer with multiple threads to saturate it?
     
    #5
  6. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    932
    Likes Received:
    679
    #6
    SeanSSD likes this.
  7. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    932
    Likes Received:
    679
    those tranceivers will work, the spec you want is 40gbase-sr4 - I'm a fan of the avago's, mostly for the nice removal handles - Avago 40GB QSFP+ AFBR-79EQPZ Transceiver Parallel Ethernet InfiniBand 850nm 40G 654323094729 | eBay

    but keep in mind if you run fiber between two QSFP transceivers, it needs to be key up - key up fiber. 90% of the MPO/MTP fiber you'll find on ebay and elsewhere is key up-key down, which will not work. using an AOC like you linked (which is just a run of fiber and two transceivers, but preassembled, usually permanently) gets rid of that issue for you, but tend to be a little pricey, however you can find deals occasionally like 20m AOC-Q-Q-40G-20M Arista Networks Compatible 40G QSFP+ Active Optical Cable | eBay

    regarding single threads being limited to 10gbe - no. 40gbE is only 4x10gbE at the PHY level - the OS, pci-e card interface, etc everything after the port just sees a 40gbE channel. However you're still not going to do much better than 10gbe (if that) with a single threaded file copy, due to CIFS/SAMBA etc being awful performance wise without RDMA etc and you'll probably end up pegging a cpu core at 10gbps blahblahblah
     
    #7
    SeanSSD likes this.
Similar Threads: 40GbE/10GbE NICs
Forum Title Date
Networking Mellanox SFP 10G nics Oct 31, 2018
Networking Dead/dying onboard NICs? Aug 1, 2018
Networking Directly Connect two QSFP NICs with Active Optical cable? Jul 24, 2018
Networking List of NICs and their equivalent OEM parts Jun 28, 2018
Networking Cheap place to pick up quad port nics (1 pcie, 1 pcix) Jun 4, 2018

Share This Page