Good and cheap 10GbE Network infrastructure

[brandmeister]

Hi there,

I was wondering wich cards should I buy.
Stumbled across a pair of Mellanox ConnectX-2 on the ebay for 99€.
My goal ist using them with my Windows 10 Workstation (Threadripper 1900X, MSI X399 Carbon AC build) and my Homeserver (Xeon E5-2620v3 on SuperMicro X10SRi-F).
Just a single P2P connection for now, but I would like to add something like a MikroTik CRS309-1G-8S+IN later on.
If I should go fiber, which 10GbE SFP+ transceivers would you recommend?
Splicing the fiber would not be a problem for me, I am working as a field technician.

Edit: my Server is running Windows Server 2012 R2 Datacenter

Thanks in advance.

 

[KC8FLB]

Brocade icx6450 switch for 4 SPF+ cages + 24/48 gb ports, see massive thread here: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-143#post-248792

HP OEM 649281 dual port 10G/40G/56G infiniband/VPI/Ethernet NICS + 10G adapters
flash them using these thread directions: https://forums.servethehome.com/ind...-ethernet-dual-port-qsfp-adapter.20525/page-7

example:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay

10G adapters: Mellanox MAM1Q00A-QSA 40Gbps to 10Gbps QSFP to SFP+ Ethernet Cable Adapter 676793872508 | eBay

someone will have to help me on the tranceivers, but I think most any SR SFP+ tranceiver should work with the mellanox cards, but there are compatability issues with other cards: Cisco SFP-10G-SR *Cisco Genuine* 10Gbase-SR SFP+ *TESTED & VERIFIED* - Warranty | eBay

get some OM4 LC-LC patch cables from fs.com and you should be all set.

 

[Mithril]

Brocade icx6450 switch for 4 SPF+ cages + 24/48 gb ports, see massive thread here: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-143#post-248792

HP OEM 649281 dual port 10G/40G/56G infiniband/VPI/Ethernet NICS + 10G adapters
flash them using these thread directions: https://forums.servethehome.com/ind...-ethernet-dual-port-qsfp-adapter.20525/page-7

example:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay

10G adapters: Mellanox MAM1Q00A-QSA 40Gbps to 10Gbps QSFP to SFP+ Ethernet Cable Adapter 676793872508 | eBay

someone will have to help me on the tranceivers, but I think most any SR SFP+ tranceiver should work with the mellanox cards, but there are compatability issues with other cards: Cisco SFP-10G-SR *Cisco Genuine* 10Gbase-SR SFP+ *TESTED & VERIFIED* - Warranty | eBay

get some OM4 LC-LC patch cables from fs.com and you should be all set.

It looks like the HP OEM 649281 will also do 40GB ethernet, is that correct?

Edit: So it looks like it may need to be cross-flashed to do so, but that it should be doable. Also it looks like this *should* be a listing for the same card, but cheaper if the model number is right: 764618-001 REF HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter | eBay

Edit2: Nope! the "FLR" means Flexible LOM" which per other threads does NOT work in a standard PCIe slot.

 

[Bert]

On this topic: Is NetApp NAE-1101 16-Port viable 10G switch for home networking? the criteria here is being reasonably simple, quiet and power efficient. I plan to use this in my garage and replace my gs724tv4.

 

[KC8FLB]

It looks like the HP OEM 649281 will also do 40GB ethernet, is that correct?

Yes, 10G/40G Ethernet and Infiniband and proprietary Mellanox 56G as well. I have three of these cards, one in a Win 10 box, one in Ubuntu 19.04 Desktop and one in my UNRAID NAS/Server all working great. I am using the 10G mellanox reducer/adapters SFP+ SR tranceivers ($7 each) and all connected to a Brocade 6450. Make sure to install the driver packages (OFED) for either Linux (make sure to install all of the package dependencies in the release notes/instructions for OFED) or Windows. UNRAID was plug (configure in GUI) and play, no driver installation needed.

If you get the HP 649281-B21, then make sure to flash them to the latest and greatest FW for Mellanox MCX354A-FCBT (equivalent card) using the instructions posted above. You also need to set the port interface type (Ethernet or Infiniband or Sense) Win 10 worked great with sensing, but I had to manually set my other two cards to Ethernet for Linux/UNRAID to work easily. I am no expert, and this was my first time setting up fiber, but I got it all working perfectly.

If you want to use 40G ethernet, check out the Brocade thread above and look at the icx6610 which has four 40G ports + 8 SFP+ 24/48 1G copper.

 

[Mithril]

Is there a similar model number to look for "hidden" OEM Connectx-3s that are natively SFP+? I will pick up 2 of these, but I would love to update some of my machines with the older connectx-2 cards, and don't really want to get a whole stack of the qsfp->sfp+ adapters.

 

[brandmeister]

I went with the Mellanox ConnectX-3 and they are working really good.