Good and cheap 10GbE Network infrastructure

brandmeister

New Member
Jan 28, 2020
1
0
1
Hi there,

I was wondering wich cards should I buy.
Stumbled across a pair of Mellanox ConnectX-2 on the ebay for 99€.
My goal ist using them with my Windows 10 Workstation (Threadripper 1900X, MSI X399 Carbon AC build) and my Homeserver (Xeon E5-2620v3 on SuperMicro X10SRi-F).
Just a single P2P connection for now, but I would like to add something like a MikroTik CRS309-1G-8S+IN later on.
If I should go fiber, which 10GbE SFP+ transceivers would you recommend?
Splicing the fiber would not be a problem for me, I am working as a field technician.

Edit: my Server is running Windows Server 2012 R2 Datacenter

Thanks in advance.
 
Last edited:

KC8FLB

Member
Aug 12, 2018
71
53
18
Brocade icx6450 switch for 4 SPF+ cages + 24/48 gb ports, see massive thread here: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-143#post-248792

HP OEM 649281 dual port 10G/40G/56G infiniband/VPI/Ethernet NICS + 10G adapters
flash them using these thread directions: https://forums.servethehome.com/ind...-ethernet-dual-port-qsfp-adapter.20525/page-7

example:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay

10G adapters: Mellanox MAM1Q00A-QSA 40Gbps to 10Gbps QSFP to SFP+ Ethernet Cable Adapter 676793872508 | eBay

someone will have to help me on the tranceivers, but I think most any SR SFP+ tranceiver should work with the mellanox cards, but there are compatability issues with other cards: Cisco SFP-10G-SR *Cisco Genuine* 10Gbase-SR SFP+ *TESTED & VERIFIED* - Warranty | eBay

get some OM4 LC-LC patch cables from fs.com and you should be all set.
 
  • Like
Reactions: EasyRhino

Mithril

Active Member
Sep 13, 2019
172
43
28
Brocade icx6450 switch for 4 SPF+ cages + 24/48 gb ports, see massive thread here: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-143#post-248792

HP OEM 649281 dual port 10G/40G/56G infiniband/VPI/Ethernet NICS + 10G adapters
flash them using these thread directions: https://forums.servethehome.com/ind...-ethernet-dual-port-qsfp-adapter.20525/page-7

example:
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay

10G adapters: Mellanox MAM1Q00A-QSA 40Gbps to 10Gbps QSFP to SFP+ Ethernet Cable Adapter 676793872508 | eBay

someone will have to help me on the tranceivers, but I think most any SR SFP+ tranceiver should work with the mellanox cards, but there are compatability issues with other cards: Cisco SFP-10G-SR *Cisco Genuine* 10Gbase-SR SFP+ *TESTED & VERIFIED* - Warranty | eBay

get some OM4 LC-LC patch cables from fs.com and you should be all set.
It looks like the HP OEM 649281 will also do 40GB ethernet, is that correct?

Edit: So it looks like it may need to be cross-flashed to do so, but that it should be doable. Also it looks like this *should* be a listing for the same card, but cheaper if the model number is right: 764618-001 REF HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter | eBay

Edit2: Nope! the "FLR" means Flexible LOM" which per other threads does NOT work in a standard PCIe slot.
 
Last edited:

Bert

Active Member
Mar 31, 2018
370
149
43
42
On this topic: Is NetApp NAE-1101 16-Port viable 10G switch for home networking? the criteria here is being reasonably simple, quiet and power efficient. I plan to use this in my garage and replace my gs724tv4.
 

KC8FLB

Member
Aug 12, 2018
71
53
18
It looks like the HP OEM 649281 will also do 40GB ethernet, is that correct?
Yes, 10G/40G Ethernet and Infiniband and proprietary Mellanox 56G as well. I have three of these cards, one in a Win 10 box, one in Ubuntu 19.04 Desktop and one in my UNRAID NAS/Server all working great. I am using the 10G mellanox reducer/adapters SFP+ SR tranceivers ($7 each) and all connected to a Brocade 6450. Make sure to install the driver packages (OFED) for either Linux (make sure to install all of the package dependencies in the release notes/instructions for OFED) or Windows. UNRAID was plug (configure in GUI) and play, no driver installation needed.

If you get the HP 649281-B21, then make sure to flash them to the latest and greatest FW for Mellanox MCX354A-FCBT (equivalent card) using the instructions posted above. You also need to set the port interface type (Ethernet or Infiniband or Sense) Win 10 worked great with sensing, but I had to manually set my other two cards to Ethernet for Linux/UNRAID to work easily. I am no expert, and this was my first time setting up fiber, but I got it all working perfectly.

If you want to use 40G ethernet, check out the Brocade thread above and look at the icx6610 which has four 40G ports + 8 SFP+ 24/48 1G copper.
 

Mithril

Active Member
Sep 13, 2019
172
43
28
Is there a similar model number to look for "hidden" OEM Connectx-3s that are natively SFP+? I will pick up 2 of these, but I would love to update some of my machines with the older connectx-2 cards, and don't really want to get a whole stack of the qsfp->sfp+ adapters.