The usual noob question(s) about cheap 10gbe again

[dreamcat4]

Hello there thinking about a simple 2 computer 10g point to point link. These are both linux machines running modern 5.2+ kernel, ubuntu 19.10. Each computer has one free pci-e 3.0 x4 slot. Which I understand can also take x8 cards if needed (with some slight modification, that I don’t mind).

IDK what I am doing. and only the last choice seem like (to me) something I can reliably count on as working. Currently, my 4 main choices seems to be one of:

1) A pair of these single port cards

2x Mellanox ConnectX-3 MCX341 SFP+10Gigabit Ethernet RMDA/RoCE | eBay
+1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £73.60

• OR

2) A pair of these dual port cards

2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
+1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £99.95

• OR

3) Dual link is possible? for 20GBe teamed ?

2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
+2 cables Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £123.95

• OR

4) Pair of AQN-107 modern Aquantia cards (RJ45)

2x US $76.95 5% OFF|LR LINK 6880BT PCIe x4 Single Port 10G Gigabit Ethernet RJ45 copper network adapter PC Express Controller lan card server Nic-in Network Cards from Computer & Office on AliExpress

with existing RJ45 cat6 cabling that I already have ( which is 1000mhz, 10gb capable)

= £129.45 total

• OR

5) Is there anything else I should be considering, that is better (=cheaper but still works) ?

Another question:

How can I be sure that the SFP+ cable is going to be compatible / work with those cards? It says it’s a used CISCO cable. Therefore should I be buying a MELLANOX branded SFP+ cable instead? Or am I just being paranoid?

Any help appreciated with links to specific product(s). Either on aliexpress or for UK / ebay. Many thanks.

 

[TRACKER]

Hello,

what about "other"

2x
Dell Broadcom 57810S Dual Port 10GBE Converged Network Adapter W1GCR | eBay

 

[dreamcat4]

2x
Dell Broadcom 57810S Dual Port 10GBE Converged Network Adapter W1GCR | eBay

Because that is at a very a similar price point (in UK) as either options 2) or 3). What is missing here: is that it's not clear to me at all why that would be better in any way to 2) or 3) given that those ones are 3.0 x4 slot xconnect-3's and without the fan. Which are all plus points for my intended usage. So then what is the possible motivation to get those ones instead?

 

[TRACKER]

One good reason is UTP cable instead SFP+
the closest option which is similar to the one, which i've suggested is
Option #3 - £123.95 - Pair of dual port cards, Dual link 20GBe teamed ?
compared to suggested by me: Total US $106.40 -> ~£83 (including delivery/taxes), that's 50% less

 

[dreamcat4]

Awesome! Yes... that would indeed save me a quite a significant amount of money and hassles. Thank you

 

[TRACKER]

you are welcome

 

[zer0sum]

If you use ConnectX-3 cards with a good DAC you can do 40 or 56 GbE instead of that measly 10Gbe you're aiming for

 

[dreamcat4]

If you use ConnectX-3 cards with a good DAC you can do 40 or 56 GbE instead of that measly 10Gbe you're aiming for

OK so if 40gbe is possible with that option, and assuming that the pci-e 3.0 x4 connection will be capping out at about 3940mb/sec theoretical bandwith... then can you suggest a known good DAC cable. Does it have to match the same brand and be an official Mellanox one? Since the one I linked to isn't, and IDK which one is best.

Also (and this is actually rather academic for me being limited to x4 slot, but it might matter more for other people who can run pcie 3.0 x8 lanes)... is there anyway to know ahead of time if it's possible to do dual teaming of both links ? Because if not then the single port cards are slightly cheaper and it would avoid wasting extra money on buying a 2nd similar DAC cable if to then only discover its not possible to pair the connections together like that. Thank you both.

 

[zer0sum]

I've used a variety of DAC's from Ebay and Amazon and they've all been fine for me.

I have my Unraid server dicrectly connected to Windows 10, and my NVME to NVME speeds across the 40GbE link are around 900-1000 MB/s

 

[dreamcat4]

Hey but shouldn't a 40gbe link be capabable to go any faster than that? For example with 2 ramdisks instead, setup on both ends of the connection.

Otherwise the max speed seems not much different than 10gbe, isn't it ?