The usual noob question(s) about cheap 10gbe again

Which option is best?

  • Option #1 - £73.60 - Pair of single port cards, 10 SFP+ cable

    Votes: 0 0.0%
  • Option #3 - £123.95 - Pair of dual port cards, Dual link 20GBe teamed ?

    Votes: 0 0.0%
  • Option #4 - £129.45 - Pair of AQN-107 modern Aquantia cards (RJ45)

    Votes: 0 0.0%

  • Total voters
    2

dreamcat4

New Member
Nov 17, 2019
5
0
1
Hello there thinking about a simple 2 computer 10g point to point link. These are both linux machines running modern 5.2+ kernel, ubuntu 19.10. Each computer has one free pci-e 3.0 x4 slot. Which I understand can also take x8 cards if needed (with some slight modification, that I don’t mind).

IDK what I am doing. and only the last choice seem like (to me) something I can reliably count on as working. Currently, my 4 main choices seems to be one of:

1) A pair of these single port cards

2x Mellanox ConnectX-3 MCX341 SFP+10Gigabit Ethernet RMDA/RoCE | eBay
+1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £73.60

  • OR
2) A pair of these dual port cards

2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
+1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £99.95

  • OR
3) Dual link is possible? for 20GBe teamed ?

2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
+2 cables Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

= £123.95

  • OR
4) Pair of AQN-107 modern Aquantia cards (RJ45)

2x US $76.95 5% OFF|LR LINK 6880BT PCIe x4 Single Port 10G Gigabit Ethernet RJ45 copper network adapter PC Express Controller lan card server Nic-in Network Cards from Computer & Office on AliExpress

with existing RJ45 cat6 cabling that I already have ( which is 1000mhz, 10gb capable)

= £129.45 total

  • OR
5) Is there anything else I should be considering, that is better (=cheaper but still works) ?

Another question:

How can I be sure that the SFP+ cable is going to be compatible / work with those cards? It says it’s a used CISCO cable. Therefore should I be buying a MELLANOX branded SFP+ cable instead? Or am I just being paranoid?

Any help appreciated with links to specific product(s). Either on aliexpress or for UK / ebay. Many thanks.
 

dreamcat4

New Member
Nov 17, 2019
5
0
1

TRACKER

Member
Jan 14, 2019
55
12
8
One good reason is UTP cable instead SFP+
the closest option which is similar to the one, which i've suggested is
Option #3 - £123.95 - Pair of dual port cards, Dual link 20GBe teamed ?
compared to suggested by me: Total US $106.40 -> ~£83 (including delivery/taxes), that's 50% less
 
Last edited:

dreamcat4

New Member
Nov 17, 2019
5
0
1
Awesome! Yes... that would indeed save me a quite a significant amount of money and hassles. Thank you :)
 

zer0sum

Active Member
Mar 8, 2013
388
142
43
If you use ConnectX-3 cards with a good DAC you can do 40 or 56 GbE instead of that measly 10Gbe you're aiming for :)

upload_2019-11-18_12-4-49.png
 
  • Like
Reactions: dreamcat4

dreamcat4

New Member
Nov 17, 2019
5
0
1
If you use ConnectX-3 cards with a good DAC you can do 40 or 56 GbE instead of that measly 10Gbe you're aiming for :)
OK so if 40gbe is possible with that option, and assuming that the pci-e 3.0 x4 connection will be capping out at about 3940mb/sec theoretical bandwith... then can you suggest a known good DAC cable. Does it have to match the same brand and be an official Mellanox one? Since the one I linked to isn't, and IDK which one is best.

Also (and this is actually rather academic for me being limited to x4 slot, but it might matter more for other people who can run pcie 3.0 x8 lanes)... is there anyway to know ahead of time if it's possible to do dual teaming of both links ? Because if not then the single port cards are slightly cheaper and it would avoid wasting extra money on buying a 2nd similar DAC cable if to then only discover its not possible to pair the connections together like that. Thank you both.
 

zer0sum

Active Member
Mar 8, 2013
388
142
43
I've used a variety of DAC's from Ebay and Amazon and they've all been fine for me.

I have my Unraid server dicrectly connected to Windows 10, and my NVME to NVME speeds across the 40GbE link are around 900-1000 MB/s :D

upload_2019-11-18_12-57-33.png
 

dreamcat4

New Member
Nov 17, 2019
5
0
1
Hey but shouldn't a 40gbe link be capabable to go any faster than that? For example with 2 ramdisks instead, setup on both ends of the connection.

Otherwise the max speed seems not much different than 10gbe, isn't it ?