The usual noob question(s) about cheap 10gbe again

Discussion in 'Networking' started by dreamcat4, Nov 17, 2019.

?

Which option is best?

  1. Option #1 - £73.60 - Pair of single port cards, 10 SFP+ cable

    0 vote(s)
    0.0%
  2. Option #2 - £99.95 - Pair of dual port cards, 10m SFP+ cable

    0 vote(s)
    0.0%
  3. Option #3 - £123.95 - Pair of dual port cards, Dual link 20GBe teamed ?

    0 vote(s)
    0.0%
  4. Option #4 - £129.45 - Pair of AQN-107 modern Aquantia cards (RJ45)

    0 vote(s)
    0.0%
  5. Option #5 - Something else

    100.0%
  1. dreamcat4

    dreamcat4 New Member

    Joined:
    Nov 17, 2019
    Messages:
    5
    Likes Received:
    0
    Hello there thinking about a simple 2 computer 10g point to point link. These are both linux machines running modern 5.2+ kernel, ubuntu 19.10. Each computer has one free pci-e 3.0 x4 slot. Which I understand can also take x8 cards if needed (with some slight modification, that I don’t mind).

    IDK what I am doing. and only the last choice seem like (to me) something I can reliably count on as working. Currently, my 4 main choices seems to be one of:

    1) A pair of these single port cards

    2x Mellanox ConnectX-3 MCX341 SFP+10Gigabit Ethernet RMDA/RoCE | eBay
    +1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

    = £73.60

    • OR
    2) A pair of these dual port cards

    2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
    +1 cable Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

    = £99.95

    • OR
    3) Dual link is possible? for 20GBe teamed ?

    2x Mellanox ConnectX-3 MCX312 Dual Port 10 Gigabit Ethernet RMDA/RoCE | eBay
    +2 cables Cisco 37-1150-02 SFP-H10GB-ACU10M Direct Attach 10Gb SFP+ 10M Cable | eBay

    = £123.95

    • OR
    4) Pair of AQN-107 modern Aquantia cards (RJ45)

    2x US $76.95 5% OFF|LR LINK 6880BT PCIe x4 Single Port 10G Gigabit Ethernet RJ45 copper network adapter PC Express Controller lan card server Nic-in Network Cards from Computer & Office on AliExpress

    with existing RJ45 cat6 cabling that I already have ( which is 1000mhz, 10gb capable)

    = £129.45 total

    • OR
    5) Is there anything else I should be considering, that is better (=cheaper but still works) ?

    Another question:

    How can I be sure that the SFP+ cable is going to be compatible / work with those cards? It says it’s a used CISCO cable. Therefore should I be buying a MELLANOX branded SFP+ cable instead? Or am I just being paranoid?

    Any help appreciated with links to specific product(s). Either on aliexpress or for UK / ebay. Many thanks.
     
    #1
  2. TRACKER

    TRACKER Member

    Joined:
    Jan 14, 2019
    Messages:
    30
    Likes Received:
    8
  3. dreamcat4

    dreamcat4 New Member

    Joined:
    Nov 17, 2019
    Messages:
    5
    Likes Received:
    0
    Because that is at a very a similar price point (in UK) as either options 2) or 3). What is missing here: is that it's not clear to me at all why that would be better in any way to 2) or 3) given that those ones are 3.0 x4 slot xconnect-3's and without the fan. Which are all plus points for my intended usage. So then what is the possible motivation to get those ones instead?
     
    #3
  4. TRACKER

    TRACKER Member

    Joined:
    Jan 14, 2019
    Messages:
    30
    Likes Received:
    8
    One good reason is UTP cable instead SFP+
    the closest option which is similar to the one, which i've suggested is
    Option #3 - £123.95 - Pair of dual port cards, Dual link 20GBe teamed ?
    compared to suggested by me: Total US $106.40 -> ~£83 (including delivery/taxes), that's 50% less
     
    #4
    Last edited: Nov 18, 2019
    nikalai and dreamcat4 like this.
  5. dreamcat4

    dreamcat4 New Member

    Joined:
    Nov 17, 2019
    Messages:
    5
    Likes Received:
    0
    Awesome! Yes... that would indeed save me a quite a significant amount of money and hassles. Thank you :)
     
    #5
  6. TRACKER

    TRACKER Member

    Joined:
    Jan 14, 2019
    Messages:
    30
    Likes Received:
    8
    :) you are welcome
     
    #6
  7. zer0sum

    zer0sum Active Member

    Joined:
    Mar 8, 2013
    Messages:
    268
    Likes Received:
    85
    If you use ConnectX-3 cards with a good DAC you can do 40 or 56 GbE instead of that measly 10Gbe you're aiming for :)

    upload_2019-11-18_12-4-49.png
     
    #7
    dreamcat4 likes this.
  8. dreamcat4

    dreamcat4 New Member

    Joined:
    Nov 17, 2019
    Messages:
    5
    Likes Received:
    0
    OK so if 40gbe is possible with that option, and assuming that the pci-e 3.0 x4 connection will be capping out at about 3940mb/sec theoretical bandwith... then can you suggest a known good DAC cable. Does it have to match the same brand and be an official Mellanox one? Since the one I linked to isn't, and IDK which one is best.

    Also (and this is actually rather academic for me being limited to x4 slot, but it might matter more for other people who can run pcie 3.0 x8 lanes)... is there anyway to know ahead of time if it's possible to do dual teaming of both links ? Because if not then the single port cards are slightly cheaper and it would avoid wasting extra money on buying a 2nd similar DAC cable if to then only discover its not possible to pair the connections together like that. Thank you both.
     
    #8
  9. zer0sum

    zer0sum Active Member

    Joined:
    Mar 8, 2013
    Messages:
    268
    Likes Received:
    85
    I've used a variety of DAC's from Ebay and Amazon and they've all been fine for me.

    I have my Unraid server dicrectly connected to Windows 10, and my NVME to NVME speeds across the 40GbE link are around 900-1000 MB/s :D

    upload_2019-11-18_12-57-33.png
     
    #9
  10. dreamcat4

    dreamcat4 New Member

    Joined:
    Nov 17, 2019
    Messages:
    5
    Likes Received:
    0
    Hey but shouldn't a 40gbe link be capabable to go any faster than that? For example with 2 ramdisks instead, setup on both ends of the connection.

    Otherwise the max speed seems not much different than 10gbe, isn't it ?
     
    #10
Similar Threads: usual noob
Forum Title Date
Networking 10G fiber noob questions Feb 24, 2018

Share This Page