PCIe 2.0/3.0 4x Lane 10Gb SFP+ card?

Discussion in 'Networking' started by eroji, May 16, 2016.

  1. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    I think it will run at pcie 2.0 x1
     
    #21
  2. lunadesign

    lunadesign Member

    Joined:
    Aug 7, 2013
    Messages:
    89
    Likes Received:
    4
    It seems clear that a PCI-E 2.0 x4 (2000 MB/s) will handle a 10G 1 port card (1250 MB/s) but I'm curious what would happen with a 2 port card (2500 MB/s).

    1) Would I be able to hit 10G on each port individually (assuming the other port is not busy at the same time)? Or does the card detect the PCI-E bandwidth limitation and cap one or both ports?

    2) Also keeping in mind that the PCI-E speed is 2000 MB/s in each direction, does that mean that I could theoretically have one port sending at 10G at the same time the other one is receiving at 10G?
     
    #22
  3. Docop

    Docop New Member

    Joined:
    Jul 19, 2016
    Messages:
    4
    Likes Received:
    0
    I guess a Mellanox 2 dual nic into a pcie3.0 x1 will most likely run at 4gb each. But those mellanox are pcie2x8 card... so anyone had tested ??
    A pci2x4 into a pci3x1 should give 8gb on 1 port.
     
    #23
  4. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    A ConnectX-2 card runs at pcie 2.0 and will not run at pcie 3.0 even if you put it in a 3.0 slot. It will also run at x1 only if the slot is x1 electrical. So putting a pcie 2.0 x4 or x8 card in a pcie 3.0 x1 slot will run at pcie 2.0 x1. Severely limiting the bandwidth to just 500 MB/s. Someone correct me if I'm wrong
     
    #24
  5. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,533
    Likes Received:
    355
    That's correct.
     
    #25
    DanielWood, fohdeesha and Jepong like this.
  6. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    I would like to put a Mellanox MNPH29C-XTR ConnectX-2 Dual Port into a x4 slot. Will the slot become a bottleneck? Considering 2x 1250MB/s is greater than 2000MB/s (pci-e 2.0 x4). ConnectX-2 is cheaper than ConnectX-3 and I'm buying multiples. I just realized I've got a similar question with lunadesign but since it has not been answered I'm asking again. I'll be putting this in a vSan host
     
    #26
    Last edited: Nov 29, 2018
  7. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,533
    Likes Received:
    355
    When your workload can saturate 2x 10gbit links, then yes.
     
    #27
    Jepong likes this.
  8. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    Thanks for answering! Let me repeat lunadesign's questions.

    1) Would I be able to hit 10G on each port individually (assuming the other port is not busy at the same time)? Or does the card detect the PCI-E bandwidth limitation and cap one or both ports?

    2) Also keeping in mind that the PCI-E speed is 2000 MB/s in each direction, does that mean that I could theoretically have one port sending at 10G at the same time the other one is receiving at 10G?
     
    #28
  9. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,045
    Likes Received:
    757
    Yes you'll be fine
     
    #29
    Jepong likes this.
  10. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    How about a single port connectx-3 in a 3.0 x1 slot?
     
    #30
  11. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,533
    Likes Received:
    355
    ~985mbyte/s throughput, not really 10gbit/s but very close
     
    #31
  12. Jepong

    Jepong Member

    Joined:
    Nov 2, 2017
    Messages:
    39
    Likes Received:
    0
    Thats what i thought . 10gigabit is 1250mbyte/s so theres a little bottleneck. 985mbyte/s translates to about less than 8gigabit/s.
     
    #32
    Last edited: Jan 9, 2019
  13. zack$

    zack$ Member

    Joined:
    Aug 16, 2018
    Messages:
    80
    Likes Received:
    26
    Is anyone able to confirm whether the Solarflare SFN6122F will work in a pcie 2.0 x4 (physical x8)? I have read the thread and am pretty certain it should, but the sfuserguide v4.12.0 is throwing me off with the following: "8 lanes Gen2 (5.0GT/s), x8 edge connector (usable in x8 and x16 slots), 127 SR‐IOV virtual functions per port."

    Is the "usable" here electric or physical?
     
    #33
  14. DanielWood

    DanielWood New Member

    Joined:
    Sep 14, 2018
    Messages:
    26
    Likes Received:
    9

    Physical. I am unaware of ANY PCIe card that will not downgrade all the way to electrical x1. In fact, that is what is often used for mining rigs. Most of the USB3 cable connected mining risers are PCIe x1. The video card runs fine on those (assuming you have supplied it with 75W capability on the PCIe slot), it is just PCIe bandwidth limited (which is largely irrelevant for mining applications).

    The question becomes where is the bottleneck. PCIe 2.0 x4 at 2GB/sec, that is a bottleneck for 2x 10G but will not bottleneck unless you are completely saturating both. PCIe 3.0 x1 is a bottleneck of around 8Gbps no matter what you have on the other side. As long as you are aware and OK with these limitations, go wild with any combo of PCIe slots and cards.

    I regularly use a PCIe x1 to x16 extender (x1 to the motherboard, x16 to the card) for flashing NICs and HBAs. The only issues I've had were when I forgot to connect the external power, the Dual 40GB NIC downgraded to PCIe 2.0. As soon as I gave it that 12V boost it needed, PCIe 3.0 x1 worked with no issue.
     
    #34
    zack$ likes this.
Similar Threads: PCIe 20/30
Forum Title Date
Networking X550-T1 in PCIe 2.0 x4: Does it run at 10 GbE? Aug 22, 2018
Networking PCIe Hot Swap or Interconnect Servers Jul 9, 2018
Networking Cheap place to pick up quad port nics (1 pcie, 1 pcix) Jun 4, 2018
Networking Infiniband PCIe card preventing boot in one server but not another May 12, 2018
Networking Advice: Myricom 10G-PCIE-8AL-S Network Adapter(s) Apr 16, 2018

Share This Page