PCIe 2.0/3.0 4x Lane 10Gb SFP+ card?

[lunadesign]

It seems clear that a PCI-E 2.0 x4 (2000 MB/s) will handle a 10G 1 port card (1250 MB/s) but I'm curious what would happen with a 2 port card (2500 MB/s).

1) Would I be able to hit 10G on each port individually (assuming the other port is not busy at the same time)? Or does the card detect the PCI-E bandwidth limitation and cap one or both ports?

2) Also keeping in mind that the PCI-E speed is 2000 MB/s in each direction, does that mean that I could theoretically have one port sending at 10G at the same time the other one is receiving at 10G?

 

[Docop]

I guess a Mellanox 2 dual nic into a pcie3.0 x1 will most likely run at 4gb each. But those mellanox are pcie2x8 card... so anyone had tested ??
A pci2x4 into a pci3x1 should give 8gb on 1 port.

 

[Jepong]

I guess a Mellanox 2 dual nic into a pcie3.0 x1 will most likely run at 4gb each. But those mellanox are pcie2x8 card... so anyone had tested ??
A pci2x4 into a pci3x1 should give 8gb on 1 port.

A ConnectX-2 card runs at pcie 2.0 and will not run at pcie 3.0 even if you put it in a 3.0 slot. It will also run at x1 only if the slot is x1 electrical. So putting a pcie 2.0 x4 or x8 card in a pcie 3.0 x1 slot will run at pcie 2.0 x1. Severely limiting the bandwidth to just 500 MB/s. Someone correct me if I'm wrong

 

[i386]

A ConnectX-2 card runs at pcie 2.0 and will not run at pcie 3.0 even if you put it in a 3.0 slot. It will also run at x1 only if the slot is x1 electrical. So putting a pcie 2.0 x4 or x8 card in a pcie 3.0 x1 slot will run at pcie 2.0 x1. Severely limiting the bandwidth to just 500 MB/s. Someone correct me if I'm wrong

That's correct.

 

[Jepong]

I would like to put a Mellanox MNPH29C-XTR ConnectX-2 Dual Port into a x4 slot. Will the slot become a bottleneck? Considering 2x 1250MB/s is greater than 2000MB/s (pci-e 2.0 x4). ConnectX-2 is cheaper than ConnectX-3 and I'm buying multiples. I just realized I've got a similar question with lunadesign but since it has not been answered I'm asking again. I'll be putting this in a vSan host

 

[i386]

Will the slot become a bottleneck?

When your workload can saturate 2x 10gbit links, then yes.

 

[Jepong]

Thanks for answering! Let me repeat lunadesign's questions.

1) Would I be able to hit 10G on each port individually (assuming the other port is not busy at the same time)? Or does the card detect the PCI-E bandwidth limitation and cap one or both ports?

2) Also keeping in mind that the PCI-E speed is 2000 MB/s in each direction, does that mean that I could theoretically have one port sending at 10G at the same time the other one is receiving at 10G?

 

[fohdeesha]

Yes you'll be fine

 

[Jepong]

How about a single port connectx-3 in a 3.0 x1 slot?

 

[i386]

~985mbyte/s throughput, not really 10gbit/s but very close

 

[Jepong]

Thats what i thought . 10gigabit is 1250mbyte/s so theres a little bottleneck. 985mbyte/s translates to about less than 8gigabit/s.

 

[zack$]

Is anyone able to confirm whether the Solarflare SFN6122F will work in a pcie 2.0 x4 (physical x8)? I have read the thread and am pretty certain it should, but the sfuserguide v4.12.0 is throwing me off with the following: "8 lanes Gen2 (5.0GT/s), x8 edge connector (usable in x8 and x16 slots), 127 SR‐IOV virtual functions per port."

Is the "usable" here electric or physical?

 

[DanielWood]

Is anyone able to confirm whether the Solarflare SFN6122F will work in a pcie 2.0 x4 (physical x8)? I have read the thread and am pretty certain it should, but the sfuserguide v4.12.0 is throwing me off with the following: "8 lanes Gen2 (5.0GT/s), x8 edge connector (usable in x8 and x16 slots), 127 SR‐IOV virtual functions per port."

Is the "usable" here electric or physical?

Physical. I am unaware of ANY PCIe card that will not downgrade all the way to electrical x1. In fact, that is what is often used for mining rigs. Most of the USB3 cable connected mining risers are PCIe x1. The video card runs fine on those (assuming you have supplied it with 75W capability on the PCIe slot), it is just PCIe bandwidth limited (which is largely irrelevant for mining applications).

The question becomes where is the bottleneck. PCIe 2.0 x4 at 2GB/sec, that is a bottleneck for 2x 10G but will not bottleneck unless you are completely saturating both. PCIe 3.0 x1 is a bottleneck of around 8Gbps no matter what you have on the other side. As long as you are aware and OK with these limitations, go wild with any combo of PCIe slots and cards.

I regularly use a PCIe x1 to x16 extender (x1 to the motherboard, x16 to the card) for flashing NICs and HBAs. The only issues I've had were when I forgot to connect the external power, the Dual 40GB NIC downgraded to PCIe 2.0. As soon as I gave it that 12V boost it needed, PCIe 3.0 x1 worked with no issue.

 

[newyork10023]

The Intel (and third-party) X550-T2 using a PCIe gen3 x4 slot is available. Still unable to find a DA2 with x4 slot, Intel or otherwise.

 

[chelmi]

I was wondering too why there would be T2 (10G RJ45) cards available in PCIe gen3 x4 available and none as DA2 (SFP+)

Now however, both Mellanox ConnectX-3 and Intel X710-DA2 are available as PCIe gen3 8x.

If my math and logic is not wrong, a PCIe gen3 8x card in a 4x slot will still run at PCIe gen3 4x electrical, hence 3938 MB/s.
As 2x10Gbe would max out at 2x 1250MB/s = 2500MB/s, 3938 MB/s should be plenty of bandwidth to let both ports run at full speed.

Is anything in my logic flawed ?

By the way, I got an Intel X520-DA2 PCIe gen2 8x plugged into a PCIe gen3 4x port.
While this works, I am still only getting 6.6 Gbit/s throughput on this one on one port which is roughly 825 MB/s.
This would not be even close to PCIe gen2 4x which should give me 2000 MB/s. It almost looks like the card would fallback to PCIe gen1 4x (or PCIe gen3 1x) for lack of a gen2 electrical port.

So before I go ahead and buy yet another card that may or may not work at full speed I would love someone confirms my above logic and math is in theory correct.

 

[i386]

I was wondering too why there would be T2 (10G RJ45) cards available in PCIe gen3 x4 available and none as DA2 (SFP+)

Now however, both Mellanox ConnectX-3 and Intel X710-DA2 are available as PCIe gen3 8x.

MY guess: 10GBE SFP+ based cards share the design with QSFP+ based cards, while 10GBaseT doesn't have faster variants that require a different pcb design.

 

[Tommy Blom]

Have à look at Mellanox ConnectX-3 MCX311A-XCAT EN 10G Ethernet SFP+
1 port pcie 3.0 x4.
Works well in Linux FreeBSD and Windows

 

[chelmi]

Have à look at Mellanox ConnectX-3 MCX311A-XCAT EN 10G Ethernet SFP+
1 port pcie 3.0 x4.

Thanks for the hint Tommy Blom, however we were discussing any potential dual port cards.
1 port is quite easy to find in PCIe 3.0 4x, but dual port for some reason is not.