WTB 40GigE cards + QSFP and cable

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
Hi Guys,

I'm new to 40GigE but I want to buy 2x 40GigE cards and then a cable and SFPs to connect two servers directly to each other for backup of 50TB of media rich content.

Do you guys know if these two cards would work together or if I need more products. The seller of the cable says that the cable has the QSFPs on the cable and from the pictures it looks like he does. But I want to make sure. So if I buy the two links below, will that connect two servers together, or will I need another product?

MCX353A-FCBT FUJITSU CONNECTX3 VPI ADAPTER CARD 1 PORT QSFP 40GBE PCI-E - LP | eBay
QSFP 40G QSFP+ Active Optical Cable 15 meter | eBay
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
What is the distance? Do you really need 15m?

And you link is expensive for a single port card.

And what target OS? And Infiniband or Ethernet? The latter I assume?
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
Ethernet on 40GigE for windows server 2016. Really the card is expensive? Most of the cards I've found are over 500$ new.

Oh, and also yes 3m would be fine in length, but seems like the 15m cable is a great price.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
Why are the ATTO cards 40gigE over$1000 but these HP used is 40$?

Order - ATTO Store
A couple of reasons. First, ATTO is probably not producing cards at the volume of HP, so the price will be higher on that alone. Plus, the ATTO card is probably new.

The HP card seems like a nice, well-tested, but removed from service part--probably from servers that were bought dirt cheap and by the pallet and then parted out. That's the only way I've figured some of the unreal prices that exist on ebay pop up on legit stuff. The fake stuff, well, those prices are low for a reason...
 

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
The HP cards are re-branded Mellanox cards. I have dozens of them and lots of people on STH have used them without problems.
The used ones on eBay are typically from huge data center deployments that are refreshing their hardware and going over to 25/50/100G+ networking from last-generation 10/40G equipment.

I recently bought a dozen of the cards from Westporter that @jmck linked above, they are working fine.
 

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
I only use the Ethernet side (both 10G and 40G), as far as I know they do both 10 and 40 IB as well. They are the same hardware as the MCX354A-FCBT cards I got in a previous batch.
The HP ones listed above have a short bracket, there are versions out there with full height for the same price point as well.

They are discussed in the forums often as well - like https://forums.servethehome.com/ind...81-b21-656089-001-to-mellanox-firmware.13893/ The ones I installed last weekend I am using with the passive QSFP -> SFP+ adapters (https://forums.servethehome.com/ind...ic-with-a-10gbe-switch-sfp.12406/#post-119153) and then Arista SFP+ DAC's to connect to a 10G switch before I get another 40G switch installed there.

I have used them with Quanta 40G switches as well, but had to try a few different DAC cables to get the link up.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
Enterprises upgraded to faster network, "sold" the old nics to recyclers.
Fixed it for you. :D It's unreal sometimes how cheap the recyclers get stuff. But apparently it's not worth the time and effort for the equipment to be resold directly from the first owners, even with a much higher return.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
Thanks guys,

I got the 40Ge cards and have been doing some testing.

I've tried 40G ethernet and 40G InfiniBand.

IB is faster than ethernet and seems to handle multipule threads better. However, we did notice that with ethernet and jumbo frames turned on, we actually improve single thread transfer speeds if pulling multipule threads at the same time.

For example we tested 40Gig Ethernet 1500MTU and got around 1000MBPS, then with two threads we got around 700MBps on each thread for a total of around 1400MBps.

With Jumbo frames we got around 1200MBPS on a single thread, but when we did two threads we got 1500 per thread for a total of 3000MBps.

Normally when we pull more than one thread it will slow down the first thread, but with ethernet and Jumbo frames it actually speeds it up.

We maxed out at 2.2GBps from NVMe to NVMe drives. Each NVMe array can produce over 6,000MBps so the bottle neck is the 40gE. Maybe we should try a different cable. We have ordered some test ATTO cards to see if those increase performance or not. I've also heard of cables which are like 12x10gE inside one cable vs the 4x10gE that I am using now.

Anyways, thanks for reading.
 

Samir

Post Liker and Deal Hunter Extraordinaire!
Jul 21, 2017
3,257
1,447
113
49
HSV and SFO
At first I was thinking you were only getting 1ge speeds until I checked the case of the 'M'. Impressive that you're hitting 2.2GBps over a network. Even at that speed, the 50,000GB would only take a little over 6hrs. Are you looking for an even shorter copy time? The max theoretical is still going to be slower than the max speed of the NVMe array.

Have you tried iperf tests to see what type of bandwidth you're getting through there? This way you could see if the full 40gb bandwidth is there or not.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Same questions as in your other thread - what optimization have you done at NIC level?
I think some CX3 had issues with TCP Offload and Packetloss (not sure whether that was only under ESX) so it might be worth trying turning that *off* too
upload_2018-4-26_7-38-2.png
 

fossxplorer

Active Member
Mar 17, 2016
554
97
28
Oslo, Norway
So these can do 40GbE? My impresison is that the versions with 40Gbps IB can only do 10GbE.
The once that can do 40GbE probably have RoCE support i guess. (for the applications that can take advantage of that)

The HP cards are re-branded Mellanox cards. I have dozens of them and lots of people on STH have used them without problems.
The used ones on eBay are typically from huge data center deployments that are refreshing their hardware and going over to 25/50/100G+ networking from last-generation 10/40G equipment.

I recently bought a dozen of the cards from Westporter that @jmck linked above, they are working fine.
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
So these can do 40GbE? My impresison is that the versions with 40Gbps IB can only do 10GbE.
Mellanox uses the same chip desing for fdr/qdr/56gbe/40gbe/10gbe cards, the firmware defines what you can use. Some cards have qdr + 40gbe or qdr + 10gbe.
It should be possible to flash a fdr + 40/56gbe firmware and uses these speeds.
The once that can do 40GbE probably have RoCE support i guess. (for the applications that can take advantage of that)
All connect-x3 (and newer) support roce v1, the pro card support the ip based roce v2.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
@zerogravity Samsung Pro 1TBs.

We just got the ATTO cards in. They are a bit faster using jumbo frames, getting about 1800MBps vs HP's 1000-1500MBps. The ATTO cards do not have any IB so it's purely ethernet.

Still maxing out around the same values. I did notice that the Atto cards come with QSFTP but no cable. However I can't seem to find any cable that comes without a QSFP. So I'm at a bit of a loss.

Do the active optical cables for 40gigE come off the QSFP?