10GB Performance issues

push3r

Member
Feb 19, 2015
52
11
8
53
@alex1002, I was just kidding man. :) But if you really want to get into 40GbE world, create another topic in the Networking forum and I am sure you will get a lot of replies from the experts here.
 

PigLover

Moderator
Jan 26, 2011
3,106
1,434
113
ConnectX 3 EN cards are 10Gbe and support RDMA over Ethernet (RoCE).

Note, however that Windows only supports RDMA in their server OS. They do not permit RDMA on consumer versions (Win8/Win10). As long as you are running 2012R2 or 2016 on both ends it works great.
 
  • Like
Reactions: push3r

PigLover

Moderator
Jan 26, 2011
3,106
1,434
113
They actually work quite well on servers with only PCIe 2.0. PCIe is backward compatible and you don't need the extra bandwidth for the EN card to support 2x10Gbe. You need it to support 2x56GB FDR on the VPI card

Actually, you can't quite get to 2x56Gbe even with PCIe 3.0 x8 - but you get close. Most applications are using the second port for redundancy/failover anyway so you never really expect 2x56G of throughput.
 
  • Like
Reactions: push3r

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,384
1,823
113
CA
2 port ~$39-49 "NIB" / x2-VPI (PCIE2)
2 port ~$99-$199 "USED" / x3-VPI (PCIE3)

That's about what I've been paying for them on ebay recently. Working on deploying first IB stuff :)
 

alex1002

Member
Apr 9, 2013
519
19
18
2 port ~$39-49 "NIB" / x2-VPI (PCIE2)
2 port ~$99-$199 "USED" / x3-VPI (PCIE3)

That's about what I've been paying for them on ebay recently. Working on deploying first IB stuff :)
What switch YOu been using them on?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,384
1,823
113
CA
Voltaire® Grid Director™ 4036. QDR (40Gb/s)

That's the plan at-least :) cables, cards, etc have been trickling in from around the world.
 

push3r

Member
Feb 19, 2015
52
11
8
53
They actually work quite well on servers with only PCIe 2.0. PCIe is backward compatible and you don't need the extra bandwidth for the EN card to support 2x10Gbe. You need it to support 2x56GB FDR on the VPI card

Actually, you can't quite get to 2x56Gbe even with PCIe 3.0 x8 - but you get close. Most applications are using the second port for redundancy/failover anyway so you never really expect 2x56G of throughput.
Thank you for passing on the knowledge.
 

push3r

Member
Feb 19, 2015
52
11
8
53
@alex1002, what the hell are you doing in your lab that requires 40GbE? :)

If it was me, I would focus on what techno I want to learn / test and the minimum hardware that would allows me to do so.
 

alex1002

Member
Apr 9, 2013
519
19
18
@alex1002, what the hell are you doing in your lab that requires 40GbE? :)

If it was me, I would focus on what techno I want to learn / test and the minimum hardware that would allows me to do so.
I never mention 40gbe. That was suggested. But 10gbe would be nice makes everything faster. I am not a patient man and now with ssds....
 

push3r

Member
Feb 19, 2015
52
11
8
53
You just bought some 10GbE gear and what are you going to do with them if you go to 40GbE? You can send them to me. :)

For sure, if you want to do 40GbE, it's plenty affordable. Do it! Maybe sell the gear you just got?
 

alex1002

Member
Apr 9, 2013
519
19
18
Can anyone suggest a 10gbe switch. I am going to begin looking for some? Image all my servers on 10gbe happiness :))
 

Aluminum

Active Member
Sep 7, 2012
431
46
28
10GbE SFP+ often costs more/port than 40Gb QDR IB, and 10Base-T sucks on many levels.

I scored a 12 port managed FDR switch (56Gb SX6012) for $400, just need to get around to making the 140mm fan shroud so it doesn't sound like a 747 on takeoff ;)

For super cheapskates doing 3 peer-to-peer links with dual port QDR cards is really cheap, maybe as low as $100 w/ twinax cables these days if you are good at best offers and sniping.
 

Baddreams

New Member
Mar 15, 2011
21
7
3
Mellanox ConnectX-2 is super cheap. ~ $15 - $20 bucks. But it's a bit dated and doesn't have Remote Direct Memory Access (RDMA). Also, when buying these cards, be sure you get the correct profile for your available slots on your server chassis. Low profile or regular profile. Finding the low profile or regular profile brackets for them is nearly impossible.
Just an FYI, if you email Mellanox they will likely send you some brackets for free, assuming they still have them.
 
  • Like
Reactions: push3r