$67 DDR Infiniband on Windows - 1,920MB/S and 43K IOPS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
It's not critical, but your memory bandwidth looks low for dual E5s. Perhaps you don't have DIMMS in all of the memory channels?
Both of those machines have 8 RDIMMs and read as quad channel with the proper speed (1600MHz for my Dell and 1333MHz for the Supermicro).

More importantly, was that 2,600MB/s for one RAM disk?
Yes, for one disk.

If so then that looks OK, so try spinning up three more RAM disks on the same machine: Just use the StarWind UI to "Add Device" three more times. Try four 4096MB RAM disks and set IOMeter to 8,000,000 sectors.

Don't RAID the RAM disks - test them as four separate disks in IOMeter. You can multi-select drives in IOMeter by control-clicking.
That's the first thing I tried in Iometer. Ctrl+clicked all 4 and got the same reduced 1780MB/s result as when I striped them.

i'm just having the worst luck lately. The more money i spend, the more disappointed I get.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
My lowly c6100 test node shows >7GB/S with four RAM disks, and it's using two old L5520 CPUs with 6x8GB 1333MHz ECC dual-rank DIMMS. You definitely have a problem somewhere, and it's turning most of the money you spent into waste. Strip one of those machines down to the bone and benchmark it frequently as you build it back up. You have some seriously powerful equipment there and I'd hate to see it underutilized.

Both of those machines have 8 RDIMMs and read as quad channel with the proper speed (1600MHz for my Dell and 1333MHz for the Supermicro).



Yes, for one disk.



That's the first thing I tried in Iometer. Ctrl+clicked all 4 and got the same reduced 1780MB/s result as when I striped them.

i'm just having the worst luck lately. The more money i spend, the more disappointed I get.
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
My lowly c6100 test node shows >7GB/S with four RAM disks, and it's using two old L5520 CPUs with 6x8GB 1333MHz ECC dual-rank DIMMS. You definitely have a problem somewhere, and it's turning most of the money you spent into waste. Strip one of those machines down to the bone and benchmark it frequently as you build it back up. You have some seriously powerful equipment there and I'd hate to see it underutilized.

Thanks. But in trying to find a baseline for my memory bandwidth results using Sandra 2013 I found their rankings page:

Top Memory Bandwidth Ranks : SiSoftware Official Live Ranker

You have to go down several pages to find a lowly dual socket system with only 8 sticks of un-overclocked RDIMMs like mine but my systems seem to be right in line with what's there.

I'll try Iometer again with the RAM disks...
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
Are you using Win2008 or Win2012? Per my testing, 2008 rocks with the ConnectX-2 cards while 2012 falls far behind.

Update: After creating a custom RDMA-enabled firmware for my Dell ConnectX-2 card, Windows 2012 is performing extremely well.

Thanks. But in trying to find a baseline for my memory bandwidth results using Sandra 2013 I found their rankings page:

Top Memory Bandwidth Ranks : SiSoftware Official Live Ranker

You have to go down several pages to find a lowly dual socket system with only 8 sticks of un-overclocked RDIMMs like mine but my systems seem to be right in line with what's there.

I'll try Iometer again with the RAM disks...
 
Last edited:

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Are you using Win2008 or Win2012? Per my testing, 2008 rocks with the ConnectX-2 cards while 2012 falls far behind.
I had 2012 until a few weeks ago and downgraded to 2008R2 for that reason. Too many issues with ConnectX and other stuff relating to my job.

Ok, so I don't know if this was cheating or just an obvious oversight on the part of el n00b (that's me) but I just added 3 workers to the one worker I was using previously for the single disk tests and assigned each worker one of the 4 RAM Disks. Using 1MB reads with 16 I/Os I got 8045MB/s

If it's valid, then do I now just share each one, map each drive and repeat the test over the network?
 

wuffers

New Member
Dec 24, 2012
19
0
0
I had 2012 until a few weeks ago and downgraded to 2008R2 for that reason. Too many issues with ConnectX and other stuff relating to my job.

Ok, so I don't know if this was cheating or just an obvious oversight on the part of el n00b (that's me) but I just added 3 workers to the one worker I was using previously for the single disk tests and assigned each worker one of the 4 RAM Disks. Using 1MB reads with 16 I/Os I got 8045MB/s

If it's valid, then do I now just share each one, map each drive and repeat the test over the network?
What kind of issues on Windows 2012? I'm about to have a new Hyper-V 2012 cluster set up and I have ConnectX-2 HBAs..
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
What kind of issues on Windows 2012? I'm about to have a new Hyper-V 2012 cluster set up and I have ConnectX-2 HBAs..
In my case it was with connectX-1 adapters and I couldn't get them to connect properly. Same adapter worked fine in Windows 8 and Wind 2008 R2 but in Win2012 not so much. In the end the main reason I switched back is because I use my file server as a license manager and not all of the companie's who's software I use have updated to a version of Reprise that Win2012 supports.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
Using four workers is perfectly valid since you are testing maximum throughput, not throughput for a particular use case. I am surprised that with Intel processors you were unable to go above 2,600MB/S with one worker, but in the end it's not important - go ahead and test across the network with your four-disk, four worker setup.

If you have time, test both throughput and IOPS - I really wonder how 40Gbit Ethernet does compared to IPoIB.

I had 2012 until a few weeks ago and downgraded to 2008R2 for that reason. Too many issues with ConnectX and other stuff relating to my job.

Ok, so I don't know if this was cheating or just an obvious oversight on the part of el n00b (that's me) but I just added 3 workers to the one worker I was using previously for the single disk tests and assigned each worker one of the 4 RAM Disks. Using 1MB reads with 16 I/Os I got 8045MB/s

If it's valid, then do I now just share each one, map each drive and repeat the test over the network?
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Using four workers is perfectly valid since you are testing maximum throughput, not throughput for a particular use case. I am surprised that with Intel processors you were unable to go above 2,600MB/S with one worker, but in the end it's not important - go ahead and test across the network with your four-disk, four worker setup.

If you have time, test both throughput and IOPS - I really wonder how 40Gbit Ethernet does compared to IPoIB.
Holy crap! I just tried Iometer on a single RAM disk (local) with 32 workers and got 21,278MB/s!! @4K IOPS = 279K, CPU%=99.8, Av Resp Time 1.7ms.

Over the network with the same test I got Max bandwidth ~1200MB/s, @4Kb IOPS = 66K, CPU%=99.8.
 
Last edited:

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Hey, dba, someone on the Mellanox forums pointed out that since I'm not connected to a switch I might be getting slow down from collisions. Do you think you could try the starwind/iometer test on two direct connected machines (bypassing the switch)?

I've re-installed windows 2012 on my file server to see if that made any difference. TheW2012 drivers automatically detect if there are firmware updates to your card and applies it if you let it - very cool feature. The firmware update and OS dind't make a difference so far; still around 1350MB/s. The W2012 mellanox drivers have many more tuning options to try though.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
renderfarmer,

I have tested both direct-connect and switch configurations for throughput and IOPS. The results were indistinguishable.


Hey, dba, someone on the Mellanox forums pointed out that since I'm not connected to a switch I might be getting slow down from collisions. Do you think you could try the starwind/iometer test on two direct connected machines (bypassing the switch)?

I've re-installed windows 2012 on my file server to see if that made any difference. TheW2012 drivers automatically detect if there are firmware updates to your card and applies it if you let it - very cool feature. The firmware update and OS dind't make a difference so far; still around 1350MB/s. The W2012 mellanox drivers have many more tuning options to try though.
 
Last edited:

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Two nodes connected directly can't generate collisions...node A transmits on Bs receive and B transmits on As receive. No collisions possible because there is only a singe talker on each wire.
 

wuffers

New Member
Dec 24, 2012
19
0
0
Performance issues on Win2012 is disappointing to hear. I'll have to test it out myself when the SAN is up. Unfortunately I need Hyper-V 2012 for my deployment..

If it works under 2008 then it should just be a driver update that fixes things..
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
True - and in fact Infiniband is connection-oriented and does not have collisions, period.
are those voltaire switches store and forward or cut-through? or what? hub?

Sounds like some sort of cut-through switch.

If you dump 40gb down a 10gb link, what happens? flow control? dropped packets? Deep buffer (latency)?
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
True - and in fact Infiniband is connection-oriented and does not have collisions, period.
Hi dba. Got your post on the Mellanox forums. Here's my Powershell dump:

PS C:\Users\Administrator> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 18 True True 40 Gbps 10.10.10.100
* 18 True True 40 Gbps fe80::dd86:1d61:...
* 13 True False 1 Gbps 10.10.10.199
* 13 True False 1 Gbps fe80::114d:f17d:...

Is that good or bad?

I would normally have said that this is an SMB2 limitation (my Workstation is on Win7) but since you got 2GB/s on win2k8 that excuse won't fly.

I'm in the middle of a project that keeps expanding in scope so I haven't had much time to fiddle with this lately. I'm getting more and more interested in trying linux on both the server and client side with NFS or even pNFS if I can figure out how what the heck is involved in getting that installed in Ubuntu or CentOS.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
You have a true mystery on your hands. Your cards are ultra-fast 40Gbit, they show up as RDMA capable (as shown in your output below), and yet you are getting disappointing throughput. Since you bought new cards, has Mellanox been any help?

Hi dba. Got your post on the Mellanox forums. Here's my Powershell dump:

PS C:\Users\Administrator> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 18 True True 40 Gbps 10.10.10.100
* 18 True True 40 Gbps fe80::dd86:1d61:...
* 13 True False 1 Gbps 10.10.10.199
* 13 True False 1 Gbps fe80::114d:f17d:...

Is that good or bad?

I would normally have said that this is an SMB2 limitation (my Workstation is on Win7) but since you got 2GB/s on win2k8 that excuse won't fly.

I'm in the middle of a project that keeps expanding in scope so I haven't had much time to fiddle with this lately. I'm getting more and more interested in trying linux on both the server and client side with NFS or even pNFS if I can figure out how what the heck is involved in getting that installed in Ubuntu or CentOS.