$67 DDR Infiniband on Windows - 1,920MB/S and 43K IOPS

[PigLover]

RDMA for SMB is not supported in Windows 7. While the card might be capable and the driver might be present, the SMB implementation in Win7 won't use it.

Of course, that does not explain why you are not even getting the "non-RDMA" level of performance dba is seeing.

Do you have any way to load Win8/Server2012 - at least temporarily - and see if the behavior changes?

 

[renderfarmer]

Tell me about it. I emailed my contacts at Mellanox and they're all on some sales trip so I haven't bugged them further but I will when they get back on the 15th.

Here's what they suggested:

1. Use our performance tuning guidehttp://www.mellanox.com/related-doc..._Guide_for_Mellanox_Network_Adapters_v1.7.pdf

2. Make sure you have the latest firmware on the adapters (should be 2.11.0500)
3. Update to WinOF 4.2 – should be the latest
4. Make sure you use ntttcp.exe tests to make sure you get an IP test first

I did the first two and I can't get 4.2 installed on Win7 but since you got 2GB/s using win2k8 machines presumably with 3.2 drivers then that's moot imo.

I looked up netttcp and it's some sort of built in windows networking benchmark. I'll give thet a spin as soon as I can but even if it said I was getting a full 5000MB/s it still wouldn't explain why I'm not getting at least the 2000MB/s you got using SMB2.

The only wild card is my optical cable which is a mellanox FDR part and not one of their 40GbE specific units. I contacted mellanox and their rep said that it will work and if it doesn't they'll sort it out for me.

 

[renderfarmer]

RDMA for SMB is not supported in Windows 7. While the card might be capable and the driver might be present, the SMB implementation in Win7 won't use it.

Of course, that does not explain why you are not even getting the "non-RDMA" level of performance dba is seeing.

Do you have any way to load Win8/Server2012 - at least temporarily - and see if the behavior changes?

I might be able to find time to install win2012 on one of my render servers this weekend and put the 40GbE card in that to see if it changes anything.

 

[PigLover]

I might be able to find time to install win2012 on one of my render servers this weekend and put the 40GbE card in that to see if it changes anything.

You need to get both ends done to be sure. If either end is non-RDMA then RDMA cannot be used. I think (but can't confirm) that RDMA was in Server2008 already - its the client side on win7 that is probably holding things back.

Again...it still does not explain why you are not even approaching dba's non-RDMA performance.

 

[dba]

Very good point PigLover. What OS is running on the other end of your setup renderfarmer? Windows 2012 is extremely fast, Windows 2008R2 is not quite a fast, and anything else will be much slower.

You need to get both ends done to be sure. If either end is non-RDMA then RDMA cannot be used. I think (but can't confirm) that RDMA was in Server2008 already - its the client side on win7 that is probably holding things back.

Again...it still does not explain why you are not even approaching dba's non-RDMA performance.

 

[renderfarmer]

Very good point PigLover. What OS is running on the other end of your setup renderfarmer? Windows 2012 is extremely fast, Windows 2008R2 is not quite a fast, and anything else will be much slower.

The most ass-tastic OS this side of Vista; Windows 7 pro... I had win8 installed on my new Dell T7600 for a bit and got BSODs left and right. Dell tech support told me they normally support a new desktop OS on their workstations when the first major update is released so I'll have to wait for Blue. I'm mor elooking forward to trying Ubuntu at this point.

I took one of my render nodes offline and am installing win2012 onto it while i keep working.

 

[dba]

I think that we found your problem: Windows 7. I would not be surprised to hear that your new Windows 2012 render node shows far better performance. In fact, if you have PCIe3 and dual-port cards, 6.4GB/S seems possible.

The most ass-tastic OS this side of Vista; Windows 7 pro... I had win8 installed on my new Dell T7600 for a bit and got BSODs left and right. Dell tech support told me they normally support a new desktop OS on their workstations when the first major update is released so I'll have to wait for Blue. I'm mor elooking forward to trying Ubuntu at this point.

I took one of my render nodes offline and am installing win2012 onto it while i keep working.

 

[renderfarmer]

Well there goes that idea. My render nodes aren't yet Win2012 certified. I got system faults pretty much right after installation...

I'll stick the card into a Win2008R2 machine to see if that might have RDMA and would help performance.

 

[Patrick]

Is Windows Server 2012 installing? If not - here's a tip: Install WS2012 on another machine. Then pop the drive out and put it on the problem node. Have done this probably twice a month on various platforms.

 

[renderfarmer]

Thanks, Patrick. It installs just fine. Once I hit the desktop the fans start spinning up and then the hard hard faults start a minute or so later: WHEA Uncorrectable Error.

I got as far as installing Intel INF chipset drivers and it didn't fix it. I checked the Supermicro site and their OS compatibility matrix lists these boards as yet incompatible with win2012.

 

[renderfarmer]

Win2012 > Win2008R2 yields 1475MB/s. That's the fastest yet, but not even close to what it should be. I'll try it in a windows 8 box next and if that also fails then I'll escalate with Mellanox directly.

 

[dba]

Is there any way that you could test Win2012<-->Win2012 or Win2008R2<-->Win2008R2?

Win2012 > Win2008R2 yields 1475MB/s. That's the fastest yet, but not even close to what it should be. I'll try it in a windows 8 box next and if that also fails then I'll escalate with Mellanox directly.

 

[renderfarmer]

I'll try 2008R2>2008R2 this weekend. I have plenty of those around and that is after all how you tested yours.

I can try sticking 2012 on my workstation instead of win8. It would hold for several hours before BSOD so hopefully it'll be long enough.

 

[dba]

You might try temporarily stripping down the Win2012 workstation to minimize crashes. I see almost zero compatibility problems with non-GPU servers.

I'll try 2008R2>2008R2 this weekend. I have plenty of those around and that is after all how you tested yours.

I can try sticking 2012 on my workstation instead of win8. It would hold for several hours before BSOD so hopefully it'll be long enough.

 

[mrkrad]

windows 2012 can use nvidia cards for remote desktop, i suppose this could complicate the gpu if you are quaddro

 

[renderfarmer]

You might try temporarily stripping down the Win2012 workstation to minimize crashes. I see almost zero compatibility problems with non-GPU servers.

My render nodes are stripped down by necessity. Just a solitary SSD plugged into the onboard SATA3 port for the OS and that's it. I'll try installing it again and see if I can get all of the updates installed. I'll also leave the default chipset drivers.

 

[renderfarmer]

Windows 2008 R2 <=> Windows 2008 R2 yielded only the original 1350MB/s

 

[renderfarmer]

Well, the consistency of the mediocrity of my 40GbE connection is remarkable: Win2012 <=> Win2012 yields the same 1350MB/s.

I confirmed that RDMA was enabled using the Get-SmbServerNetworkInterface as before.

I'm officially stumped and will be taking this back to Mellanox.

For anyone interested, I was able to get Win2012 to run without incident on my render nodes by leaving the default chipset and sata drivers unlike before where I tried feeding it the latest from intel.

 

[renderfarmer]

I'm starting to wonder if the cable is in fact the bottleneck. According to the networking tab in Windows Server Performance Monitor I can't get past ~13.5Gbps and the max throughput of a single Fiber Optic strand in the cable is rated at 14Gbps.

I wanted to buy a 40GbE specific cable as per the Mellanox website but none were available in the channel so Mellanox told me that the 56Gbps FDR cable will work so that's what I bought instead.

 

[renderfarmer]

I ran NTttcp as per my Mellanox contact's advice and managed to get 23Gbps by throwing several threads at the job. I used the following settings:

ntttcps -m 8,0,10.10.10.111 -l 1048576 -n 100000 -w -a 16 -t 10
ntttcpr -m 8,0,10.10.10.111 -l 1048576 -rb 2097152 -n 100000 -w -a 16 -fr -t 10


On a single thread I get 15Gbps. On two threads I get 20Gbps.

dba; If you have time and the inclination I'd really be interested in knowing what results you get using similar settings on NTttcp with your QDR IB or DDR IB cards.