I've been playing with a couple of ConnectX-3 EN cards. Got a couple of servers set up with Windows Server 2012 R2.
I have large Ramdisks on each server (40GB). For some reason I can get SMB Direct to work. I can get SMB Multichannel to work. But I can't get both to work at the same time for the same transfer. Seems odd. Everything I can find on the web suggests that both should work together. But:
Test 1:
Test 2:
Test 3:
Just to double check, with both servers still in the final configuration, I copied the same large file to a non-server Windows 8 machine that also has a Connect-3 EN card and both links active (no SMBdirect allowed on non-server Windows). In this case I got Multichannel (both links showing traffic).
Seems strange. I've googled around, read pretty much all of Jose's blocks, etc. For whatever reason I get SMBdirect (RDMA) -OR- SMBmultichannel, but never seem to get both.
DBA or others who have played with this: any thoughts?
I have large Ramdisks on each server (40GB). For some reason I can get SMB Direct to work. I can get SMB Multichannel to work. But I can't get both to work at the same time for the same transfer. Seems odd. Everything I can find on the web suggests that both should work together. But:
Test 1:
- Disable both SMBdirect and SMBmultichannel on both servers. On both servers, in powershell:
- Set-NetOffloadGlobalSetting -NetworkDirect Disabled
- Set-SmbServerConfiguration -EnableMultiChannel $false
- Set-SmbClientConfiguration -EnableMultiChannel $false - Copy a large file (30GB BluRay Rip) between ramdisks
- Observe:
- Appx. 400MBs total transfers
- Only one 10Gbe link is showing active traffic @ about 4.1Gbps - Clearly both SMBdirect and SMBmultichannel are not active.
Test 2:
- Start as you left it in Test 1
- Enable SMBmultichannel on both servers. On both servers, in powershell:
- Set-SmbServerConfiguration -EnableMultiChannel $true
- Set-SmbClientConfiguration -EnableMultiChannel $true - Copy a large file (30GB BluRay Rip) between ramdisks
- Observe:
- Appx. 700MBs total transfers
- Both 10Gbe link show active traffic @ about 3Gbps and 4Gbps, respectively. - Clearly SMBmultichannel is now active and having positive effect.
Test 3:
- Start as you left it in Test 2
- Disable SMBmultichannel on both servers. On both servers, in powershell:
- Set-SmbServerConfiguration -EnableMultiChannel $false
- Set-SmbClientConfiguration -EnableMultiChannel $false
[*)Enable SMBdirect on both servers. On both servers, in powershell:
- Set-NetOffloadGlobalSetting -NetworkDirect Enabled - Copy a large file (30GB BluRay Rip) between ramdisks
- Observe:
- Appx. 800MBs total transfers
- Only one 10Gbe link shows active traffic @ about 8Gbps. - Clearly no SMBmultichannel, but speeds are almost doubled the first test so I assume SMBdirect (RDMA) is now active and having positive effect.
Just to double check, with both servers still in the final configuration, I copied the same large file to a non-server Windows 8 machine that also has a Connect-3 EN card and both links active (no SMBdirect allowed on non-server Windows). In this case I got Multichannel (both links showing traffic).
Seems strange. I've googled around, read pretty much all of Jose's blocks, etc. For whatever reason I get SMBdirect (RDMA) -OR- SMBmultichannel, but never seem to get both.
DBA or others who have played with this: any thoughts?