iSCSI SAN With Windows Server 2016

[Nader]

Hi Everyone, hope everything is going well on your side of the fence. I am new here and in need of help getting RDMA to work on my little storage/SAN project on Windows Server 2016.

I bought 2 of these Mellanox MHQH29C-XTR cards on Ebay
I have updated the firmware to 2.9.8350
My client is Windows 10 and my server is windows 2016

Both PCs are directly attached with a HP/Mellanox Infiniband QSFP to QSFP cables bought on Ebay. The transfer rate is very disappointing because I can't seem to get RDMA to work.

Max transfer rate I am getting is 240MBps using a ramdisk. I have used similar blog posts to try to get RDMA to work but I am failing miserably. I have used OFED version 4.6 and 5.25, disable Eset on both machines and I still got the same results.

I am barely getting 2Gbps out of the 32Gbps that these cards are rated at. I am new to this InfiniBand stuff but I never thought it would be so easy to fail at this.

If someone can shed some light, any suggestions or help will be greatly appreciated.

Thanks.

 

[i386]

What's the output running these commands in powershell?

Get-NetOffloadGlobalSetting
Get-NetAdapterRdma
Get-SmbServerNetworkInterface

 

[Nader]

Hi i386, thanks for helping. I have only one DAC connecting both PCs, that's why one of the interfaces has an IPV6 address.



PowerCLI C:\> Get-NetOffloadGlobalSetting


ReceiveSideScaling : Enabled
ReceiveSegmentCoalescing : Enabled
Chimney : Disabled
TaskOffload : Enabled
NetworkDirect : Enabled
NetworkDirectAcrossIPSubnets : Blocked
PacketCoalescingFilter : Disabled




PowerCLI C:\> Get-NetAdapterRdma

Name InterfaceDescription Enabled
---- -------------------- -------
RDMA01 Mellanox ConnectX-2 IPoIB Adapter #2 True
RDMA02 Mellanox ConnectX-2 IPoIB Adapter True



PowerCLI C:\> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 12 True True 32 Gbps 192.168.2.10
* 12 True True 32 Gbps fe80::9085:c9c:7174:2913
* 2 False False 1 Gbps 192.168.1.100

 

[jwc]

Please let me know if this is any assistance to you Deploying Windows Server 2012 with SMB Direct (SMB over RDMA) and the Mellanox ConnectX-2/ConnectX-3 using InfiniBand – Step by Step

Also, just out of curiosity, why are you trying iSCSI and not SMB 3.0 direct?

 

[Nader]

Hi jwc, thanks for the help. That is the same blog post I followed and could not get RDMA to work.

Also, just out of curiosity, why are you trying iSCSI and not SMB 3.0 direct?

Can you explain the part about using SMB 3.0 Direct?

Honestly, I don't have to use iSCSI. I can just create shares for most of the stuff that I am doing.

This storage server will be for file/media sharing and my virtualization lab.

 

[randalldeflagg]

do you have iSCSI bound directly to the Mellanox cards?

Server Manager > File and Storage Services
Right click on the server name under the list of servers
Select iSCSI target settings
Uncheck any network that is not tied to the Mellanox cards. Try the benchmark again

 

[i386]

I learned that Windows Client os don't support rdma.

 

[frogtech]

Hi jwc, thanks for the help. That is the same blog post I followed and could not get RDMA to work.



Can you explain the part about using SMB 3.0 Direct?

Honestly, I don't have to use iSCSI. I can just create shares for most of the stuff that I am doing.

This storage server will be for file/media sharing and my virtualization lab.

Smb is automatically "used" with Windows network shares, the protocol used is dependent on the host and client version of Windows. However it is not seen as shared storage like iscsi and can't be used as a cluster shared volume though you could point your hypervisor vhd location to it.

Iscsi Doesn't automatically aggregate available bandwidth from spare connections, if I'm not mistaken you need to setup multipath IO for that. He is probably asking why you aren't using SMB because it has and implements multichannel capabilities without additional configuration meaning all links available between client/host will be utilized for transfer.

Sent from my LG-H811 using Tapatalk