Cannot get SMB Multichannel to work (Win 8.1 <-> 10)

[MrFlppy]

Hi,

I posted some questions in the general SMB 3.0 thread but since I'm still running into trouble using only a single switch I'm creating a separate thread - maybe others are having these issues, too.

Configuration:

- Switch: TP-Link TL-SG2210P (manual http://www.tp-link.us/resources/document/TL-SG2210P_V1_UG.pdf )
- PC 1: 4C/8T CPU, Intel I350-T4 V2 two 1 Gbps connections to the switch, Windows 8.1 Pro x64
- PC 2 4C/8T CPU, Intel I350-T2 V2 two 1 Gbps connections to the switch, Windows 10 Pro x64

- The switch has the latest firmware installed and has been reset to factory defaults to avoid any past LAG settings interfering

- All Intel NICs have the latest Intel drivers installed (from the ethernet connections package 21.0, I extracted the drivers only to avoid issues with the Intel management software)

- Neither at the PCs nor the switch are any LAG/teaming settings enabled manually

- Each IPV4 address is set manually, each NIC can ping every other NIC

- Large file transfers via Windows Explorer (accessed the shares via \\PCxy, not their IP addresses) are stuck at 113 MB/s (118 MB/s with Jumbo Frames enabled), during which only one NIC on each PC is at 99 %, the other one is at 0 % activity

- Windows PowerShell cmdlets show on each PC:

1) "get-NetAdapterRSS": RSS is active at the connected NICs
2) "get-SMBServerNetworkInterface": "RSS Capable : True"
3) "get-SMBClientNetworkInterface": "RSS Capable : True"
4) "get-SMBServerConfiguration": "EnableMultiChannel : True"
5) "get-SMBClientConfiguration": "EnableMultiChannel : True"

I'm a little baffled what to check next since SMB Multichannel should "just" work with suitable software/hardware combinations

Thanks in advance for any help!

 

[iLya]

I didn't think that doing a normal file copy from server to server does SMB.
The place that you are copying to or copying from needs to be an SMB source/destination like an SMB share.
I have a Scale Out File Server deployed and it hosts an SMB share.
I might be completely wrong here as I am learning my self about SMB but my source only sends ~400Mbps/s and other nodes that are part of the SOFS cluster are seeing traffic on both NICs.
I don't have LACP/LAG configured and I don't have teaming either.

 

[cesmith9999]

do a robocopy transfer with the /mt:16 switch. and see if you get better results.

Chris

 

[MrFlppy]

@iLya
I hope I'm not missunderstanding you but how would a copy process from a SMB share to a local destination not go via SMB?

@cesmith9999
Shame on me I've never used robocopy - could you give a quick example with a \\PC2\Share\File.xy copy to C:\Path\To\Copy?

Thanks for your efforts!

 

[iLya]

@MrFlppy, I've missed the part where you mention that you access a share on each server. That is why I asked the question but I can see now that you mention in your original post that you access each system using the share name and not the PC directly.

FYI, I've been using DiskSpd utility for generating IO on my systems which is also capable of doing multiple threads and a few other things.
URL to download: TechNet Diskspd Utility: A Robust Storage Testing Tool (superseding SQLIO)
Source: GitHub - Microsoft/diskspd: DISKSPD is a storage load generator / performance test tool from the Windows/Windows Server and Cloud Server Infrastructure Engineering teams
A quick example of how to use: Using Microsoft DiskSpd to Test Your Storage Subsystem

BTW, I am attempting to setup a Hyperconverged cluster with Hyper-V with SOFS using SMB3 across 4 nodes and have a big need for SMB3 working correctly just like you so I am interested in making this work as well.
However I am limited with my dual SATA Drives in each node and I am not sure if I am hitting the limit on the two drives or my dual 1GB NICs so i'll be attempting to add some RAM disks for caching to validate the bottleneck and then move to optimizing my QoS policies as that is another issue that I might be running into due to the Hyperconverged configuration.

 

[Marsh]

@MrFlppy

I might have missed it in your post.
Did you put each NIC in a separate subnet?
Check task manager -> Performance -> Ethernet , to see if SMB multi-channel is working correctly across multiple Ethernet.

 

[MrFlppy]

@Marsh

No they are all in the same subnet, hence the statement "Each IPV4 address is set manually, each NIC can ping every other NIC".
Or do you mean that they should be in a different subnet?

 

[Marsh]

do you mean that they should be in a different subnet?

Yes, each NIC should have different subnet IP.

 

[iLya]

I've read it in multiple places that this might need to be the case but I could never understand why that was a requirement and if the subnets needed to be routable (talk between subnets) or if they just needed to be able to talk between each other.

This to me would mean that you would then want to create a single Virtual Switch and with the NICs teamed you can create multiple Virtual Adapters to get better throughput otherwise you'll just end up having multiple NICs where only SMB gets a benefit and everything else might actually suffer.

 

[MrFlppy]

Yes, each NIC should have different subnet IP.

Tried it, the corresponding NICs can ping each other but still only one NIC is with 99 % activity during large file transfers.

 

[iLya]

Yes, each NIC should have different subnet IP.

I've been able to confirm that you don't need NICs on different subnets to get the SMB to work correctly.
I was able to create a RAM disks on the servers and share a folder from hose RAM disks, I was then able to use the DiskSpd tool to generate a load on that share and was able to get ~950Mbps/s on both NICs at the same time.
The need for the RAM disk was to get around the single SATA drive throughput limit of ~120 MB/s.
I was also able to perform a normal windows copy operation and got ~220 MB/s transfer

 

[MrFlppy]

You bastard!

 

[iLya]

You bastard!

Now I have to remember how I did all of this to make sure that I can make it work again.
Keep in mind that I am doing this using Storage Spaces Direct and I have Hyper-V as well as Scale Out File Server running across all 4 nodes so I get impacted by the S2D doing all of the storage replication as well as cloning from the share hosted across all of these nodes

Not sure if it matters but the only thing that I remember changing earlier today was to disable IPv6 on my NICs. I just double checked and 2 of my 4 nodes that had Virtual Adapters (1 on each server) had IPv6 enabled and it was still working. I disabled it on all of them just for consistency sake.

Another thing of note is that I did install Multipath I/O Feature on my Windows 2016 cluster as it was recommended by System Center Virtual Machine Manager at which point it also then configured that feature in windows. I wonder if that also triggers some additional capabilities that make this work.

 

[MrFlppy]

IPv6 is not used/deactivated on all devices in the (small) home network.

As far as I know SMB Multichannel should work with non-Server Editions Windows 8 and later so Multipath I/O (Server-exclusive feature?) should not be it.

 

[iLya]

Some additional details of how MPIO is configured on my servers:


The list of MPIO Devices is:

3PARdataW
DGC DISK
DGC LUNZ
DGC RAID 0
DGC RAID 1
DGC RAID 10
DGC RAID 3
DGC RAID 5
DGC VRAID
EMC SYMMETRIX
EQLOGIC_100E-00
HP HSV300
IBM 2145
MSFT2005iSCSIBusType_0x9
MSFT2011SASBusType_0xA
NETAPP LUN
NETAPP LUN C-Mode
Vendor 8Product 16

I am not sure but it appears that the number of spaces matters I just can't tell how much is required.

 

[DavidRa]

Right so MPIO isn't required for SMB multichannel - I can confirm I have multiple servers without the MPIO feature installed but they both do SMBMC successfully. The list of MPIO devices is a list of physical disk types which should be treated as MPIO-capable, in order that they show up only once in Device Manager and you don't get device corruption by having them appear as disk D:, E:, F: and G:.

Vendor8 Product16 is a placeholder to indicate that the 24 character MPIO device ID is in fact an 8 character Vendor ID and a 16 character Product ID - so if we take the IBM 2145 example that would be:

IBM[space][space][space][space][space]2145[space][space][space][space][space][space][space][space][space][space][space][space]

Oh and the devices you have listed there are:

• 3PAR Array LUNs (Fibre Channel)
  
• DGC - Data General Clariion (EMC Clariion) LUNs (Fibre Channel)
• EMC Symmetrix LUNs (Fibre Channel)
• EqualLogic LUNs (iSCSI)
• HP Storage Virtualised LUNs (Fibre Channel)
• IBM DS and other array LUNs (Fibre Channel)
• Microsoft iSCSI Target Devices (iSCSI)
• Microsoft SAS Devices (plain SAS)
• NetApp LUNs (Fibre Channel)

Don't have those SANs? Don't need MPIO or it's not working for you

The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0 is the goto reference and agrees that two clients should be able to do SMBMC. One non-obvious caveat - both adapters must be connected at the same speed, and I assume you've checked that all adapters have negotiated 1Gbps correctly? There's a limit of 8 connections per client/server pair, so if you have a stack of mapped drives - that could be limiting it too. Perhaps try the extra troubleshooting in section 5 (you might almost start on the last part of 5.1 at this stage, but it won't hurt to re-check earlier items again).

 

[cesmith9999]

@cesmith9999
Shame on me I've never used robocopy - could you give a quick example with a \\PC2\Share\File.xy copy to C:\Path\To\Copy?

Thanks for your efforts!

robocopy \\sourceserver\sourceshare \\destinationserver\destinationshare /e /mt:16 /r:1 /w:1 /nfl /ndl

This is my typical robocopy script when I am doing manual file copies.

Robocopy is almost an art form. There are a lot of parameters. robocopy /help will give you 2-3 screens full of information.

Chris

 

[cesmith9999]

@DavidRa thank you for your excellent reply. now I do not need to reply.

Chris

 

[MrFlppy]

[...] The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0 is the goto reference and agrees that two clients should be able to do SMBMC. One non-obvious caveat - both adapters must be connected at the same speed, and I assume you've checked that all adapters have negotiated 1Gbps correctly? There's a limit of 8 connections per client/server pair, so if you have a stack of mapped drives - that could be limiting it too. Perhaps try the extra troubleshooting in section 5 (you might almost start on the last part of 5.1 at this stage, but it won't hurt to re-check earlier items again).

Yes, all NICs run at 1 Gbps, no paket loss.

I re-checked everthing in that linked guide, RSS seems to be fine everywhere the only thing that irritates me is that RDMA is mentioned in most articles about SMB Multichannel. But RDMA is only a requirement for SMB Direct, or am I wrong?

Maybe a lead (from 5.1):

Step 3: Verify the SMB connection

On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right version of SMB and that SMB Multichannel is working:

Get-SmbConnection

Get-SmbMultichannelConnection

Get-SmbMultichannelConnection -IncludeNotSelected

Get-SmbConnection shows "Dialect : 3.02" and "NumOpens : 3" but the cmdlets Get-SmbMultichannelConnection and Get-SmbMultichannelConnection -IncludeNotSelected do not show anything, meaning after entering them and pressing [Enter] nothing happens and PowerShell awaits fresh input.

Additionally I noticed a new driver entry with the latest drivers for the I350-T2/T4 V2: I do not know the real English term for it but roughly translated it means "DMA Converging" with options "Disabled" and various time values, from 250 µs (shortest) to 10 ms (longest). What does this mean? To my knowledge the I350 shouldn't have any DMA features?

 

[cesmith9999]

RDMA is only a feature in some 10GB adapters and above. it is not required for 1GB adapters.

Chris