SMB 3.0 and Bandwidth Aggregation

[PigLover]

Odditory made a comment in another thread about SMB 3.0 and Bandwidth Aggregation that has really got me thinking. Rather than hijack Sboesh's thread I thought I'd start a new one for discussion just on this topic:

As an FYI if you decide to upgrade to Windows8 and Server 2012, teaming is moot for home usage unless you're after redundancy, since SMB3.0 aggregates links automatically. As you may already be aware, NIC teaming doesn't spread your transfers across the aggregated nics.

With dual Intel nics on both my Win8 workstation and Server2012 I got about 225-240MB/s on file transfers. For kicks I added a cheap $20 dual-Intel GbE nic to both server and workstation and connected two more cables and transfers doubled again to about 440-460MB/s - and you can see it working in task manager with all four nics hitting a hair away from theoretical max. Dual 10GbE nics and Infiniband nics are next on the playlist. Crazier yet is you can hot-add and remove nics DURING a transfer and SMB3 ratchets up or down on the fly - totally dynamic. I'm sure it'll be a boon for WANs as well.

quick screenshot of nics in task manager while copying from server to single ssd -- however throughput gets much higher when i'm copying from fast array on server to ssd array on workstation - about 980Mb/s per nic sustained and totally symmetrical.

Lots of questions - so I went google'ing and found a couple of pretty good write-ups on this topic:

A powerpoint/training on SMB3.0
A video specifically on Bandwidth aggregation
An above-average TechNet blog on the same topic

So in its basics it looks pretty straightforward, but I've got a bunch of questions about some reasonable configurations.

For example, what if the server has 1 or 2 10Gbe links but the client has a group of 3 or 4 (or more) 1Gbe links? Can it effectively figure this out and set up four streams to the client, spread across the two 10Gbe links for load balancing?

Do all of the links need to be on the same subnet? Or could one link go to one subnet and one link go to a second subet (presumably on a second switch/router - poor man's router redundancy)?

If you can split subnets, could you share a single link to the "normal" network with 1 (or more) direct point-to-point links to the server? This would be interesting if the client-server direct links used some non-traditional link layer like Infiniband or point-to-point 10Gbe?

Lots of silly questions like this running around in my otherwise empty head...

Also - while this is all very interesting, the relative immaturity of Storage Spaces (compared to ZFS) might make the entire topic moot for a couple of years. Which begs another question: does anybody know if there are plans to support Samba 4.0 on Solaris (or a breed of Linux with ZOL support)? At least one of the links above suggests that Samba 4.0 supports SMB 3.0.

Just rambling...

 

[cactus]

Last I read, there isnt going to be SMB Multichannel or Direct in Samba 4.0, hopefully that has/will change.

 

[odditory]

Also - while this is all very interesting, the relative immaturity of Storage Spaces (compared to ZFS) might make the entire topic moot for a couple of years.

Moot, not at all. SMB3.0 was designed with enterprise in mind, and enterprise will continue using hardware RAID, SANs and NAS's as they have been. I dont think MS made any assumptions about enterprise adopting Storage Spaces, but rather smallbiz.

As for the rest of your questions, SMB3.0 is enabled by default in both Server 2012 and Windows 8, there was literally zero configuration, I even gave all the NICs random IP's within the same subnet rather than sequential ones just to see if it made any diff (it didn't). But I haven't read up on any whitepapers yet explaining how the multichannel is negotiated (assuming the mechanics are even documented) or what the default rules are, but somehow each side figures it out. The re(configuration) of SMB3.0 is done in Powershell, which is also where you monitor it working, disabling/enabling, capabilities of each NIC (RDMA, etc) and I only spent a few minutes poking around but I'm sure there are ways of setting up rules for custom configurations and routing.

Edit: And reading this here its more than obvious that custom rules and routing are possible, even though this passage is talking about RDMA capable 10GbE adapters. I imagine a lot of it simply has to do with it following your existing TCP/IP routing table since it rides on top of it, but there's an additional stack of features that SMB3.0 brings which would have their own configurable ruleset.

RDMA over Ethernet (iWARP) is an industry-standard low latency protocol developed by the IETF and supported by the industry's leading 10Gb Ethernet adapters. Since it is built on top of TCP/IP, it is routable and scalable from just a few to thousands of co-located or geographically dispersed endpoints and is resilient to any data center topology or speed. It further can work independent of the DCBX protocol, thereby decoupling the server and switch refresh cycles.

 

[odditory]

Also worth noting: Unplugging ethernet cables from the NICs randomly for test purposes at either server or workstation during a long transfer doesn't phase it, you simply see the aggregate throughput reduce by that adapter's worth of throughput, but it continues without pause. Plugging them back in and the adapters rejoined the transfer immediately, reflected in the throughput graph.

Reminds me of the way torrent P2P is able to work with peers coming online or dropping off randomly but the transfer continuing, totally resilient.

 

[Jeggs101]

So I get double bandwidth without needing to setup a switch? What about if you have two links and a Linux or OI server --- still no need to config the switch?

 

[PigLover]

SMB Multipath requires both ends to support it (both server and client). Currently only Windows8, Server2012 and their derived products support it. The Samba 4.0 project is implementing SMB 3.0, but as pointed out above their first release - probably first few releases - will not support Multipath.

So no, you can't get the benefit if your server is OI or Linux based.

 

[hagak]

Anyone know when SMB will support MS indexing?

 

[omniscence]

Will SMB 3.0 also detect dedicated links between two machines? For example two 1Gb links to the switch from each Windows 8 machine and two additional 1 Gbe links in a switchless direct connection?

 

[odditory]

^ Yes. Tried it both ways. Once with all workstation & server NICs plugged into switch, once with half the connections plugged into switch and the other half direct.

The fact that plugging or unplugging the connections during a transfer, and let's say manually changing an IP on a NIC during the transfer, results in the two sides reconnecting instantly makes me think there's some logic happening such that each side lets the other know what interfaces/ip's it has available for multichannel connections.

 

[bp_968]

Maybe Im asking this due to my lack of experience with OI and ZFS but couldnt you have an ESX setup with a VM of OI setup as a SAN/NAS and have a windows server 2012 VM attach to a block of drives via iSCSI to the OI VM and provide SMB3 to all the win8 clients? That way you get ZFS & SMB3?

 

[cactus]

Maybe Im asking this due to my lack of experience with OI and ZFS but couldnt you have an ESX setup with a VM of OI setup as a SAN/NAS and have a windows server 2012 VM attach to a block of drives via iSCSI to the OI VM and provide SMB3 to all the win8 clients? That way you get ZFS & SMB3?

Yes, but your slowest link is still iSCSI so the SMB3 multi-path may not be of help. I can see doing OI with vmxnet3 to windows 2012 with vmxnet3 to leverage multiple 1GbE links. If you only have one or two clients, going with QDR IB from OI server to client seems like a less complex and higher throughput solution though.

 

[hagak]

I actually did this setup exactly, and found that my SMB transfers with windows 2012 were a good bit slower than with SMB on OI so went to just sharing the OI directly. I originally wanted the Server 2012 to do the sharing so that I could take advantage of the Indexing, however the speed reduction was too much.

Note my ALL-in-one is no slouch either. 2x E52640 32GB RAM, and the ZFS pool is 13 Constellation ES.2 drives in RAID 10 config. My Bonnie testes are ~1400 MB/s reads and ~750 MB/s writes. My read/writes for the iSCSI on the windows 2012 was ~350MB/s (VMXNET3 on both VMs) however the SMB shares from 2012 would never exceed ~50MB/s while the OI SMB shares would saturation my link at around 115MB/s.

 

[nitrobass24]

Any update on Samba4? Was this included in the release...I cant find much about it.

 

[sboesch]

SMB 3.0 is part of Samba 4, as of now, there is no multi-channel support for the SMB 3.0 protocol. I am earnestly waiting for this as well. I can not find any info as for when multi-channel support will be implemented.

Any update on Samba4? Was this included in the release...I cant find much about it.

 

[Patrick]

Very interesting. Somewhat excited for this one.

 

[gigatexal]

this might be a dumb question but is samba still single threaded? or does this multipathing stuff effectivley make it multithreaded?

 

[sboesch]

The Samba project has always been big on keeping the project workable across the platforms that use it, and for keeping it small, and efficient. I like the idea of each SMB connection being its own process. Making it multithreaded might sound good, but it would actually break its portability and cross platform usage.

this might be a dumb question but is samba still single threaded? or does this multipathing stuff effectivley make it multithreaded?

 

[gigatexal]

i dont see how they cant throw in some sort of control function that checks number of threads the system can use or number of nics and spawns new worker threads to do things more parallel and potentially more efficiently.

that being said the fact that such a project exists and works so well as an OSS project i should just stfu because i've used it for what it's worth a lot

 

[cactus]

My guess is samba is IO bound not CPU bound and the minimal performance improvement they would get going multithread would not offset the additional work for file locking and thread coherence. If your bottleneck is the storage pool you are reading from, it does not matter how many threads you have waiting for data. If you are doing other computation besides reading/writing data, then threading/multi-process will help performance.

Last I looked at the the samba4 roadmap there was no plan on SMB3 direct or SMB3 multipath. The project is looking more to become a fully functional domain controller replacement which requires neither of the former technologies.

 

[nitrobass24]

Last I looked at the the samba4 roadmap there was no plan on SMB3 direct or SMB3 multipath. The project is looking more to become a fully functional domain controller replacement which requires neither of the former technologies.

Nail on the head!
Everything they are doing is about being compatible/replacement of MSFT AD DS.