Can you guys help me with VLANs and subnetting?

[Dajinn]

I have a cluster of servers as such:

4 virtualization servers

4 converged storage servers

1 gigabit ethernet switch

1 QDR infiniband switch

Environment: Windows Server 2012 R2

Network Diagram https://gyazo.com/cefc430cb4092724ae78b7a021d1f23b

Each server has at least >=1 GbE port. Each server has a dual-port ConnectX-2 QSFP infiniband adapter installed.

Now, each server is connected to the gigabit switch and each server is connected to the IB switch. However, the ethernet and IB switch are not interconnected in any way.

What I want to do is keep my circle of servers only to communicate with each other on the IB switch. That is, there should be no way for a share or any communication to be possible over the gigabit ethernet interfaces. I only want the gigabit interfaces to be used for internet connectivity and remote desktop purposes.

I've gotten a lot of mixed answers on this, the most popular one is "use subnets, machines won't randomly just hop across networks for fun". It's a great answer and I still believe it's my ultimate solution but people seem to be presenting it incorrectly.

See my infiniband network adapters are already on their own subnet separate from 192.168.1.0...they're on 172.31.255.0. If I open a share using the IP instead of the hostname and the share name like "\\172.31.255.2\apps" and perform a file transfer, I see clearly with my own eyes in the Windows performance monitor in the task manager that the gigabit ethernet links are being saturated even though I asked it to open the folder at the IB interface.

Ideally what I want to do is thus:

All infiniband adapters - 172.31.255.0, we'll call it subnet A

All virtualization nodes - 10.0.0.0/24, subnet B, accessible to my 192.168.1.0 subnet for internet/RDP

All storage nodes - 10.1.0.0/24, subnet C, accessible to my 192.168.1.0 subnet for internet/RDP

As I understand segregating my two types of nodes like that virtually eliminates any possible of something like L2 discovery between the servers on multiple adapters and enforces communication over only the 172.31.255.0 subnet.

However, I'm a bit lost as to "how to" do this or implement it.

I understand that I'll need to use VLANs but I think my biggest problem is that my gigabit ethernet switch is not a Layer 2 switch so I cannot add static routes and the IPs I give the VLAN interfaces are only for management not for actual subnetting. My switch is the powerconnect 5448.

For me to accomplish this, do I need to replace my L2 switch with a L3 switch that is capable of routing? My "upper-most" networking device is an ASUS router, RT-N66U, that has one long cable going out to the switch for internet/client connectivity to the servers for RDP. Can I set routing in there?

If I don't need a Layer 3 switch can I use tags/egress/ingress filtering to keep packets from other VLANs from going into other VLANs?

The way I keep seeing this in my head is I create a VLAN on a group of ports with a base IP of like, 10.0.0.1 as the gateway of that VLAN and all the clients under it are .1-254 and then I create a route from 10.0.0.1 255.255.255.0 192.168.1.1.

Can anyone sort of walk me through how exactly you do these things? I created a VLAN on my Dell switch and added ports to the group but I don't think it's exactly what I want yet.

Please advise.

 

[CreoleLakerFan]

It sounds like you are trying to force CIFS connectivity to the Infiniband network. If that is what you are trying to accomplish, it will be a lot easier to unbind "File and Printer Client for Microsoft Networks" from the GbE NICs than a bunch of convoluted subnetting and routing.

 

[Dajinn]

The only issue is this only applies to Windows environments, eventually I'm going to want to branch out and use ESXi and other converged solutions. Thoughts?

 

[CreoleLakerFan]

The only issue is this only applies to Windows environments, eventually I'm going to want to branch out and use ESXi and other converged solutions. Thoughts?

If you went the route you are proposing you would have to put each hosts "front-net" adapter in it's own VLAN/subnet. Then you'd need to route and filter traffic between each front-net VLAN.. That may be manageable with your four host setup, but it's simply impractical to manage on scale of any significance. In a multihomed server you simply bind the desired services to the appropriate NIC.

The same principle applies with ESXi, bind physical adapters to virtual switches, then assign virtual nics in your VMs to the appropriate vswitch. You would still unbind MS Client from the vmnet adapter in your VM on the frontnet vswitch, etc. With ESX, you can only run things like iSCSI over a management interface, so you would just make sure you use an IB adapter for your iSCSI configuration.

 

[PnoT]

What you are seeing is called SMB Multi-Channel at work. SMB3.0 and multi-channel will communicate with the destination server over ANY adapters by default for SMB traffic. You have two paths to an SMB share... it will attempt and use both unless you setup smb constraints.

*EDIT*

Sorry wrong link Restrict SMB connections | Networking content from Windows IT Pro

This way it's super simple to setup and keep the right traffic going over the proper adapters without much of a fuss. Set the constraints and watch your transfers again. You can monitor smb multi-channel with powershell and see your partners and what adapters they're using.

Here is a link to the setup and some monitoring commands: The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0 - Jose Barreto's Blog - Site Home - TechNet Blogs

Honestly, you don't need VLANs in that setup at all unless you just want to learn about them.

 

[Dajinn]

What you are seeing is called SMB Multi-Channel at work. SMB3.0 and multi-channel will communicate with the destination server over ANY adapters by default for SMB traffic. You have two paths to an SMB share... it will attempt and use both unless you setup smb constrained delegation which tells SMB you ONLY want it to run over IB.

Here's some reading Hyper-V over SMB: Set SMB Constrained Delegation via PowerShell | Thomas Maurer

This way it's super simple to setup and keep the right traffic going over the proper adapters without much of a fuss. Set the delegation and watch your transfers again. You can monitor smb multi-channel with powershell and see your partners and what adapters they're using.

I read that earlier but it doesn't address how to tell SMB to only run over IB. It just addressed an access denied issue? Did I miss something? Further reading suggestions constrained delegation has to deal more with re-issuing authentication to access a resource than it does specifying particular network adapters for shares.

 

[PnoT]

Edited my previous post sorry.

If you want to monitor any RDMA traffic you'll have to setup perfmon and the proper filters.

 

[Dajinn]

Nice, thanks for the link on interface constraints. I'll check it out tomorrow, need to take a break since I've been messing with everything all day. I'll probably still pick up a L3 switch at one point since I feel like subnetting and routing are good skills to have.

 

[Dajinn]

If you went the route you are proposing
you would have to put each hosts "front-net" adapter in it's own VLAN/subnet. Then you'd need to route and filter traffic between each front-net VLAN.. That may be manageable with your four host setup, but it's simply impractical to manage on scale of any significance. In a multihomed server you simply bind the desired services to the appropriate NIC.

The same principle applies with ESXi, bind physical adapters to virtual switches, then assign virtual nics in your VMs to the appropriate vswitch. You would still unbind MS Client from the vmnet adapter in your VM on the frontnet vswitch, etc. With ESX, you can only run things like iSCSI over a management interface, so you would just make sure you use an IB adapter for your iSCSI configuration.

I have a concern about this, if I unbind file/printer sharing from an interface then my clients can't connect to the share. My question is, if I unbind all of my hypervisor's ethernet adapters from file/printer sharing is it okay to leave the file/printer sharing binding enabled on my storage nodes? I'm a little afraid that requests can still go out from adapters where file/printer sharing is not enabled as long as the destination is enabled...because the description itself is "Allows other computers to access files on a Microsoft network". Meaning, the ethernet adapter could initiate a connection on it. Or worse, an infiniband adapter on the hypervisor could try to reach out to any open interfaces on my storage node and get a response from the ethernet adapter because it has file sharing enabling...

Also,

would anyone have any hints on how to troubleshoot plummeting network transfer speeds when reading from a source server? On one of my test beds when I copy a large file (30GB) from a storage node over IB I get 1.2GB/s for like, maybe half of the transfer and then it hits ~220MB/s usually at the exact same place in the transfer at the exact same time. On the other hand, if I take that same file and transfer it back across IB, I get 750-800MB sustained write. Any ideas?

 

[PnoT]

How much free RAM do you have in each machine when doing these tests? If you open task manager and start a transfer you'll see that your RAM is being consumed. If you don't have enough RAM to completely transfer the file you revert to disk speeds because it can't be flushed from RAM quick enough and you're back to the usual bottlenecks. If you really want to test create 2 RAM drives (1 per server) and move files to / from those locations.

 

[Dajinn]

How much free RAM do you have in each machine when doing these tests? If you open task manager and start a transfer you'll see that your RAM is being consumed. If you don't have enough RAM to completely transfer the file you revert to disk speeds because it can't be flushed from RAM quick enough and you're back to the usual bottlenecks. If you really want to test create 2 RAM drives (1 per server) and move files to / from those locations.

Okay so you're right on the RAM being consumed, however, it's not anywhere near using all of my RAM. This is what happens.

This screenshot is literally the mark where the RAM hits however much it wants to totally consume and my transfer drops off from 1.2GB, the 800 MB/s you see is just like the first number it dropped down to

The next screenshot is just the statistics after it's continued to transfer the file at 200~MB/s for a few seconds, the RAM consumed doesn't seem to change and I'm not anywhere near maxing out the capacity. I don't have any VMs running either.

Also, this happens randomly too. I can start a file transfer of the same over like 3-4 times and start at 1.2GB/s and other times when I start a transfer it will be start off at like 200MB/s and never exceed more than maybe 330MB/s if I'm lucky. Not using much RAM here either.

 

[PnoT]

Are you using RAM drives on both ends? You also have to keep in mind that some of that file, especially if you're using it to go back and forth, is set aside in cache and can affect initial speeds.

 

[Dajinn]

No I'm not, I've been testing it with a file just to see if I got any benefit from file transfers. I'm fairly certain it is using the IB connection so I'm not too worried about that. Just pretty depressing(lol)I guess to see an initial burst transfer to 2 Gbps...at any rate, it shouldn't be a big deal. At the least, I'm not actually doing -any- of the writing to the clients you see above in the screenshots where I'm transferring over. I just wanted IB so that there wasn't a networking bottleneck when the CPU is reading from the VHDX over the network.

 

[wildchild]

If you are using esxi , its very much simpler.
Create 2 vswitches.. call them
storage - bind iscsi to that.
Network - bind 1gb interfaces to that, trunk if need is there

All storage related stuff will go over storage, rest will go over others.

For your storage server, dont set a gw to your ib adapter..

Problem solved

 

[Dajinn]

Are you using RAM drives on both ends? You also have to keep in mind that some of that file, especially if you're using it to go back and forth, is set aside in cache and can affect initial speeds.

https://forums.servethehome.com/ind...r-infiniband-switch-write-up.6890/#post-61346

Check that post for iperf/ramdisk testing.

 

[Mike Bailey]

Just to jump in on the networking side, it's absolutely possible to segment the network the way you want.

I'm not sure what version of the Tomato firmware that you're running, but the regular "Tomato" doesn't support VLANs. You can check what they support here: Tomato (firmware) - Wikipedia, the free encyclopedia

*Assuming* you have a VLAN capable firmware, you can segment the router such that it has three VLANs exposed:

- VLAN 1 - AKA Native VLAN, all desktops can sit here with untagged traffic
- VLAN 2 - Hypervisors
- VLAN 3 - Storage Hosts

VLAN 2 & 3 you'll have to trunk from whatever port you're running from the router to your PowerConnect. In the example diagrams I made, there's a "VLAN 4" but this doesn't really exist. It's purely for illustration purposes.

From a Layer 3 perspective your network will look like:

And from a Layer 2 perspective it will look like:

In terms of the interfaces for the router you'll have:
- 192.168.1.1/24, no VLAN (aka "VLAN 1" / Native VLAN)
- 10.0.0.1/24, VLAN 2
- 10.1.0.1/24, VLAN 3

For the PowerConnect you'll do:
- Access VLAN 2 Tagged for any Hypervisor ports
- Access VLAN 3 Tagged for any Storage Host ports.

For the GbE adapters:
- Default Gateway = 10.0.0.1 (Hypervisors) or 10.1.0.1 (Storage Hosts)
- Subnet Mask = 255.255.255.0
- Unbind File and Printer Sharing for Microsoft Networks (Storage Hosts)
- Unbind Client for Microsoft Networks (Hypervisors)

The Infiniband adapters, which I presume are running IPoIB, you'll do:
- Default Gateway = Do not define
- Subnet Mask = 255.255.255.0
- Bind File and Printer Sharing for Microsoft Networks and Client for Microsoft Networks

Then in Windows itself, you configure your Storage Hosts:
- Set-SmbServerConfiguration -EnableMultiChannel $True

On your Hypervisors:
- Set-SmbClientConfiguration -EnableMultiChannel $True
- New-SmbMultichannelConstraint -ServerName Your_Storage_Host_Name -InterfaceAlias IB_Adapter_1,IP_Adapter_2

You'll run the New-SmbMultichannelConstraint cmdlet for each storage host you want to connect to from the hypervisor. This will force traffic to negotiate multiple active/active links from the hypervisor to the storage host.

I don't have IB running in my lab (yet) but I'm basically doing the same thing with GbE adapters instead of IB adapters.

On a completely unrelated note: The edit post functionality for this forum is horrible. It's excruciatingly painful to actually edit something on a Surface Pro 3.

 

[Dajinn]

Amazing Mike, simply amazing.

 

[Mike Bailey]

Thanks Dajinn. I got stuck in an airport waiting for a flight back to New York (delayed by the Pope coming in..) so it gave me something to do.

Does that answer the questions that you had? I have a few ideas of what you would do if you were to put a L3 switch in place of the L2. Let me know if you're interested.

 

[Patrick]

@Mike Bantegui awesome post.

 

[Dajinn]

Thanks Dajinn. I got stuck in an airport waiting for a flight back to New York (delayed by the Pope coming in..) so it gave me something to do.

Does that answer the questions that you had? I have a few ideas of what you would do if you were to put a L3 switch in place of the L2. Let me know if you're interested.

Yes it does answer my question. I am interested and I am in the process of swapping out my L2 switch with a L3 switch. I've got a Cisco C3560E arriving later today.

PM if you want to continue to work this out as there's a few caveats here to your suggestions(nothing major just some hardware things). I'd love to hear from you!