I have a cluster of servers as such:
4 virtualization servers
4 converged storage servers
1 gigabit ethernet switch
1 QDR infiniband switch
Environment: Windows Server 2012 R2
Network Diagram https://gyazo.com/cefc430cb4092724ae78b7a021d1f23b
Each server has at least >=1 GbE port. Each server has a dual-port ConnectX-2 QSFP infiniband adapter installed.
Now, each server is connected to the gigabit switch and each server is connected to the IB switch. However, the ethernet and IB switch are not interconnected in any way.
What I want to do is keep my circle of servers only to communicate with each other on the IB switch. That is, there should be no way for a share or any communication to be possible over the gigabit ethernet interfaces. I only want the gigabit interfaces to be used for internet connectivity and remote desktop purposes.
I've gotten a lot of mixed answers on this, the most popular one is "use subnets, machines won't randomly just hop across networks for fun". It's a great answer and I still believe it's my ultimate solution but people seem to be presenting it incorrectly.
See my infiniband network adapters are already on their own subnet separate from 192.168.1.0...they're on 172.31.255.0. If I open a share using the IP instead of the hostname and the share name like "\\172.31.255.2\apps" and perform a file transfer, I see clearly with my own eyes in the Windows performance monitor in the task manager that the gigabit ethernet links are being saturated even though I asked it to open the folder at the IB interface.
Ideally what I want to do is thus:
All infiniband adapters - 172.31.255.0, we'll call it subnet A
All virtualization nodes - 10.0.0.0/24, subnet B, accessible to my 192.168.1.0 subnet for internet/RDP
All storage nodes - 10.1.0.0/24, subnet C, accessible to my 192.168.1.0 subnet for internet/RDP
As I understand segregating my two types of nodes like that virtually eliminates any possible of something like L2 discovery between the servers on multiple adapters and enforces communication over only the 172.31.255.0 subnet.
However, I'm a bit lost as to "how to" do this or implement it.
I understand that I'll need to use VLANs but I think my biggest problem is that my gigabit ethernet switch is not a Layer 2 switch so I cannot add static routes and the IPs I give the VLAN interfaces are only for management not for actual subnetting. My switch is the powerconnect 5448.
For me to accomplish this, do I need to replace my L2 switch with a L3 switch that is capable of routing? My "upper-most" networking device is an ASUS router, RT-N66U, that has one long cable going out to the switch for internet/client connectivity to the servers for RDP. Can I set routing in there?
If I don't need a Layer 3 switch can I use tags/egress/ingress filtering to keep packets from other VLANs from going into other VLANs?
The way I keep seeing this in my head is I create a VLAN on a group of ports with a base IP of like, 10.0.0.1 as the gateway of that VLAN and all the clients under it are .1-254 and then I create a route from 10.0.0.1 255.255.255.0 192.168.1.1.
Can anyone sort of walk me through how exactly you do these things? I created a VLAN on my Dell switch and added ports to the group but I don't think it's exactly what I want yet.
Please advise.
4 virtualization servers
4 converged storage servers
1 gigabit ethernet switch
1 QDR infiniband switch
Environment: Windows Server 2012 R2
Network Diagram https://gyazo.com/cefc430cb4092724ae78b7a021d1f23b
Each server has at least >=1 GbE port. Each server has a dual-port ConnectX-2 QSFP infiniband adapter installed.
Now, each server is connected to the gigabit switch and each server is connected to the IB switch. However, the ethernet and IB switch are not interconnected in any way.
What I want to do is keep my circle of servers only to communicate with each other on the IB switch. That is, there should be no way for a share or any communication to be possible over the gigabit ethernet interfaces. I only want the gigabit interfaces to be used for internet connectivity and remote desktop purposes.
I've gotten a lot of mixed answers on this, the most popular one is "use subnets, machines won't randomly just hop across networks for fun". It's a great answer and I still believe it's my ultimate solution but people seem to be presenting it incorrectly.
See my infiniband network adapters are already on their own subnet separate from 192.168.1.0...they're on 172.31.255.0. If I open a share using the IP instead of the hostname and the share name like "\\172.31.255.2\apps" and perform a file transfer, I see clearly with my own eyes in the Windows performance monitor in the task manager that the gigabit ethernet links are being saturated even though I asked it to open the folder at the IB interface.
Ideally what I want to do is thus:
All infiniband adapters - 172.31.255.0, we'll call it subnet A
All virtualization nodes - 10.0.0.0/24, subnet B, accessible to my 192.168.1.0 subnet for internet/RDP
All storage nodes - 10.1.0.0/24, subnet C, accessible to my 192.168.1.0 subnet for internet/RDP
As I understand segregating my two types of nodes like that virtually eliminates any possible of something like L2 discovery between the servers on multiple adapters and enforces communication over only the 172.31.255.0 subnet.
However, I'm a bit lost as to "how to" do this or implement it.
I understand that I'll need to use VLANs but I think my biggest problem is that my gigabit ethernet switch is not a Layer 2 switch so I cannot add static routes and the IPs I give the VLAN interfaces are only for management not for actual subnetting. My switch is the powerconnect 5448.
For me to accomplish this, do I need to replace my L2 switch with a L3 switch that is capable of routing? My "upper-most" networking device is an ASUS router, RT-N66U, that has one long cable going out to the switch for internet/client connectivity to the servers for RDP. Can I set routing in there?
If I don't need a Layer 3 switch can I use tags/egress/ingress filtering to keep packets from other VLANs from going into other VLANs?
The way I keep seeing this in my head is I create a VLAN on a group of ports with a base IP of like, 10.0.0.1 as the gateway of that VLAN and all the clients under it are .1-254 and then I create a route from 10.0.0.1 255.255.255.0 192.168.1.1.
Can anyone sort of walk me through how exactly you do these things? I created a VLAN on my Dell switch and added ports to the group but I don't think it's exactly what I want yet.
Please advise.