Help: evaluating a basic network design

[robert1307]

Hello! I'm going to start a small project and going to lease a rack in a real DC for that. The final design is provided below:



My concern is the switch in the middle: Arista 7048T-A, I've never configured devices like that before. But it shouldn't be too hard for my basic design, right?
Each server has two NICs: one for internal (NIC2) and one for external (NIC1) communications.
All I need is to create one VLAN for internal communication between my servers (this is a mandatory requirement) and somehow assign IPs from public block one per switch port (should I create another VLAN here? I don't need my servers to see each other on NIC1, only on NIC2).

So this is my first question: assuming switch has an empty running-config, which commands should be used to achieve the above state?

Second concern. I don't want to spend too much nights in DC trying to configure the above. I wish to simulate DC environment at home first, using a basic WiFi router with OpenWrt as on the schema below:



Assuming if setup above will work at home, same will work in DC, I'll just need to change IPs assignments inside the switch from 192.168.1.x to real public IPs, and change a gateway IP right?
Is this a stupid idea to simulate network like that? Is schema 1 equivalent to the schema 2? Basically - what will be the diff in running-config in switch for schema 1 and 2?

Thanks in advance for all comments and recommendations.

 

[DavidWJohnston]

In order to provide a complete answer, especially specific switch commands, some more info is probably needed like:

• What are you running on the servers? Hypervisor, single OS? What kind of server workload(s)?
• How is your /24 netblock assigned? Do you need to handle BGP/peering, or just assign static IPs directly and the DC gives you a default gateway to route through?
• Why is the INT VLAN mandatory? Is it to avoid bandwidth charges for inter-server (East-West) traffic? Is it layer-2 only? (No routing) Would a crossover cable be sufficient? If not, why not? (I'm not suggesting doing this, it's just to help me understand your workload)
• How are you handling routing and firewall? pfSense, or is it managed by the DC? If so, how?
• Is this setup going to be expanded and evolve? There is a trade-off between initial simplicity and ease of expandability. Why pay for a /24 if you only need 2 IPs?

The things I may suggest, depending on the answer:

- Usually, you wouldn't assign IPs on the basis of switch port - You create VLANs for each of your subnets, then assign ACCESS ports to allow specific ports to join a specific VLAN untagged, then assign TRUNK ports to ports that need to be VLAN tagged and join multiple VLANs, like a pfSense VM or router. It's the OS that specifies the IP, instead of it being "assigned" by the switch.

- Simulating it at home is not a stupid idea at all - In fact it's essential to check your assumptions and pre-config as much as possible before entering the colo space.

If you wouldn't mind providing a bit more detail about the OSs/VMs/routing/firewall or if possible just generally what workloads you'll be running, how you'll access it remotely, etc, I would like to give you a more complete answer, sorry for all the questions.

Dave

 

[robert1307]

Thanks a lot for your questions!

• What are you running on the servers? Hypervisor, single OS? What kind of server workload(s)?

These are single OS servers, no hypervisors. Each server should be able to access internet (and be accessible from internet) as well as access other servers through the private VLAN

• How is your /24 netblock assigned? Do you need to handle BGP/peering, or just assign static IPs directly and the DC gives you a default gateway to route through?

Just assign static IPs directly, DC gives a default gateway to route through

• Why is the INT VLAN mandatory? Is it to avoid bandwidth charges for inter-server (East-West) traffic? Is it layer-2 only? (No routing) Would a crossover cable be sufficient? If not, why not? (I'm not suggesting doing this, it's just to help me understand your workload)

Crossover cable will not be sufficient as there will be more than 2 servers in my setup. It's to avoid extra latencies and bandwidth charges.

• How are you handling routing and firewall? pfSense, or is it managed by the DC? If so, how?

No firewalls in my setup are required (I've explicitly asked DC to disable all firewalls at their end too). This is one of the system requirements as well.
I was going to handle routing through the layer 3 switch (as shown on a diagram).

• Is this setup going to be expanded and evolve? There is a trade-off between initial simplicity and ease of expandability. Why pay for a /24 if you only need 2 IPs?

DC provides a minimal block of /27 (29 usable ip addresses with justification) and I'll have about 10 servers in my setup, 2 external connections per server (one for IPMI, one for the NIC1 interface)

> It's the OS that specifies the IP, instead of it being "assigned" by the switch.
Right, but I need to make switch aware of which port each server is connected to.

 

[kapone]

I may be missing something here...

- You have multiple servers "publicly" accessible. Do each of them run a specific service? Each handles a specific port?
- I don't see a load balancer in there, so your servers are not acting as one.
- If they are not acting as "one" how does the internal VLAN fit into the picture?

There's crucial pieces missing here. You can't just slap multiple servers together with a public IP each AND interconnected to each other. There's essentially...no security in this layout.

 

[robert1307]

>There's crucial pieces missing here. You can't just slap multiple servers together with a public IP each AND interconnected to each other. There's essentially...no security in this layout.
And that's fine for my experiments! The only purpose of internal VLAN is reducing the latency, not to make something more secure.
I just need to understand if switch can be technically configured like that or I must have another switch/router in this setup to achieve my basic goals.

 

[kapone]

switch can be technically configured like that

Of course it can. That's simple.

But, you realize the internal VLAN is meaningless, right? e.g.

Server 1 - Public IP 64.64.64.61
Server 2 - Public IP 64.64.64.62

Regardless of whether you have an internal VLAN or not, server 1 can talk to server 2 at the switch/layer 3 level. The "latency" is no different than your internal VLAN, which would be at the switch level as well.

Edit: And out of curiosity, what are the basic goals?

 

[robert1307]

If this can be done without VLAN I'm fine with this too, I just need to make sure that both NICs will be involved and one NIC will only serve queries from my own servers and another will serve public queries.

 

[kapone]

one NIC will only serve queries from my own servers and another will serve public queries

If that's the case, no internal VLAN still stands, unless...you're capping out the 1gbps port. That would be difficult, but not entirely impossible.

Edit: But I'm still unclear, how two servers with two different public IPs, will handle the "same" application, without a load balancer. And if it's not the same application, why are they talking to each other?

 

[robert1307]

Ok, this is a basic Ad Network platform setup.

There are X HTTP balancers doing a simple pre-filtering and distributing incoming queries from external publishers among Y internal HTTP servers. These HTTP servers should query external (public) DSPs as well as "internal" DSPs (using NIC2).

So the schema looks as below:

 

[kapone]

So...in your case, is:

- one server the balancer and one server the "HTTP" server?
- or both run both?

 

[robert1307]

One is balancer, one is HTTP server. Thanks.

 

[kapone]

One is balancer, one is HTTP server. Thanks.

Option 1:
- Create one VLAN for public IPs. Add the port on the switch that is connected to the DC network, and your NIC ports from servers to it. Now they are publicly accessible, can access the internet, and can talk to each other on the public IPs (but it'll happen on the switch, as they are both connected to the same switch.

Option 2:
- Do the same as above.
- Create another VLAN on the switch for your"internal VLAN", add the relevant ports from your servers to it, no default route for this VLAN, no gateway IP address. This is simply a L2 VLAN.
- Configure your application for inter-server communication using these IPs only, and public communication using the public IPs.

Footnote: Still have to get the DNS entries correct, but that's a different question.

 

[robert1307]

Thanks! Created an upwork job so if you like to make some pocket money you're welcome to apply: https://www.upwork.com/ab/applicants/1631393376072486912/job-details. I expect it's < 1h of work to configure switch as required in two setups, or do you think it will take more time to come up with a working config?

 

[kapone]

Thanks! Created an upwork job so if you like to make some pocket money you're welcome to apply: https://www.upwork.com/ab/applicants/1631393376072486912/job-details. I expect it's < 1h of work to configure switch as required in two setups, or do you think it will take more time to come up with a working config?

Sorry, no need for pocket money and I barely have time to do my stuff.

I was simply being helpful.

 

[robert1307]

Sure! Np! Sorry I just don't want to ask someone to do the job for me for free. I understand the options you've described, I'm just not sure which commands should be used to generate a working config...

 

[DavidWJohnston]

Thanks for the answers, I think I understand better now, though what you're proposing is unusual. If this was being used as part of a business network holding/handling valuable data subject to monetary loss if hacked, the business' insurance company(s) would never OK this config - And if there was ever an incident, the companies feeding you the data could suffer a loss and you may be liable.

- For IPMI and out-of-band management in general, usually what you'd do is have one OOB management server with 2 IPs, 1 for its own IPMI and 1 for inbound VPN/RDP/SSH for remotely connecting. Then once you land on that, you're local and can access any other console or IPMI. No need for so many external IPs. By my reading, you only need 2.

- Like another commenter said I don't think a second NIC will save you any latency, unless you come close to 1Gbps or a high packet ingress/egress rate, and start to have queueing delays. I'll assume the extra NIC is actually worth the trouble.

- You have a multi-layer switch but it's not necessary. If you don't care about isolation, you don't even need VLANs. Just plug all your stuff in, set a management IP for the switch (or just forget about that), and configure static IPs on everything on the default VLAN. No config necessary. Even a dumb switch would work.

If this is all correct, and I haven't missed something, you could try the simplest possible config, which I don't like, but I think it would work:

1. The switch should come out-of-box with 1 default VLAN, and all ports set as ACCESS ports on that default VLAN with zero TRUNKs.
2. Plug in your 10G SFP+ to access the DC network's default gateway
3. Configure the switch's management interface to be 64.64.64.x, correct DC default gateway and mask
4. Set a good password for the switch's SSH mgmt interface since it will be exposed to the internet
5. Plug in your 10 IPMIs, config each IPMI to take 64.64.64.x. Set good passwords on those too
6. Plug in your 10 NIC1s, set the IPs statically to 64.64.64.x in each of your OSs, specify DC default gateway as gateway, correct mask
7. Plug in your 10 NIC2s, set the IPs to 10.31.3.x in each OS with 255.255.255.0 mask, no default gateway
8. Use static IPs in your application config to make sure traffic flows through the intended NIC
9. Consider at least running software firewalls to limit inbound traffic on only the ports you need to answer queries

For the commands to set the switch mgmt IP, take a look at this quick start guide: https://www.arista.com/assets/data/...1ru-gen1_Chapter_5_Configuring_the_Switch.pdf

Even though you don't need to set the switch management IP, at least if you want to expand your system later you can get to it without entering the colo.

Edit: Before entering the colo, set all your IPs on your own time. You can use the real external IPs. You could set your router to be the same IP as the DC's default gateway for testing. Then when you do the physical install all you need to do is plug everything in.

 

[robert1307]

Thanks for such a detailed explanation. I think your steps 1-9 is exactly what I need to do.
Few questions:
- For step 3, should management interface port on the switch be shorted with one of 48 other ports? Do you have any good reference of how to do that? Also for "interface management port" - I know there is a serial management port (so only direct connection to USB/COM is supported, is this interface management 0?). Then there is an ethernet management port (interface management 1?). And also Arista user manual also mentions interface management 2, however I don't see a physical socket on my switch for this third port).
- For step 7 - how does switch know which server is listening on which of 48 ports?

 

[DavidWJohnston]

Yes, you can connect a short cable from the mgmt port to another port. The mgmt port is a device port, and will not cause a switching loop. Just to be safe though, try it first. The better way to do this is to configure an in-band management IP and leave the mgmt port empty. You could check the manual for how to do it.

As far as the switch knowing which server is on which port - If you're asking how the servers find each other on the network - They use the ARP protocol. One box sends an ARP broadcast onto the interface in the same subnet as the desired target, everything on that subnet segment hears the ARP broadcast, but only the intended recipient responds with their MAC address to send traffic to. (The actual details are a bit more complex) ARP is the bridge between L2 and L3.

The switch automatically maintains a MAC table in its memory by snooping the traffic so it knows how to switch packets to the correct ports. It all happens behind-the-scenes and in your case no config is necessary, it's automatic.

 

[robert1307]

>The better way to do this is to configure an in-band management IP and leave the mgmt port empty
Couldn't find this in Users manual . There is a similar chapter: Assigning an IP Address to a Specific Ethernet Management Port, but it refers to the out-band management port. For in-band I've seen only some discussions on forums where people were shorting in and out band ports.

>everything on that subnet hears the ARP broadcast
How does switch knows which server is on which subnet? Or ARP is just always being sent to all 48 ports before being cached in the ARP table on the switch?

 

[DavidWJohnston]

Some switches don't have explicit in-band and out-of-band management IP config; not sure about the Aristas. It could possibly be done by assigning the switch an IP on the default VLAN, then creating an ACL allowing SSH inbound to reach the management kernel.

In your case, I would probably recommend a short cable because it's easy to understand and debug. Make sure you test a power cycle after saving the config to make sure it comes up properly.

The broadcasts are "flooded" to all ports which are members of the same VLAN (In your case, all of them). Once the switch learns the MAC-port mapping it forwards instead of floods. Because your NIC1s and NIC2s have different MACs, and are plugged into different ports, the switch will learn where to forward the traffic - But it doesn't care about subnets - Only MACs. (When using L3 capability, then it cares about subnets and IPs)

Maybe try reading this article: How Switches Work