ESXi+Ubuntu VLAN confusion

[mobycl1ck]

Hello STH forum,
I was given a task was to create a Ubuntu server virtualized under VSpere with a custom configuration.

Requirements:
From VSphere, senior admin added 4 NICs to the machine and told me to configure the NICs and their VLANs, as follows:

NIC1 10.100.100.24/24 - VLAN 400
NIC2 10.100.100.96.24 - VLAN 401
NIC3 10.100.100.97.24 - VLAN 402
NIC 10.100.100.98.24 - VLAN 402

When installing the Ubuntu Server I added the VLAN tag 400 to the first NIC that was added. Then senior admin added the other 3 NICs and told me to configure those.
I managed to install vlan package, add the 8021q module and created the vlans accordingly.
But there is something I must be doing wrong, being that once the interfaces are set up and activated, I can not reach the gateway , nor the internet.
Questions: Does each NIC has to have an IP and the VLAN has to have another?
Do I have to give to each NIC an IP, or only to its VLAN?
Can I add all the VLANs to a single virtual NIC, or do i have to set up VLAN tagged to each NIC ? How to I prioritize traffic to the first NIC?
Can anyone kindly point me in the right direction?

 

[nabsltd]

NIC1 10.100.100.24/24 - VLAN 400
NIC2 10.100.100.96.24 - VLAN 401
NIC3 10.100.100.97.24 - VLAN 402
NIC 10.100.100.98.24 - VLAN 402

Forgetting the VLAN part, you have multiple interfaces on the same subnet (10.100.100.0/24). This means you'll have to manually set up the route table by setting route priorities for each interface. NIC1 would have the lowest weight. But, having to do this usually means that something is wrong with the overall network design.

That said, I'd go yell at the network admin because in addition to the multiple IPs on the same subnet, the network is configured with the same subnet split among multiple VLANs. This is a routing table nightmare, because you'd need explicit routes to IPs that are on the local subnet.

For example, if a machine with IP 10.100.100.100/24 was on VLAN 401, then the VM you describe could not contact it, because it would ARP for the MAC address of 10.100.100.100 on NIC1, and get no response. You would have to have an explicit route to that host that uses NIC2.

 

[mobycl1ck]

Thanks for the quick response!
Well, you spotted right on, the network topology and configuration was never disclosed in details, hence my initial confusion and set up mishap. That would be corrected very soon, as the network admin will begin his work tomorrow.
Nevertheless, how can I set up a route priority among all NICs'?
To my knowledge, I set manually the NICs in Ubuntu, without any IP lease, and let VLANs having IPs. However, I need to set up a routing table to say which one gets to be the first in line.

 

[Rand__]

I would have thought that there is a typo in the IPs ?

NIC1 10.100.100.24/24 - VLAN 400
NIC2 10.100.100.96.24 - VLAN 401
NIC3 10.100.100.97.24 - VLAN 402
NIC 10.100.100.98.24 - VLAN 402

It might make more sense if it was

NIC1 10.100.100.24/24 - VLAN 400
NIC2 10.100.96.24/24 - VLAN 401
NIC3 10.100.97.24/24 - VLAN 402
NIC4 10.100.98.24/24 - VLAN 402

Else the question is whether VLAN assignment really is supposed /expected to be in the server admin's hands? Usually network admins want control over that...

So the q is whether your VMWare Nics have a trunk assigned (so you can define VLAN on OS side) or if you actually have four NICs with 4 different assigned VLANs?

 

[mobycl1ck]

Sorry, I was in a hurry to write the post and I wrote those IP with the typo you've spotted.
Yes, those are the IP handed.
The network guy was/is in vacation, so that was handed to me by another admin.

So the q is whether your VMWare Nics have a trunk assigned (so you can define VLAN on OS side) or if you actually have four NICs with 4 different assigned VLANs?

Well, that was on my mind also. Truth to be said, the other admin simply stated that i need to configure the VM like than, added the NICs through VSpere GUI and told me to set up those NICs with the aforementioned IPs.
No further explication, and no answers were given to my questions. I am new here, so that's that.
What I suspect is that there are 4 different networks defined and each one has a VLAN assigned to it, and the Ubuntu VM needs to communicate with all networks, hence the request.

 

[Rand__]

That would be dead simple as that would basically only mean to make sure you map the assigned Network to correct NIC and VLAN and then setup the proper IP on the correct NIC.
Thats easily accomplished if you can connect/disconnect each adapter from the VMWare gui, else you need to know the order or MACs associated.

 

[mobycl1ck]

So, basically take each MAC address for each NIC and set up the .yaml file accordingly and add the VLAN NIC trunks to each NIC?
Is that it?

 

[Rand__]

I assume you can add everything to one yaml for netplan with the different nic names, the mac to vmware nic to vlan mapping can be done outside ubuntu

 

[mobycl1ck]

the VLAN tags are already set up from vShere, when the other admin added the NICs, it showed the VLAN tag at the end of each NIC.
Basically, I need to set up each NIC, declare the VLANs, set each VLAN interface accordingly to the IP list and that's about it, right?

 

[zachj]

the VLAN tags are already set up from vShere, when the other admin added the NICs, it showed the VLAN tag at the end of each NIC.
Basically, I need to set up each NIC, declare the VLANs, set each VLAN interface accordingly to the IP list and that's about it, right?

If the vlan is specified in the virtual port group in vsphere then you don’t need to do anything to set vlan within Ubuntu; all you need to do is set IP address in net plan on each interface.

you can determine which interface should be assigned which ip address by looking in the vsphere UI (right click the VM and edit settings) to see for each of the four interfaces what MAC address is assigned to each port group.

 

[Rand__]

If the vlan is specified in the virtual port group in vsphere then you don’t need to do anything to set vlan within Ubuntu; all you need to do is set IP address in net plan on each interface.

you can determine which interface should be assigned which ip address by looking in the vsphere UI (right click the VM and edit settings) to see for each of the four interfaces what MAC address is assigned to each port group.

this

 

[mobycl1ck]

If the vlan is specified in the virtual port group in vsphere then you don’t need to do anything to set vlan within Ubuntu; all you need to do is set IP address in net plan on each interface.

you can determine which interface should be assigned which ip address by looking in the vsphere UI (right click the VM and edit settings) to see for each of the four interfaces what MAC address is assigned to each port group.

Thanks a bunch!
That clears a lot. However, I somehow think that two things happened.
One, the other admin somehow "tested" me to see how i would deal with the task, without saying anything else two - I showed how green and noob I am at this.
Oh well, lesson learned.

 

[nabsltd]

Nevertheless, how can I set up a route priority among all NICs'?

You can't. It's not possible with the listed config, since a "priority" won't work. You'll need explicit /32 routes to every other machine on the subnet.

And, this only means that you might be able to send a packet. You still might not be able to receive one, unless you enable ARP filtering on each interface. Without it, another machine could ARP for 10.100.96.24, and your machine could respond with the MAC address of the 10.100.100.24 interface, which isn't on the same VLAN.

 

[mobycl1ck]

Network guy came today at the office and the situation was cleared out in the most simple and "stupid"way. The VSphere has all the VLAN configuration, there is no need to set up that in a guest VM. Just the IP , gateway and DNS.
Next time, I'll make sure to ask twice, to avoid any other errors.

 

[Rand__]

So, as expected
Good luck with the next task