Proxmox VE Open vSwitch (OVS) Discussion

[EluRex]

Any tips on setting that up?

It's very tricky and not as straight forward using ovs-vsctl commands. It must be done in the gui and check make sure vmbr0 are both active and autostart (using the command method, I can't never get the bridge active). All configs are written into /etc/network/interfaces and sometime requires small modification


Right now I am trying to create a linux bond and attache it to a OVS bridge on another node, so far with no success, I guess I need to use OVS Bond instead.

Lastly, don't create any vm until you have setup the cluster or otherwise its impossible to create a cluster with nodes that has a same vmid. Also, once the cluster is created, you cannot change the hostname nor its IP (now I understand the beauty of openstack) SD Networking on pve 4.0 is very tedious

For IB ports, I am trying to get sr-iov working so that I don't have to use a linux bridge for the IB to eth

 

[canta]

And that is other than doing:
apt-get install openvswitch-switch -y

yap , OVS will be installed ..

I did manually to install OVS

 

[EluRex]

ovs bond and bridge issue solved.... for proxmox an ip must be assign to a bridge interface in order for it to be active.

 

[Patrick]

ovs bond and bridge issue solved.... for proxmox an ip must be assign to a bridge interface in order for it to be active.

View attachment 1080

Maybe a dumb question, but how did you swap the normal bridge for the OVS bridge? I am a bit worried that I will lose connection by doing so and already have the default gateway on the standard bridge.

 

[EluRex]

Maybe a dumb question, but how did you swap the normal bridge for the OVS bridge? I am a bit worried that I will lose connection by doing so and already have the default gateway on the standard bridge.

For eth0 steps are

1. Remove the All IP Info from the eth0
   
2. Delete the vmbr0 (Linux bridge)
   
3. Create vmbr0 as ovs-bridge with IP/MASK/Gateway & slave ports [eth0]
4. Add First OvsIntPort same ip as ovs-bridge but without Gateway (typically it is the default vlan 1)
5. For VLAN only, may skip this step. Add Second OvsIntPort with different subnet mask and ip and VLAN Tag 100 or whatever
6. Reboot and only at this point your network settings will take effect. This is because all the config is done at /etc/network/interfaces
   
7. If network setting failed and you lost your connections to ssh or 8006, then, you will need to use ipmi and then edit /etc/network/interface just config auto lo and auto eth0 only and reconfigure it from step 1 again.

For bond0 steps are

1. Remove the All IP Info from the eth0 eth1
   
2. Delete the vmbr0 (Linux bridge)
   
3. Create bond0 with no ip information and slave ports eth0 eth1 and bond_mode=balance-slb lacp=active (for my case)
   
4. Create vmbr0 as ovs-bridge with IP/MASK/Gateway & slave ports [bond0]
5. Add First OvsIntPort same ip as ovs-bridge but without Gateway (typically it is the default vlan 1)
6. For VLAN only, may skip this step. Add Second OvsIntPort with different subnet mask and ip and VLAN Tag 100 or whatever
7. Reboot and only at this point your network settings will take effect. This is because all the config is done at /etc/network/interfaces
   
8. If network setting failed and you lost your connections to ssh or 8006, then, you will need to use ipmi and then edit /etc/network/interface just config auto lo and auto eth0 only and reconfigure it from step 1 again.

 

[rubylaser]

I normally just do all the configuration via /etc/network/interfaces and don't even bother with the web GUI for this part. I comment out the current bond0 in /etc/network/interfaces add my Openvswitch configuration and reboot once and your vms should just use the OVS bridge without needing any further config. The Proxmox wiki actually has a good number of standard configurations to look at. Example 3 is setup like the default bond interface except that it uses a bond instead of a single interface.

Open vSwitch - Proxmox VE

 

[EluRex]

if you don't use gui... I thought you need to ovs-vsctl addport xxxx those commands

 

[rubylaser]

if you don't use gui... I thought you need to ovs-vsctl addport xxxx those commands

Nope, you can add them right into /etc/network/interfaces and have them just work on Proxmox (this is how I did mine). I didn't do any of the ovs commands from the CLI to get this working

 

[Patrick]

Not going to lie here. I am in the Fremont DC. Messed up something when I did the OVS on the first node and killed the entire test network there

 

[rubylaser]

Not going to lie here. I am in the Fremont DC. Messed up something when I did the OVS on the first node and killed the entire test network there

I assume you can still connect via IPMI and roll back to the Linux bridge configuration to at least get networking going again. Also, what are you trying to accomplish with your OVS setup (LACP, VLANing, etc). And, what does your config look like?

 

[Patrick]

I was just hoping to replicate the standard bridge and eventually do VLAN'ing. Whatever I did literally took down the network so I ended up just re-installing proxmox on that node and things are working.

Also thinking I may put this all into a new thread.

 

[Patrick]

Making this its own thread regarding Proxmox VE Open vSwitch configuration.

 

[EluRex]

OVS is a bit tricky with PVE.... it happened to me once and I end up have to reinstall... I found PVE by itself is greate... but if you want your pve to have ovs, cluster + other adv feature... it becomes very tricky!

 

[rubylaser]

I was just hoping to replicate the standard bridge and eventually do VLAN'ing. Whatever I did literally took down the network so I ended up just re-installing proxmox on that node and things are working.

Also thinking I may put this all into a new thread.

Before you do anything with you /etc/network/interfaces, you should always backup the current config so you can easily rollback changes. There shouldn't be a need to completely re-install Proxmox just for a network config screwup

Here's what I have in my /etc/network/interfaces. This is similar to the default Linux bonding setup except, I'm bonding 4 nics.

# Bond Intel nics together
allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eth1 eth2 eth3 eth4
        ovs_options bond_mode=balance-slb vlan_mode=native-untagged

# Bridge for our bond and vlan virtual interfaces (our VMs attach to this bridge)
auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan1

# Virtual interface untagged traffic
allow-vmbr0 vlan1
iface vlan1 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options vlan_mode=access
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 10.1.1.2
        netmask 255.255.255.0
        gateway 10.1.1.1
        dns-nameservers 10.1.1.1 8.8.8.8 8.8.4.4