Proxmox isolating vm from host, hyper & user (network and volumes)

[vingo]

Need to run some sketchy applications in a linux or windows vm, let's call it SketchVM.

What I want to accomplish:
- SketchVM to have access to the internet.
-Malware on SketchVM cannot infect proxmox hypervisor
-Malware on SketchVM cannot infect other vms or write to their drives
-Malware on SketchVM cannot infect the pc, that I use SketchVM with, through proxmox web portal.
-Malware on SketchVM cannot cause havoc on my LAN
-Malware on SketchVM does not infect my physical pfsense edgebox

I have no idea where to even get started, to create several of these VMs, that can run an OS + install apps + reach out to the internet.

Please let me know how I may achieve the above mentioned goal. Thank you !

 

[vingo]

PS: Just forecasting the " dont run sketchy apps " , yeah I need to, just safely : )

 

[CyklonDX]

1) enable your cpu mitigations; newer cpu the better...
2) pass whole disk/device to the kvm / some1 smart could exploit reading virtio format / uefi or something to get outside or onto other vms.
3) use separate bios image - just for this box.
4) create new internal lan for this box - so that it can access "internet" but cannot access anything within other networks. You can do it by setting up vlans or just set up different network subnets. Potentially you would want it to go directly to a vpn so it would mask your ip.
5) don't use web portal as remote desktop solution. Someone smart could exploit the vnc protocol to execute stuff.
6) enable TPM.


This is to run sketchy chinese/iranian/israeli viruses.

 

[vingo]

1) enable your cpu mitigations; newer cpu the better...
2) pass whole disk/device to the kvm / some1 smart could exploit reading virtio format / uefi or something to get outside or onto other vms.
3) use separate bios image - just for this box.
4) create new internal lan for this box - so that it can access "internet" but cannot access anything within other networks. You can do it by setting up vlans or just set up different network subnets. Potentially you would want it to go directly to a vpn so it would mask your ip.
5) don't use web portal as remote desktop solution. Someone smart could exploit the vnc protocol to execute stuff.
6) enable TPM.


This is to run sketchy chinese/iranian/israeli viruses.

Thanks a lot for helping and taking the time to write the base list. I have a few questions:

1) -> Yes noted, lscpu shows apllied migitations

2) -> ideally I will have many SketchVMs (30-50), is there a way to isolate VM drives from hyper and from each other without passing throughof a physical drive, as passing through that many drives would be VERY costly ?

3) -> For which box, theSketchVM ? Do a Pixi boot, separate image boot ? Please clarify what you mean.

4) -> Can send a bunch of Vlans over from pfsense phsyical box, but isthere a way to accomplish this w/ proxmox alone ?

5) -> Will have to use the SketchVMs, if not web portal what is a more secure way to do ?

6) -> TPM for VMs, Hyper ?

Please have a look and let me know what you think.

 

[CyklonDX]

2) -> ideally I will have many SketchVMs (30-50), is there a way to isolate VM drives from hyper and from each other without passing throughof a physical drive, as passing through that many drives would be VERY costly ?

Its up to you - there are plenty ways to get around that; question is willing to do research as i have only idea about it within normal kvm not proxmox. (partitions, zfs iscsi pooling etc..., or you think its ok for it to run in virtio image/raw)

3) -> For which box, theSketchVM ? Do a Pixi boot, separate image boot ? Please clarify what you mean.

Each risky vm should have their own efi image file. (note this is just example)

(in case of using tpm it would be bit different.)

4) -> Can send a bunch of Vlans over from pfsense phsyical box, but isthere a way to accomplish this w/ proxmox alone ?

Yes, you can tag your network configuration with vlan, in proxmox (you need to research it.) You can create vmbridges or whatever you want...

5) -> Will have to use the SketchVMs, if not web portal what is a more secure way to do ?

RDP, or some custom software like parsec, teamviewer and so on...

6) -> TPM for VMs, Hyper ?

Yes. You can even create your own signing certs, or even limit applications that can be ran on the system if you feel anal.

 

[vingo]

Its up to you - there are plenty ways to get around that; question is willing to do research as i have only idea about it within normal kvm not proxmox. (partitions, zfs iscsi pooling etc..., or you think its ok for it to run in virtio image/raw)



Each risky vm should have their own efi image file. (note this is just example)
View attachment 27044
(in case of using tpm it would be bit different.)




Yes, you can tag your network configuration with vlan, in proxmox (you need to research it.) You can create vmbridges or whatever you want...



RDP, or some custom software like parsec, teamviewer and so on...



Yes. You can even create your own signing certs, or even limit applications that can be ran on the system if you feel anal.

Had thought about one piece of the physical HDD/SSD isolation:
-To avoid having to purchase 30 physical drives to pass through, is there a way to virtualise drives or partitions for pass through instead ?
-What is the best way to run install, on hdd for proxmox install and another for VMs ?
-Does the separate physical drive for VMs require a common swap space? What parts of the VMs need to communicate with the hypervisor, is there any reason for those drives to access each other and how can they be separated?

 

[CyklonDX]

Well its something you have to decide - what do you want to trade - what is acceptable, how far do you want to go. There will always be a way get around things.

The virtio are the virtualized disk/s into image file. You can use them they aren't bad as i've described. Its just in realms of possible.

You can passthrough partitions by label or uuid, its a lot of work though, and doesn't provide protection for other systems on the disk.
The whole idea of not sharing disk and directly mounting the port was to protect you from some kind of disk cache exploit. Same approach as meltdown and spectre attacks; so its pointless at that point; and virtio is better approach.

-What is the best way to run install, on hdd for proxmox install and another for VMs ?

If i wanted to be super secure, i would :

mount most of the main system folders in read-only mode. (by changing fstab), and potentially made them on separate 'disks' with different encryption on them - to throw a wench if you ever wanted to fix something or someone trying to read anything; next get SD card with bios for kvm's - and common files like grub. After putting your bios on it; physically make it read-only



The main point of attacks on vms is going to be country/gov level of attack. Like glow letters.
1) internal network (your bridges and so on)
2) bios/efi (quite often its done first as its oldest type of an attack)
3) kernel & filesystem (this goes into 4, 5, 6 in certain forms, as its possible to include machine code when read for even host machine to do something its not supposed to do - same rules apply on subnetting how cisco switches/routers read packets - you can create faulty subnets and send packets that will penetrate/get inside your network as bit wise the cpu in cisco switches will switch that packet to go somewhere else where it should never go.)
4) graphics cards bios (you can protect that by specifying bios rom to the gpu when starting)
5) memory
6) memory channels
7) cpu caches (4, 5, 6 are part of spectre/meltdown type of an attacks)
8) disk caches (while virtio keeps the process isolated, normal disk shares disk cache between all partitions)

There are more human vectors of attack. Nothing is 100% safe.

 

[vingo]

Well its something you have to decide - what do you want to trade - what is acceptable, how far do you want to go. There will always be a way get around things.

The virtio are the virtualized disk/s into image file. You can use them they aren't bad as i've described. Its just in realms of possible.

You can passthrough partitions by label or uuid, its a lot of work though, and doesn't provide protection for other systems on the disk.
The whole idea of not sharing disk and directly mounting the port was to protect you from some kind of disk cache exploit. Same approach as meltdown and spectre attacks; so its pointless at that point; and virtio is better approach.




If i wanted to be super secure, i would :

mount most of the main system folders in read-only mode. (by changing fstab), and potentially made them on separate 'disks' with different encryption on them - to throw a wench if you ever wanted to fix something or someone trying to read anything; next get SD card with bios for kvm's - and common files like grub. After putting your bios on it; physically make it read-only

View attachment 27120

The main point of attacks on vms is going to be country/gov level of attack. Like glow letters.
1) internal network (your bridges and so on)
2) bios/efi (quite often its done first as its oldest type of an attack)
3) kernel & filesystem (this goes into 4, 5, 6 in certain forms, as its possible to include machine code when read for even host machine to do something its not supposed to do - same rules apply on subnetting how cisco switches/routers read packets - you can create faulty subnets and send packets that will penetrate/get inside your network as bit wise the cpu in cisco switches will switch that packet to go somewhere else where it should never go.)
4) graphics cards bios (you can protect that by specifying bios rom to the gpu when starting)
5) memory
6) memory channels
7) cpu caches (4, 5, 6 are part of spectre/meltdown type of an attacks)
8) disk caches (while virtio keeps the process isolated, normal disk shares disk cache between all partitions)

There are more human vectors of attack. Nothing is 100% safe.

Wow really appreciate the help and your time spent replying.

Clever approach, assuming that the switch does indeed write protect as intended.

Do you mind if post a few questions along the way as I implement?

 

[CyklonDX]

sure. Go ahead.

 

[Pete.S.]

I would run all the sketchy VMs on it's own sketchy hypervisor and have the hypervisor connected straight to pfsense on it's own dedicated interface.

Problem with complicated setups is that misconfiguration is a much bigger risk than anything else.
This type of physical separation automatically takes care of a lot of problems and risk with logical separation and potential misconfiguration.

Next level of physical separation would be to not run the sketchy VMs at all in house. Run it somewhere else. Bare metal servers and also VMs can be rented at low cost so that might be an option. But I'd just run it on it's own hardware and, from a security point of view, treat it as already compromised.

 

[CyklonDX]

You can isolate them physically by having more than 1 physical nic's (not one going off mobo) that has different pci id; then you can isolate it more neatly without relying on virtualization.

I would advise to configure them with vpn on switch/router where it would be connected - this way they won't be exposed to your network at all.
But you will have to secure that router port; i.e. you need to make sure this port doesn't broadcast any routing information, and/or switching tables. (so all packets trying to go there would get discarded, and vice versa - and you can also make a 'trap' network - that is exposed but presents nothing usable - a simple vm that acts as if it was a router but isn't - i recommend making it look as if it was some kind of CIA/FBI honeypot - i.e. add graphics and logos of those agencies.)

 

[zer0sum]

We really need a bit more detail on SketchVM and how it's getting infected. What are the exact attack surfaces and vectors involved?

I like a little security by obscurity at times, so I'd do something like this:

• Proxmox base operating system
  • nested ESXi with hardware passthrough of disks and network card
    • Opnsense firewall as gateway
      • configured with wireguard tunnel to throwaway vps
      • all traffic goes over tunnel as default gw
    • Windows hosts as SketchVM's
      • Comodo AV with sandbox settings to run all windows PE's isolated
      • gateway is pfsense firewall

That would be a super confusing setup to hack and you'd need to escape the Comodo sandbox (highly unlikely), and then a nested ESXi instance.

There are obviously a few ways you can accomplish things like this, but it really does depend on the exact threat vectors

 

[CyklonDX]

We really need a bit more detail on SketchVM and how it's getting infected. What are the exact attack surfaces and vectors involved?

OP wanted to build a tight ship for things he cannot think off - and what potentially wasn't found (publicized) yet; for lets say installing chinese gov malware and such.