Proxmox 5.1 Boot drive size

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

chilipepperz

Active Member
Mar 17, 2016
212
64
28
54
Can I get away with mirrored 32GB SATA DOMs for Proxmox 5.1 boot drives? All VM's will run off NVMe but wanted to know if I could use the two orange connectors for drives.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Should be ok. I ran proxmox off a 30gb Intel sad - so I know it should fit.

Sent from my VS996 using Tapatalk
 
  • Like
Reactions: Patrick

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
Yep, I agree with @PigLover . You shouldn't be writing much data, if any at all, to your root OS partition. PID files, etc, should be going into RAM disk and logs should be written to your bulk storage or sent to a syslog server.
 

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
i only use a 16gb usb disk which is more than enough... just move /var/log /var/tmp /var/cache and swap onto other hdd pool will be fine
 

Rhinox

Member
May 27, 2013
144
26
18
Yep, I agree with @PigLover . You shouldn't be writing much data, if any at all, to your root OS partition. PID files, etc, should be going into RAM disk and logs should be written to your bulk storage or sent to a syslog server.
Unfortunatelly, this is not true for Proxmox. In standard configuration it writes to disk all the time. Not much, but there are small writes every 2-3 seconds, even when no VM is running. Those are killing consumer-SSDs very quickly...

BTW it is "Proxmox Cluster Filesystem" doing this. It writes its status periodically to /etc/pve. You can disable some of this (pve-ha-lrm/pve-ha-crm) if you are not using cluster-fs, but not everything. I would not recommend installing Proxmox on DOM. Those are usually aimed for ESXi, not for high write-ops...
 
  • Like
Reactions: lowfat

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
Thank you @Rhinox for this (somewhat surprising) information. I'm new to Proxmox and I was planning on installing it on my main virt host, but now that I've read this I will reconsider the disk configuration or the OS/hypervisor.
 

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
I have been using proxmox on USB dom on production for more then 2 yrs with more 30+ node ... i use Sandisk ultrafit using mirror Disks
Sure thing. That's good anecdotal evidence, and I'd probably have a similar experience. The reason it's a concern for me is that frequent writing to consumer-grade USB sticks is not what I would consider a best practice... especially if you're trying to use the hypervisor in an enterprise environment.

I would imagine that flash memory controllers are a lot better than they used to be, with better wear-levelling, etc. So this problem is probably not as acute as it used to be, but it's still not good practice.

Finally, the reason why this is important is that if you configure and then send it off to some remote location for colocation, the last thing you want is extra, unnecessary problems caused by USB sticks wearing out like this.

We need a Debian distribution that does minimal writing when the OS is running on a flash device. I guess this could be achieved withj some combination of OverlayFS, tmpfs, SquashFS, JFFS2, and UBIFS. Does anybody know if such a project already exists? The closest I can think of, off the top of my head, is Tails. I understand that NanoBSD does something similar to this already.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
I have been using proxmox on USB dom on production for more then 2 yrs with more 30+ node ... i use Sandisk ultrafit using mirror Disks
What is the risk of running Proxmox on USB drives? (barring @llbeking opinion) I am also thinking of using mirror USB drives. Is it worthwhile changing the USB drives after a fixed period of time regardless of whatever wear it has?
 

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
I will not argue with what is best practice or not.

Just let you know, it can be done and its very reliable in a offsite co-location IDC.

A good USB DOM can handle Proxmox VE Cluster as I stayed you need to move
  • /var/log
  • /var/cache
  • /var/tmp
  • /var/lib/docker (if you are also running docker)
  • swap
There are reliable SLC chip USB DOM (cost USD 25)
upload_2018-1-18_16-10-0.png

and also my favor dual SanDisk Ultra Fit. easily found anywhere
upload_2018-1-18_16-10-9.png

What I found most troubling issue with ANY USB DOM is its over power protection issue and never is the write wear-out failure

usb device descriptor read/64, error -32 typically caused by bad/incompatible USB 3.0 controller on the motherboard, so that's why test your USB DOM first.

if you run NETDATA or other monitoring app, you will find once Proxmox is booted, there is very very little io goes to the root fs and that is a fact. So why waste a sata port when a USB A connector on the motherboard can do the job.
 
Last edited:
  • Like
Reactions: epicurean

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
A good USB DOM can handle Proxmox VE Cluster as I stayed you need to move
  • /var/log
  • /var/cache
  • /var/tmp
  • /var/lib/docker (if you are also running docker)
  • swap
In other words, an "out of the box" Promox installation is insufficient for protection against wearing out the flash, and needs to be configured as above? Is this written anywhere in the Proxmox documentation? Can you please provode a link? Remember, even though I am highly experienced with Debian, I am a complete newbie to Proxmox.

There are reliable SLC chip USB DOM (cost USD 25)
I have one of these cheap USB DOMs but I haven't used it yet. I don't know if it's SLC (I would presume MLC due to the price). Have you got a link to yours please?

In the screenshot you posted, what is "eVtran V03S SLC"?

and also my favor dual SanDisk Ultra Fit. easily found anywhere
I love these USB sticks and I often use them as my OS boot drive.

What I found most troubling issue with ANY USB DOM is its over power protection issue and never is the write wear-out failure

usb device descriptor read/64, error -32 typically caused by bad/incompatible USB 3.0 controller on the motherboard, so that's why test your USB DOM first.
This is new information to me as it is very useful. I didn't know about this at all.

if you run NETDATA or other monitoring app, you will find once Proxmox is booted, there is very very little io goes to the root fs and that is a fact. So why waste a sata port when a USB A connector on the motherboard can do the job.
But doesn't this contract what was stated above? In other words, see the following quote:

In standard configuration it writes to disk all the time. Not much, but there are small writes every 2-3 seconds, even when no VM is running. Those are killing consumer-SSDs very quickly...
@Rhinox is talking about "standard configuration", whereas I get the impression you are talking about a non-standard configuration as per your list of file systems and mount points above. Is this correct?
 

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
In other words, an "out of the box" Promox installation is insufficient for protection against wearing out the flash, and needs to be configured as above? Is this written anywhere in the Proxmox documentation? Can you please provode a link? Remember, even though I am highly experienced with Debian, I am a complete newbie to Proxmox.
netdata provides very good monitoring of the system

I have one of these cheap USB DOMs but I haven't used it yet. I don't know if it's SLC (I would presume MLC due to the price). Have you got a link to yours please?

In the screenshot you posted, what is "eVtran V03S SLC"?
if you google it, you find evtran store right away. https://www.aliexpress.com/store/2842018 now is @ usd 16 and I always take off its plastic over



@Rhinox is talking about "standard configuration", whereas I get the impression you are talking about a non-standard configuration as per your list of file systems and mount points above. Is this correct?
What Rhinox saying are log info write to /var/log/corosync directory and that's why I said you need to move entire /var/log with PERMISSION intact to your hdd pool

/var/cache is used for apt download and compile

/var/tmp is used for pve web gui for uploading iso temp holder directory

and for swap, please follow HOWTO use a zvol as a swap device · zfsonlinux/pkg-zfs Wiki · GitHub if you are using zfs pool.

I found a lot of USB 3.0 mobo host would over power the usb dom... in order to avoid these error -32 I would use evtran ssd or just plug usb dom to usb 3.1 port or usb 2.0 port.

and they works in my own environment very stable. and I can't vouch for other people's environment. I do have them on company production server running many vms and lxc and the configuration is rock solid.
 
Last edited:

Jon Massey

Active Member
Nov 11, 2015
339
82
28
37
Dual Sandisk Ultra Fit 32GB for me in ZFS mirror so if one does shit the bed I've got half a chance to swap it out.
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
We need a Debian distribution that does minimal writing when the OS is running on a flash device. I guess this could be achieved withj some combination of OverlayFS, tmpfs, SquashFS, JFFS2, and UBIFS. Does anybody know if such a project already exists? The closest I can think of, off the top of my head, is Tails. I understand that NanoBSD does something similar to this already.
Not exactly Debian, but ESOS does a good job running from USB, completely unmounts and only writes are when the current config is changed. (USB mounted, config written, unmount)
 
  • Like
Reactions: EluRex

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
@EluRex Thank you for the useful information and interesting discussion.

@Jon Massey I agree. My opinion is that if you're using consumer-grade USB sticks on a production server then running a mirrored RAID configuration is pretty fundamental.

@_alex I never heard of ESOS before but it looks interesting and implements several of the principles that I have in mind. Thank you for the recommendation.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
Dual Sandisk Ultra Fit 32GB for me in ZFS mirror so if one does shit the bed I've got half a chance to swap it out.
Do you also move these out of the usb drives?
  • /var/log
  • /var/cache
  • /var/tmp
  • /var/lib/docker (if you are also running docker)
  • swap