Proxmox 5.1 Boot drive size

Discussion in 'Linux Admins, Storage and Virtualization' started by chilipepperz, Dec 5, 2017.

  1. chilipepperz

    chilipepperz Active Member

    Joined:
    Mar 17, 2016
    Messages:
    169
    Likes Received:
    50
    Can I get away with mirrored 32GB SATA DOMs for Proxmox 5.1 boot drives? All VM's will run off NVMe but wanted to know if I could use the two orange connectors for drives.
     
    #1
  2. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,675
    Likes Received:
    1,049
    Should be ok. I ran proxmox off a 30gb Intel sad - so I know it should fit.

    Sent from my VS996 using Tapatalk
     
    #2
    Patrick likes this.
  3. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Yep, I agree with @PigLover . You shouldn't be writing much data, if any at all, to your root OS partition. PID files, etc, should be going into RAM disk and logs should be written to your bulk storage or sent to a syslog server.
     
    #3
  4. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    70
    i only use a 16gb usb disk which is more than enough... just move /var/log /var/tmp /var/cache and swap onto other hdd pool will be fine
     
    #4
  5. Rhinox

    Rhinox Member

    Joined:
    May 27, 2013
    Messages:
    144
    Likes Received:
    24
    Unfortunatelly, this is not true for Proxmox. In standard configuration it writes to disk all the time. Not much, but there are small writes every 2-3 seconds, even when no VM is running. Those are killing consumer-SSDs very quickly...

    BTW it is "Proxmox Cluster Filesystem" doing this. It writes its status periodically to /etc/pve. You can disable some of this (pve-ha-lrm/pve-ha-crm) if you are not using cluster-fs, but not everything. I would not recommend installing Proxmox on DOM. Those are usually aimed for ESXi, not for high write-ops...
     
    #5
  6. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Thank you @Rhinox for this (somewhat surprising) information. I'm new to Proxmox and I was planning on installing it on my main virt host, but now that I've read this I will reconsider the disk configuration or the OS/hypervisor.
     
    #6
  7. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    70
    I have been using proxmox on USB dom on production for more then 2 yrs with more 30+ node ... i use Sandisk ultrafit using mirror Disks
     
    #7
    _alex and Jon Massey like this.
  8. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Sure thing. That's good anecdotal evidence, and I'd probably have a similar experience. The reason it's a concern for me is that frequent writing to consumer-grade USB sticks is not what I would consider a best practice... especially if you're trying to use the hypervisor in an enterprise environment.

    I would imagine that flash memory controllers are a lot better than they used to be, with better wear-levelling, etc. So this problem is probably not as acute as it used to be, but it's still not good practice.

    Finally, the reason why this is important is that if you configure and then send it off to some remote location for colocation, the last thing you want is extra, unnecessary problems caused by USB sticks wearing out like this.

    We need a Debian distribution that does minimal writing when the OS is running on a flash device. I guess this could be achieved withj some combination of OverlayFS, tmpfs, SquashFS, JFFS2, and UBIFS. Does anybody know if such a project already exists? The closest I can think of, off the top of my head, is Tails. I understand that NanoBSD does something similar to this already.
     
    #8
  9. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    512
    Likes Received:
    18
    What is the risk of running Proxmox on USB drives? (barring @llbeking opinion) I am also thinking of using mirror USB drives. Is it worthwhile changing the USB drives after a fixed period of time regardless of whatever wear it has?
     
    #9
  10. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    70
    I will not argue with what is best practice or not.

    Just let you know, it can be done and its very reliable in a offsite co-location IDC.

    A good USB DOM can handle Proxmox VE Cluster as I stayed you need to move
    • /var/log
    • /var/cache
    • /var/tmp
    • /var/lib/docker (if you are also running docker)
    • swap
    There are reliable SLC chip USB DOM (cost USD 25)
    upload_2018-1-18_16-10-0.png

    and also my favor dual SanDisk Ultra Fit. easily found anywhere
    upload_2018-1-18_16-10-9.png

    What I found most troubling issue with ANY USB DOM is its over power protection issue and never is the write wear-out failure

    usb device descriptor read/64, error -32 typically caused by bad/incompatible USB 3.0 controller on the motherboard, so that's why test your USB DOM first.

    if you run NETDATA or other monitoring app, you will find once Proxmox is booted, there is very very little io goes to the root fs and that is a fact. So why waste a sata port when a USB A connector on the motherboard can do the job.
     
    #10
    Last edited: Jan 18, 2018
    epicurean likes this.
  11. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    In other words, an "out of the box" Promox installation is insufficient for protection against wearing out the flash, and needs to be configured as above? Is this written anywhere in the Proxmox documentation? Can you please provode a link? Remember, even though I am highly experienced with Debian, I am a complete newbie to Proxmox.

    I have one of these cheap USB DOMs but I haven't used it yet. I don't know if it's SLC (I would presume MLC due to the price). Have you got a link to yours please?

    In the screenshot you posted, what is "eVtran V03S SLC"?

    I love these USB sticks and I often use them as my OS boot drive.

    This is new information to me as it is very useful. I didn't know about this at all.

    But doesn't this contract what was stated above? In other words, see the following quote:

    @Rhinox is talking about "standard configuration", whereas I get the impression you are talking about a non-standard configuration as per your list of file systems and mount points above. Is this correct?
     
    #11
  12. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    70
    netdata provides very good monitoring of the system

    if you google it, you find evtran store right away. https://www.aliexpress.com/store/2842018 now is @ usd 16 and I always take off its plastic over

    [​IMG]


    What Rhinox saying are log info write to /var/log/corosync directory and that's why I said you need to move entire /var/log with PERMISSION intact to your hdd pool

    /var/cache is used for apt download and compile

    /var/tmp is used for pve web gui for uploading iso temp holder directory

    and for swap, please follow HOWTO use a zvol as a swap device · zfsonlinux/pkg-zfs Wiki · GitHub if you are using zfs pool.

    I found a lot of USB 3.0 mobo host would over power the usb dom... in order to avoid these error -32 I would use evtran ssd or just plug usb dom to usb 3.1 port or usb 2.0 port.

    and they works in my own environment very stable. and I can't vouch for other people's environment. I do have them on company production server running many vms and lxc and the configuration is rock solid.
     
    #12
    Last edited: Jan 18, 2018
  13. Jon Massey

    Jon Massey Active Member

    Joined:
    Nov 11, 2015
    Messages:
    338
    Likes Received:
    80
    Dual Sandisk Ultra Fit 32GB for me in ZFS mirror so if one does shit the bed I've got half a chance to swap it out.
     
    #13
  14. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    851
    Likes Received:
    89
    Not exactly Debian, but ESOS does a good job running from USB, completely unmounts and only writes are when the current config is changed. (USB mounted, config written, unmount)
     
    #14
    EluRex likes this.
  15. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    @EluRex Thank you for the useful information and interesting discussion.

    @Jon Massey I agree. My opinion is that if you're using consumer-grade USB sticks on a production server then running a mirrored RAID configuration is pretty fundamental.

    @_alex I never heard of ESOS before but it looks interesting and implements several of the principles that I have in mind. Thank you for the recommendation.
     
    #15
  16. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    70
    esos I like it very much.... it is SCST (very essential for SRPT and iSER rdma) type of storage linux distribution.
     
    #16
  17. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    512
    Likes Received:
    18
    Do you also move these out of the usb drives?
    • /var/log
    • /var/cache
    • /var/tmp
    • /var/lib/docker (if you are also running docker)
    • swap
     
    #17
Similar Threads: Proxmox Boot
Forum Title Date
Linux Admins, Storage and Virtualization UEFI booting Proxmox 5.2 on ZFS? Aug 23, 2018
Linux Admins, Storage and Virtualization Migrate proxmox to new bootdiscs Apr 14, 2018
Linux Admins, Storage and Virtualization Proxmox NVIDIA GPU passthrough Nov 14, 2018
Linux Admins, Storage and Virtualization Proxmox Mail gateway Relay? Nov 13, 2018
Linux Admins, Storage and Virtualization Advice needed: Proxmox vs vSphere Oct 15, 2018

Share This Page