I've been running a home server off Proxmox since roughly v4.4 and overall things have been great, until lately. The more i/o intensive things I add into my current PVE node the more my nodes shortcomings are really starting to bite me. Ideally I'd like to correct this before the house of cards really comes down. Apologies this might be a little long winded, just attempting to provide as much detail as possible in hopes for some solid advice on how to fix and learn from my mistakes.
Basically what I'm running into is with things like my NVR storage, Plex, and a few other apps that regularly access data on my storage pool the io wait times on proxmox are extraordinarily high often around 30%, sometimes even peaking at 90%, hence why I'd like to break apart my singular large storage pool into smaller vdevs with dedicated functions to hopefully keep read/write and io wait times down a little.
Goal:
This seems like it should be pretty straight forward on paper, with exception of a few issues. I want a fresh Proxmox install on a 1u ztsystems server I procured, and convert my current supermicro 4u proxmox node into the TrueNAS server. The issue is, I have no idea what the best way to go about doing this actually is. Is it possible (more specifically, is it SAFE!!) to shut down my current proxmox server, pull all of the disks out, throw in a pair of new disks, install TrueNAS, reinstall the disks and import my entire existing storage pool? I realize in a perfect world, I'd have the current proxmox node running, spin up a brand new server w/disks to install TrueNAS on, then migrate the data across the network but the issue with that is I don't have a second supermicro 4u server or another 50tb of cold spare disks laying around and frankly with some house repairs looming, won't have the budget to pull that off anytime soon.
Secondary question:
Basically what I'm running into is with things like my NVR storage, Plex, and a few other apps that regularly access data on my storage pool the io wait times on proxmox are extraordinarily high often around 30%, sometimes even peaking at 90%, hence why I'd like to break apart my singular large storage pool into smaller vdevs with dedicated functions to hopefully keep read/write and io wait times down a little.
Goal:
- Single Standalone PVE 8.0 node with ample ram and local kvm storage, bare metal on enterprise SSD
- 1u server, 2x e5-2600 cpus, 128gb ram
- 10gb fiber
- Single TrueNAS (scale?) node managing multiple zfs pools of disks
- 4u supermicro chassis, 2x e5-2600 cpus, 256gb ram
- 10gb fiber
- Dedicated pools for:
- Plex
- NVR
- Short term storage
- Long term storage
- Single Standalone PVE 6.4 node, 256gb ram
- 4u supermicro chassis, 2x e5-2600 cpus, 256gb ram
- 2x bonded gigabit copper
- Bare Metal: 2x 2tb 7200 SAS disks (single mirrored vdev)
- NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000cca01b77db50-part3 ONLINE 0 0 0
scsi-35000cca01cb78cb4-part3 ONLINE 0 0 0
- NAME STATE READ WRITE CKSUM
- Storage Pool: 50TB comprised of various 5400/7200 SAS/SATA disks in matches pairs of mirrored vdev, various folders shared via SMB
- NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_7HKRZVRJ ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_7HKS4R5J ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_1SJKAEMZ ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_1DGEZHWZ ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_1SJ7VDSZ ONLINE 0 0 0
ata-WDC_WD80EMAZ-00WJTA0_1SGZNRRZ ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
ata-WDC_WD120EMFZ-11A6JA0_9JGP9JAT ONLINE 0 0 0
ata-WDC_WD120EMFZ-11A6JA0_9JG498KT ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
scsi-35000cca2523fd390 ONLINE 0 0 0
scsi-35000cca25221fea4 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
scsi-35000cca252244af8 ONLINE 0 0 0
scsi-35000cca25221f850 ONLINE 0 0 0
- NAME STATE READ WRITE CKSUM
This seems like it should be pretty straight forward on paper, with exception of a few issues. I want a fresh Proxmox install on a 1u ztsystems server I procured, and convert my current supermicro 4u proxmox node into the TrueNAS server. The issue is, I have no idea what the best way to go about doing this actually is. Is it possible (more specifically, is it SAFE!!) to shut down my current proxmox server, pull all of the disks out, throw in a pair of new disks, install TrueNAS, reinstall the disks and import my entire existing storage pool? I realize in a perfect world, I'd have the current proxmox node running, spin up a brand new server w/disks to install TrueNAS on, then migrate the data across the network but the issue with that is I don't have a second supermicro 4u server or another 50tb of cold spare disks laying around and frankly with some house repairs looming, won't have the budget to pull that off anytime soon.
Secondary question:
Is there a better way to arrange vdevs to maximize storage capacity yet still have ample safety net if a drive is lost? Thankfully, *knock on wood* I've only had a single disk in a couple of my vdevs have issues, and removing/reinserting the disks allowing a resilver seems to have "corrected" the issues and the pool remains stable.
Thanks in advance!