Redoing my Proxmox server

[Leemur]

Hi all,

I am in the process of re-doing my proxmox server and thought you guys might had some input.
I use it to run various game servers, web applications, Mail server, MariaDB, Emby and Nextcloud.

Current setup:
MB: SuperMicro X11SSH-LN4F
CPU: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
RAM: 64GB
HBA: 2x SAS2008 in IT mode
OS drives: 2x Kingston SSDNow V300 60GB
VM storage: 3x Kingston SSDNow V300 240GB, 3x Samsung SM863 240GB (ZFS RAID 10)
Data Storage: 10x WD Red Plus 6TB (WD60EFRX) (ZFS RAID 10)
Chassis: Shortened Norco RPC-4020 4U.

# zpool list -v
NAME                                                   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data                                                  27.3T  11.6T  15.7T        -         -    10%    42%  1.00x    ONLINE  -
  mirror                                              5.45T  2.34T  3.11T        -         -    11%  42.9%      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3408163              -      -      -        -         -      -      -      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3407159                -      -      -        -         -      -      -      -  ONLINE
  mirror                                              5.45T  2.32T  3.14T        -         -    10%  42.5%      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3408999              -      -      -        -         -      -      -      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3409356              -      -      -        -         -      -      -      -  ONLINE
  mirror                                              5.45T  2.32T  3.13T        -         -    11%  42.6%      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3407736              -      -      -        -         -      -      -      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3406823              -      -      -        -         -      -      -      -  ONLINE
  mirror                                              5.45T  2.31T  3.14T        -         -    10%  42.4%      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX31D3406038              -      -      -        -         -      -      -      -  ONLINE
    wwn-0x50014ee20a8e62fc                                -      -      -        -         -      -      -      -  ONLINE
  mirror                                              5.45T  2.31T  3.15T        -         -    10%  42.3%      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN1_WD-WX11D4454640              -      -      -        -         -      -      -      -  ONLINE
    ata-WDC_WD60EFRX-68MYMN0_WD-WX41D3408901              -      -      -        -         -      -      -      -  ONLINE
rpool                                                   55G  7.57G  47.4G        -         -    23%    13%  1.00x    ONLINE  -
  mirror                                                55G  7.57G  47.4G        -         -    23%  13.8%      -  ONLINE
    ata-KINGSTON_SV300S37A60G_50026B723107BC0C-part3      -      -      -        -         -      -      -      -  ONLINE
    ata-KINGSTON_SV300S37A60G_50026B723107BB18-part3      -      -      -        -         -      -      -      -  ONLINE
vm                                                     666G   461G   205G        -         -    45%    69%  1.00x    ONLINE  -
  mirror                                               222G   181G  40.9G        -         -    47%  81.6%      -  ONLINE
    wwn-0x5002538c40253a88                                -      -      -        -         -      -      -      -  ONLINE
    wwn-0x5002538c4015ca5a                                -      -      -        -         -      -      -      -  ONLINE
  mirror                                               222G   181G  40.8G        -         -    51%  81.6%      -  ONLINE
    wwn-0x5002538c40253a8d                                -      -      -        -         -      -      -      -  ONLINE
    wwn-0x5002538c4015ca5f                                -      -      -        -         -      -      -      -  ONLINE
  mirror                                               222G  98.6G   123G        -         -    39%  44.4%      -  ONLINE
    wwn-0x50026b77511344f1                                -      -      -        -         -      -      -      -  ONLINE
    wwn-0x50026b77511347ab                                -      -      -        -         -      -      -      -  ONLINE

ZFS Datasets:

NAME                        USED  AVAIL     REFER  MOUNTPOINT
data                       11.6T  14.8T      268G  /data
data/backup                2.12T  14.8T     2.12T  /data/backup
data/media                 8.56T  14.8T      446G  /data/media
data/media/movies          5.77T  14.8T     5.77T  /data/media/movies
data/media/music            155G  14.8T      155G  /data/media/music
data/media/tv-shows        2.02T  14.8T     2.02T  /data/media/tv-shows
data/media/upload           185G  14.8T      185G  /data/media/upload
data/vm                     670G  14.8T     16.0G  /data/vm
rpool                      7.57G  45.7G      104K  /rpool
vm                          461G   184G      323G  /vm
...

'data/backup' Is used by PBS installed directly on PVE
'data/media' NFS share to Emby and then shared trough SAMBA from within the VM
'data/vm/' contains Mount point for Nextcloud data
'vm' VM/Container storage

For backup I also run an offsite PBS that syncs with the on-site and backups the media folder.


Reasons for redoing:

• Upgrade to Proxmox VE 7
• Redo datasets
• easier management/handling of data shares

Possible solutions for data:

1. Share all data directly from Proxmox and use bind mount in containers
2. As above, but with LXC container for SMB server
3. Install TrueNAS in VM with HBA passthrough for data disks
4. Install TrueNAS in VM with HBA passthrough for VM and data disks
5. .... Your Idea ....

1 & 2 will probably be a pain with permissions
3 will let Proxmox handle everything related to the VM's and containers.
4 Dedicated NAS for all storage, Proxmox only have to handle the VM's and containers.

What are your thoughts, how would you use the existing hardware?
Let me know if you need more information