Hi all,
In the last few weeks, I’ve been learning about “homelab” and how to make one from scratch.
Been using linux daily and on clusters for years but "using" is the keyword. Having built all my PCs in the last 20 years or so, I have never built a machine with the specs below and set up anything like this.
Just looking for a quick review and suggestions on my notes below.
This HPI will be the backbone of an attempt to start a business with 2 friends and achieve a lifelong dream of boomer free, financial independence.
Cheers!
PS: this seems to be the correct place to post this, if I missed the obvious sub forum please let me know.
OS/purpose: Proxmox 8.0 to host 1 main ubuntu-server vm, 1 main win11 pro vm, 1 TrueNas Scale to hold the storage vdevs and maybe some task specific VMs and cool containers.
3 users will use this server as a workstation (both on ubuntu-server and win11).
I will try to use it as my “home” PC as well, will try PCIe passthrough for my current PC GPU and test gaming on Win11 VM.
Questions: is it better to create the zpool under proxmox or under a TrueNas like VM? Does this question even make sense?
Filesystem: zfs, sync=disabled, no SLOG sdev, no L2ARC sdev
Redundancy requirement: Nothing more than mirror or z1, will rely on scheduled backups/snapshots of everything. Convenience of a mirror is more important than $/GB. Planning to add a separate file storage server with raidz2 in near future.
Storage hardware: Already bought, but open to suggestions if it will make a tangible difference. However, I cannot add more storage without using a PCIe card. All bays, slots full otherwise (building in a Fractal Torrent). To be honest I bought the storage hardware to max out the mobo and the Torrent with near max GB. List of drives below.
Rest of the hardware (bought from zac1 on STH):
ASRock Rack ROME2D16-2T Server Motherboard Dual Socket Dual 10G
2 x AMD EPYC 7742 64c/128t CPU 2.25GHz (3.4Ghz Turbo)
256GB RAM: 16 x 16GB DDR4 3200 EEC
zpool will look like this:
special vdev: 2 x 2TB NVMe SSDs in mirror
vdev: 2 x 4TB SATA SSDs in mirror
vdev: 2 x 4TB SATA SSDs in mirror
vdev: 2 x 16TB SATA HDDs in mirror
TOTAL POOL: ~ 24TB - will be kept at 90% = 21-22TB (enough)
I have not built and booted the machine yet, so I don't know the actual drive names. Assuming:
How to change the drive reference in a zfs pool from /dev/sdX to /dev/disk/by-id (ata-XXXXXX)
- NVMe Drives: `/dev/nvme0n1`, `/dev/nvme1n1`
- SATA SSDs: `/dev/sda`, `/dev/sdb`, `/dev/sdc`, `/dev/sdd`
- SATA HDDs: `/dev/sde`, `/dev/sdf`
Will I be on the right track when I finish building the machine?
In the last few weeks, I’ve been learning about “homelab” and how to make one from scratch.
Been using linux daily and on clusters for years but "using" is the keyword. Having built all my PCs in the last 20 years or so, I have never built a machine with the specs below and set up anything like this.
Just looking for a quick review and suggestions on my notes below.
This HPI will be the backbone of an attempt to start a business with 2 friends and achieve a lifelong dream of boomer free, financial independence.
Cheers!
PS: this seems to be the correct place to post this, if I missed the obvious sub forum please let me know.
OS/purpose: Proxmox 8.0 to host 1 main ubuntu-server vm, 1 main win11 pro vm, 1 TrueNas Scale to hold the storage vdevs and maybe some task specific VMs and cool containers.
3 users will use this server as a workstation (both on ubuntu-server and win11).
I will try to use it as my “home” PC as well, will try PCIe passthrough for my current PC GPU and test gaming on Win11 VM.
Questions: is it better to create the zpool under proxmox or under a TrueNas like VM? Does this question even make sense?
Filesystem: zfs, sync=disabled, no SLOG sdev, no L2ARC sdev
Redundancy requirement: Nothing more than mirror or z1, will rely on scheduled backups/snapshots of everything. Convenience of a mirror is more important than $/GB. Planning to add a separate file storage server with raidz2 in near future.
Storage hardware: Already bought, but open to suggestions if it will make a tangible difference. However, I cannot add more storage without using a PCIe card. All bays, slots full otherwise (building in a Fractal Torrent). To be honest I bought the storage hardware to max out the mobo and the Torrent with near max GB. List of drives below.
Rest of the hardware (bought from zac1 on STH):
ASRock Rack ROME2D16-2T Server Motherboard Dual Socket Dual 10G
2 x AMD EPYC 7742 64c/128t CPU 2.25GHz (3.4Ghz Turbo)
256GB RAM: 16 x 16GB DDR4 3200 EEC
zpool will look like this:
special vdev: 2 x 2TB NVMe SSDs in mirror
vdev: 2 x 4TB SATA SSDs in mirror
vdev: 2 x 4TB SATA SSDs in mirror
vdev: 2 x 16TB SATA HDDs in mirror
TOTAL POOL: ~ 24TB - will be kept at 90% = 21-22TB (enough)
I have not built and booted the machine yet, so I don't know the actual drive names. Assuming:
How to change the drive reference in a zfs pool from /dev/sdX to /dev/disk/by-id (ata-XXXXXX)
- NVMe Drives: `/dev/nvme0n1`, `/dev/nvme1n1`
- SATA SSDs: `/dev/sda`, `/dev/sdb`, `/dev/sdc`, `/dev/sdd`
- SATA HDDs: `/dev/sde`, `/dev/sdf`
Bash:
zpool create -o ashift=12 -o autoexpand=on -o autoreplace=on -o sync=disabled -f tank mirror sda sdb mirror sdc sdd mirror sde sdf
zpool add tank special mirror nvme0n1 nvme0n2
zfs create -o recordsize=32k -o compression=on tank/ssd_dataset1
zfs create -o recordsize=64k -o compression=on tank/ssd_dataset2
zfs create -o recordsize=128k -o compression=on tank/hdd_dataset1
zfs create -o recordsize=1M -o compression=on tank/hdd_dataset2
The nature of what we do is applied science, R&D/Innovation, numerical simulations of the atmosphere, climate, ocean circulation, storm surge, wave/hydrodynamics, CFD, running a bunch of mpi compiled fortran code with slurm etc. We sometimes deal with large datasets (global climate model datasets, AIS vessel datasets, large ascii/binary model inputs outputs etc.) which are accessed via python/fortran codes or 3rd party software as needed. Sometimes we generate 100k .csv files of 20kb size for no good reason, or 500 high-res png files to make an animation mp4. Pretty mixed usage.
Last edited: