Hi all,
I'm considering setting up a Proxmox server, but I'm lost in the sea of possibilities
Maybe someone here has set up a similar system and could offer some advice...
What I'm looking to build is primarily a file server, with some containers or VMs (internal webservers, databases with light use etc) on top. The file server should ideally offer around 20TB of space and should be able to saturate at least an 10G connection when reading and writing small-ish files (image sequences mostly). 15 users will be accessing constantly, with a mix of 1G and 10G connections.
Here's my thinking so far:
Basic specs:
CPU: some dual Xeon with 8 or 10 cores each
RAM: 128GB DDR3 ECC
Money is an issue (when is it not)... so 20TB of enterprise flash is a no go, unfortunately.
I have a HP D2700 SAS enclosure here which does SAS 6Gb and SATA 3Gb with up to 25 2.5" drives. I don't have to use it, but it's available. I also have an LSI RAID controller for said enclosure with battery backup.
Option 1: Get a server or enclosure with 12 3.5" drives, fill with 4TB SAS HDDs and run as RAID10 (mirrored zfs) Get PCIe SSDs (Optane 900p?) or 12G SAS SSDs (if I get a server with a 12G backplane) and use as cache drives for L2ARC, ZIL etc.
Option 2: Similar to above, but instead get an HBA with external SAS and connect the HP D2700, put 25 1TB SAS drives in in a RAID6 (raidz2) configuration. Caching again as in Option 1
Option 3: Fill the D2700 with 500GB or 1TB *consumer* SATA 3G SSDs (they will peak at 300MB/s each due to 3G SATA) and live dangerously. Run as RAID5 with the RAID controller card, no ZFS and no caching, since the SSDs should be fast enough.
Option 4: Get Supermicro SC216 chassis do similar to above. Or a Chenbro RM23624 (looks like a great case with 12G)
I'm aware of the longevity issues with consumer SSDs. Write-back would be enabled through the RAID card.
So there I am. The 3.5" drives would certainly provide much more space for the money, but I'm not sure about the speed. Caching with the Optanes (or similar) would be very fast, but anything uncached... not sure. The 2.5" SAS drives would in total be more expensive, but I would expect them to be faster (simply by virtue of being more drives, though again, not sure with the whole vdev thing...)
The consumer SSD option would probably the fastest and easiest to set up by far, but it would leave me worried - has anyone been running a RAID or zfs purely with consumer SSDs (Samsung EVO 860 for example) and can share some experiences? Any help or opinions would be welcome!
I'm considering setting up a Proxmox server, but I'm lost in the sea of possibilities
Maybe someone here has set up a similar system and could offer some advice...What I'm looking to build is primarily a file server, with some containers or VMs (internal webservers, databases with light use etc) on top. The file server should ideally offer around 20TB of space and should be able to saturate at least an 10G connection when reading and writing small-ish files (image sequences mostly). 15 users will be accessing constantly, with a mix of 1G and 10G connections.
Here's my thinking so far:
Basic specs:
CPU: some dual Xeon with 8 or 10 cores each
RAM: 128GB DDR3 ECC
Money is an issue (when is it not)... so 20TB of enterprise flash is a no go, unfortunately.
I have a HP D2700 SAS enclosure here which does SAS 6Gb and SATA 3Gb with up to 25 2.5" drives. I don't have to use it, but it's available. I also have an LSI RAID controller for said enclosure with battery backup.
Option 1: Get a server or enclosure with 12 3.5" drives, fill with 4TB SAS HDDs and run as RAID10 (mirrored zfs) Get PCIe SSDs (Optane 900p?) or 12G SAS SSDs (if I get a server with a 12G backplane) and use as cache drives for L2ARC, ZIL etc.
Option 2: Similar to above, but instead get an HBA with external SAS and connect the HP D2700, put 25 1TB SAS drives in in a RAID6 (raidz2) configuration. Caching again as in Option 1
Option 3: Fill the D2700 with 500GB or 1TB *consumer* SATA 3G SSDs (they will peak at 300MB/s each due to 3G SATA) and live dangerously. Run as RAID5 with the RAID controller card, no ZFS and no caching, since the SSDs should be fast enough.
Option 4: Get Supermicro SC216 chassis do similar to above. Or a Chenbro RM23624 (looks like a great case with 12G)
I'm aware of the longevity issues with consumer SSDs. Write-back would be enabled through the RAID card.
So there I am. The 3.5" drives would certainly provide much more space for the money, but I'm not sure about the speed. Caching with the Optanes (or similar) would be very fast, but anything uncached... not sure. The 2.5" SAS drives would in total be more expensive, but I would expect them to be faster (simply by virtue of being more drives, though again, not sure with the whole vdev thing...)
The consumer SSD option would probably the fastest and easiest to set up by far, but it would leave me worried - has anyone been running a RAID or zfs purely with consumer SSDs (Samsung EVO 860 for example) and can share some experiences? Any help or opinions would be welcome!