Hey team!
TLDR: Planning a new box, I want a SAS12 enclosure. Aslo, how do we host a bunch of containers and a ZFS? Bare metal or virtualized?
First, let me describe what I have, what I love and what I want improved. I got an R620 dirt cheap with 368Gb RAM, bumped CPUs to 12C's (max possible). I added the simple Samsung NVMe drive on a PCI riser for the IO-dependent tasks. It connected to NetApp DS4246 via LSI SAS2308. It runs the TrueNAS Scale on metal. All the usual home self-hosted stuff in Docker, controlled by Portainer.
It serves me well, but it have limitations. The hardware is getting old. The TrueNAS is bit too restrictive. I don't get much use out of it, other than ZFS management. The NetApp is acting up, may be there is something going on with hardware, but when systems boots, at some point it "resets" the enclosure. All the /dev/sd* drives appear in faulty state. Then the same bunch of drives gets mapped to next set of drive letters and work fine from there. This happened under Ubuntu, it happens in TrueNAS. So if I need a restart, I disconnect the enclosure, boot the NAS, connect the enclosure and manually zpool import it. It annoys the heck out of me.
So I bought an R640x, added a couple of NVMe drives and keep it powered off, until I figure out how to use it. Since I am posting this to Enclosures forum, let's start with the JBOD: I would love to have something newer for this. Preferably with SAS12. Not sure how many drive bays is better. Dell MD1400 and add enclosures when I need to expand? Or pursue a larger enclosure, like MD3460 or NetApp DE6600? The 240V would be a challenge. I don't have a UPS or a power line for this. Also, a power consumption is a factor. The noise is secondary. Something close to DS4246 power/noise range would be ideal.
If only NetApp had a shelf with SAS12G, that would be all I need.
Next thing, is better suited for other forums. But just in case, let me post it here. What is the best way to run this system? I want to keep the Portainer, but I want to move from docker-compose to K8s. I need ZFS. I would prefer to run it all on a VM. Ideally, ZFS would be virtualized, if it makes sense? Or is it a bad idea? I would love to see some examples how people home-lab a big storage pool. On one hand, it's nice to not have to think about the ZFS on the VM and just have a big available storage. On other hand, it's good to have a control over the ZFS cache memory consumption. I might have a memory-intensive task, from time to time.
Sorry about the long post.
Thank you!
TLDR: Planning a new box, I want a SAS12 enclosure. Aslo, how do we host a bunch of containers and a ZFS? Bare metal or virtualized?
First, let me describe what I have, what I love and what I want improved. I got an R620 dirt cheap with 368Gb RAM, bumped CPUs to 12C's (max possible). I added the simple Samsung NVMe drive on a PCI riser for the IO-dependent tasks. It connected to NetApp DS4246 via LSI SAS2308. It runs the TrueNAS Scale on metal. All the usual home self-hosted stuff in Docker, controlled by Portainer.
It serves me well, but it have limitations. The hardware is getting old. The TrueNAS is bit too restrictive. I don't get much use out of it, other than ZFS management. The NetApp is acting up, may be there is something going on with hardware, but when systems boots, at some point it "resets" the enclosure. All the /dev/sd* drives appear in faulty state. Then the same bunch of drives gets mapped to next set of drive letters and work fine from there. This happened under Ubuntu, it happens in TrueNAS. So if I need a restart, I disconnect the enclosure, boot the NAS, connect the enclosure and manually zpool import it. It annoys the heck out of me.
So I bought an R640x, added a couple of NVMe drives and keep it powered off, until I figure out how to use it. Since I am posting this to Enclosures forum, let's start with the JBOD: I would love to have something newer for this. Preferably with SAS12. Not sure how many drive bays is better. Dell MD1400 and add enclosures when I need to expand? Or pursue a larger enclosure, like MD3460 or NetApp DE6600? The 240V would be a challenge. I don't have a UPS or a power line for this. Also, a power consumption is a factor. The noise is secondary. Something close to DS4246 power/noise range would be ideal.
If only NetApp had a shelf with SAS12G, that would be all I need.
Next thing, is better suited for other forums. But just in case, let me post it here. What is the best way to run this system? I want to keep the Portainer, but I want to move from docker-compose to K8s. I need ZFS. I would prefer to run it all on a VM. Ideally, ZFS would be virtualized, if it makes sense? Or is it a bad idea? I would love to see some examples how people home-lab a big storage pool. On one hand, it's nice to not have to think about the ZFS on the VM and just have a big available storage. On other hand, it's good to have a control over the ZFS cache memory consumption. I might have a memory-intensive task, from time to time.
Sorry about the long post.
Thank you!