Build’s Name: Cheza
Operating System/ Storage Platform: FreeBSD 10.3 or Solaris 11.3 / ZFS
CPU: Pending – Intel E5-2630L v2
Motherboard: Pending – Supermicro X9SRL-F
Chassis: AIC RSC-4ETS w/HotSwap Kit for 2x2.5” Rear Drives
Drives:
24 x HGST HUH728080AL4200 8TB (Storage)
2 x Samsung 850 Pro 256GB (OS)
RAM:
4 x 32GB DDR3 ECC LRDIMM for 128GB total (Pending)
Add-in Cards:
LSI 9300-4i4e HBA for Storage Drives
3Ware 9750-4i Raid Controller for OS Mirror
Dual Port 10GB SFP+ NIC (Chelsio or Intel TBD)
Power Supply: AIC Provided 700W 1+1 redundant PSU 80+ Platinum
Other Bits: Managed PDU and UPS systems in place
Usage Profile: Primary File Server (SMB, AFP, NFS, FTP, TFTP)
Its been about 5 years since my last build which resulted in a FreeBSD 9 hardware raid based file server using 16 x 2TB drives. I’m now out of warranty on the drives and had two fail (had to purchase replacements to extend my old builds life temporarily until I can get this new one going). That system, with its primary array and a few small arrays I had added over the years has about 33TB of usable space. My usual rule of thumb is when I rebuild I triple the storage capacity at minimum.
So this new build is pretty much that though slightly more than triple. While I definitely wanted to move up to SAS 12Gb/s I also had a limitation of depth for the racks I have mounted in my utility room (Middle Atlantic CWR-18-26) so my choice of chassis was VERY limited. No Supermicro, CiDesign, Intel, or any other brand had anything that could work (Norco was the only one that had an option and I wasn’t thrilled about using them). I had initially started planning this build close to a year ago however due to the chassis limitation I had kept waiting and hoping for a company to release their updated models. AIC finally did that with the RSC-4ETS and J4024 (JBOD) as they are only 22” deep which is perfect. So at least that settled that issue at least.
Motherboard and CPU – Though I am going to use ZFS, the only other services that will run on this machine are SMB, AFP and NFS so its not a very CPU intensive build. I figured sticking with a single cpu would be fine and look like I may settle on the X9SRL-F as I should easily have enough capacity for RAM given the large array I am plan. And ultimately I picked the E5-2630LV2 as it's the lowest TDP of the E5-2600 series that would work in the board and still should provide more than enough processing power than I would need. Initially I am estimating needing 128GB of ram and with 32GB modules, that's only four of the 8 slots available so I can upgrade if needed to more later.
The plan is to fill the chassis with as many 8TB drives as I can to get this built out and not have to worry later about having to upgrade or change anything. I’m still researching ZFS as I don’t have that much experience with it so the number of 8TB could change based on best practices but initially the plan is raidz3 as well as 3 HSP. I would pickup a new 9300-4i4e HBA to connect the internal expander as well as provide an external port if I decided later I needed to add more storage to this machine as I can pickup a JBOD chassis or even use one of the two old ones I have with the appropriate cable (they are SAS 6Gb/s)
I’ve had OS drives fail I always mirror a pair of them at this point. I have plenty of old drives I can use, including a brand new paid of 320GB WD SATA’s, I am considering just picking up two 850 Pro 256GB to save a little power and that I would prefer to use drives that have warranties. I have a few spare 9750-4i’s hanging around here so I figured I would just use that to handle the mirroring to simplify the OS install and all.
With the rebuild, I am also planning a smaller file server to use for VM storage and will be rebuilding my ESX cluster. That part isn’t happening as quickly however I already have the networking infrastructure in place to support the upgrade to a 10GB backbone. Depending on what’s available and cost effective I am planning either a Chelsio or Intel dual SFP+ nic.
Other than settling on the hardware the biggest part of the planning for this machine is the ZFS array using the 8TB drives. It's a lot of drives and given their large capacity, rebuilds are not going to be quick so safety is key however they are also not cheap so I have to strike a balance as to not waste too many in parity. Either way I will be allocating enough HSP as well for all potential failures (if it's a single large raidz3 then 3xHSP, if its 2 smaller raidz2 then 4xHSP).
I also need to decide on what SSD to pickup for Zil and then figure out where to put it physically as I may not have enough open hot swap bays for another drive.
FreeBSD 10.3 still doesn’t look to have an actual hot spare solution without custom scripting things so I am considering going to Solaris 11.3 as there is a native solution there for it. I will not do a cold spare solution or even a hot spare that requires manual intervention to fail over as I travel quite a bit and am not always able to get access to the machine to initial a recovery if needed. I don’t have any experience with Solaris to date so this will be a new learning experience on that front too for me (in addition to ZFS). I know also that while the hardware I am planning is supported within FreeBSD, I need to go back and look through the hardware lists for Solaris to confirm its all supported there as well which I have not done yet.
So to summarize I guess I’m looking for input on the following:
Operating System/ Storage Platform: FreeBSD 10.3 or Solaris 11.3 / ZFS
CPU: Pending – Intel E5-2630L v2
Motherboard: Pending – Supermicro X9SRL-F
Chassis: AIC RSC-4ETS w/HotSwap Kit for 2x2.5” Rear Drives
Drives:
24 x HGST HUH728080AL4200 8TB (Storage)
2 x Samsung 850 Pro 256GB (OS)
RAM:
4 x 32GB DDR3 ECC LRDIMM for 128GB total (Pending)
Add-in Cards:
LSI 9300-4i4e HBA for Storage Drives
3Ware 9750-4i Raid Controller for OS Mirror
Dual Port 10GB SFP+ NIC (Chelsio or Intel TBD)
Power Supply: AIC Provided 700W 1+1 redundant PSU 80+ Platinum
Other Bits: Managed PDU and UPS systems in place
Usage Profile: Primary File Server (SMB, AFP, NFS, FTP, TFTP)
Its been about 5 years since my last build which resulted in a FreeBSD 9 hardware raid based file server using 16 x 2TB drives. I’m now out of warranty on the drives and had two fail (had to purchase replacements to extend my old builds life temporarily until I can get this new one going). That system, with its primary array and a few small arrays I had added over the years has about 33TB of usable space. My usual rule of thumb is when I rebuild I triple the storage capacity at minimum.
So this new build is pretty much that though slightly more than triple. While I definitely wanted to move up to SAS 12Gb/s I also had a limitation of depth for the racks I have mounted in my utility room (Middle Atlantic CWR-18-26) so my choice of chassis was VERY limited. No Supermicro, CiDesign, Intel, or any other brand had anything that could work (Norco was the only one that had an option and I wasn’t thrilled about using them). I had initially started planning this build close to a year ago however due to the chassis limitation I had kept waiting and hoping for a company to release their updated models. AIC finally did that with the RSC-4ETS and J4024 (JBOD) as they are only 22” deep which is perfect. So at least that settled that issue at least.
Motherboard and CPU – Though I am going to use ZFS, the only other services that will run on this machine are SMB, AFP and NFS so its not a very CPU intensive build. I figured sticking with a single cpu would be fine and look like I may settle on the X9SRL-F as I should easily have enough capacity for RAM given the large array I am plan. And ultimately I picked the E5-2630LV2 as it's the lowest TDP of the E5-2600 series that would work in the board and still should provide more than enough processing power than I would need. Initially I am estimating needing 128GB of ram and with 32GB modules, that's only four of the 8 slots available so I can upgrade if needed to more later.
The plan is to fill the chassis with as many 8TB drives as I can to get this built out and not have to worry later about having to upgrade or change anything. I’m still researching ZFS as I don’t have that much experience with it so the number of 8TB could change based on best practices but initially the plan is raidz3 as well as 3 HSP. I would pickup a new 9300-4i4e HBA to connect the internal expander as well as provide an external port if I decided later I needed to add more storage to this machine as I can pickup a JBOD chassis or even use one of the two old ones I have with the appropriate cable (they are SAS 6Gb/s)
I’ve had OS drives fail I always mirror a pair of them at this point. I have plenty of old drives I can use, including a brand new paid of 320GB WD SATA’s, I am considering just picking up two 850 Pro 256GB to save a little power and that I would prefer to use drives that have warranties. I have a few spare 9750-4i’s hanging around here so I figured I would just use that to handle the mirroring to simplify the OS install and all.
With the rebuild, I am also planning a smaller file server to use for VM storage and will be rebuilding my ESX cluster. That part isn’t happening as quickly however I already have the networking infrastructure in place to support the upgrade to a 10GB backbone. Depending on what’s available and cost effective I am planning either a Chelsio or Intel dual SFP+ nic.
Other than settling on the hardware the biggest part of the planning for this machine is the ZFS array using the 8TB drives. It's a lot of drives and given their large capacity, rebuilds are not going to be quick so safety is key however they are also not cheap so I have to strike a balance as to not waste too many in parity. Either way I will be allocating enough HSP as well for all potential failures (if it's a single large raidz3 then 3xHSP, if its 2 smaller raidz2 then 4xHSP).
I also need to decide on what SSD to pickup for Zil and then figure out where to put it physically as I may not have enough open hot swap bays for another drive.
FreeBSD 10.3 still doesn’t look to have an actual hot spare solution without custom scripting things so I am considering going to Solaris 11.3 as there is a native solution there for it. I will not do a cold spare solution or even a hot spare that requires manual intervention to fail over as I travel quite a bit and am not always able to get access to the machine to initial a recovery if needed. I don’t have any experience with Solaris to date so this will be a new learning experience on that front too for me (in addition to ZFS). I know also that while the hardware I am planning is supported within FreeBSD, I need to go back and look through the hardware lists for Solaris to confirm its all supported there as well which I have not done yet.
So to summarize I guess I’m looking for input on the following:
- ZFS:
- Any ZFS experts here to help with recommendations on design for the array? The chassis can hold up to 24 drives max and I am looking to find the best design that also provides the most amount of storage without introducing too much risk.
- Recommendations on what drive to use for zil? And should I use a mirrored pair for this?
- Feedback on if I should bother creating an L2ARC as well? Given I plan on at least 128GB of ram for the machine, I should surpass the 1GB -> 1TB recommendations and can always throw some more ram in the machine later.
- 10GB SFP+ NIC performance and compatibility. I’ve used my share of Intel and Chelsio cards and from my understanding both still work great?
- Am I correct that there is no HSP support within FreeBSD still to allow for automated rebuilds?