Need Advice Building a 16+ ZFS NAS

Thomas H

Member
Dec 2, 2017
61
21
8
49
I want to build a 16 drive ZFS NAS with 10G NIC. Possibly want to expand in the future. Currently, I have 10 HDDs and 6 SSDs. These will be in two ZFS RAIDZ2 pools respectively. General usage will be slower HDDs for media and faster SDDs for VMs. For 10G networking I have HP 649281-B21 adapter cards. I have several servers, Lenovo ThinkServer TS440 and Dell PowerEdge R330, but they are not ideal for this. Both have limited drive bays and PCIe expansion capabilities. TS440 can do 8X drives with 1xPCIe3 x16, 1xPCIe2 x4 and R330 can do 4X drives with 2xPCIe3.

I have seen threads like Supermicro 836 chassis, backplane, SAS expanders but do not know how those work. What do you recommend?
 

gea

Well-Known Member
Dec 31, 2010
2,472
834
113
DE
SuperMicro cases are in general professional quality but very loud (serverroom cases).
As an option, you can use Jbod cases. You connect them via exernal 12G SAS to one of your servers.

The cases are available with an expander or only miniSAS connectors to connect disks directly to an HBA.
You need then enough ports on the HBA(s). In general expander solutions are best with SAS disks while work with Sata.

For fast SSDs, look at 12G SAS HBAs like Broadcom 9300 or OEM. For disks, 6G HBAs like a 9207 or even 2008 chipset based ones are ok (original BroadCom or OEM from Dell, HP, IBM etc). If possible use an HBA with IT firmware, IR firmware is ok, raid-5 firmware is not ok. In any case, use only BroadCom HBA with LSI chipsets like 3008, 2307 or 2008.

The HP 649281-B21 may limit your OS choices to Linux. For ZFS, I would prefer Unix with Solarish (where ZFS comes from) or Free-BSD solutions. I would more prefer Intel based nics where driver support in general is best.

Two Pools (one from disks and a faster one from SSD) is a good solution. With the very new Open-ZFS Allocation Class feature it may be possible to use one pool with disks based vdevs and SSD based vdevs where you can use the SSD vdevs for metadata, dedup tables or small io ex for dedicated filesystems for VMs, see https://forums.servethehome.com/ind...ses-performance-benchmarks.26111/#post-241295
 

csp-guy

Active Member
Jun 26, 2019
188
59
28
Hungary, Budapest
Hi Thomas,

I think in home enviroment (if we are talking home usage of this hardware) you can spare some money.

If you have a good mid/big tower case with a lot of 5.25 space, you can use this 5.25->3.5 hdd cage.

5-in-3 Device Module Hard Disk Cage SAS/SATA Expander Enclosure

Few weeks ago I bought 3 of them, it is 15 disk storage capacity, and I will use in a well cooled atx midi tower.

Cheers.
 
Last edited:

BeTeP

Well-Known Member
Mar 23, 2019
489
306
63
The HP 649281-B21 may limit your OS choices to Linux
how so? Mellanox ConnectX-3 is one of best supported cards.

I have several servers, Lenovo ThinkServer TS440 and Dell PowerEdge R330, but they are not ideal for this. Both have limited drive bays and PCIe expansion capabilities. TS440 can do 8X drives with 1xPCIe3 x16, 1xPCIe2 x4 and R330 can do 4X drives with 2xPCIe3.
Keep your R330 and add a cheap external JBOD enclosure to it.
 
  • Like
Reactions: Thomas H

gea

Well-Known Member
Dec 31, 2010
2,472
834
113
DE
Yes, from the main three OS families Linux, Windows and Unix (Free-BSD. OSX, Solarish), only the first two have support for nearly any hardware while Unix has its own advantages in some areas with storage one of them.
 

BeTeP

Well-Known Member
Mar 23, 2019
489
306
63
I am aware of @gea's affinity. The card is fully supported in Solaris and FreeBSD. I am not 100% about macOS. But if they don't it's just because Apple chose to blacklist it for some reason.
 

gea

Well-Known Member
Dec 31, 2010
2,472
834
113
DE
Solaris and Free-BSD: yes
Illumos (free Solaris fork): no
OSX: no (I suppose)
 

Thomas H

Member
Dec 2, 2017
61
21
8
49
SuperMicro cases are in general professional quality but very loud (serverroom cases).
As an option, you can use Jbod cases. You connect them via exernal 12G SAS to one of your servers.

The cases are available with an expander or only miniSAS connectors to connect disks directly to an HBA.
You need then enough ports on the HBA(s). In general expander solutions are best with SAS disks while work with Sata.

For fast SSDs, look at 12G SAS HBAs like Broadcom 9300 or OEM. For disks, 6G HBAs like a 9207 or even 2008 chipset based ones are ok (original BroadCom or OEM from Dell, HP, IBM etc). If possible use an HBA with IT firmware, IR firmware is ok, raid-5 firmware is not ok. In any case, use only BroadCom HBA with LSI chipsets like 3008, 2307 or 2008.

The HP 649281-B21 may limit your OS choices to Linux. For ZFS, I would prefer Unix with Solarish (where ZFS comes from) or Free-BSD solutions. I would more prefer Intel based nics where driver support in general is best.

Two Pools (one from disks and a faster one from SSD) is a good solution. With the very new Open-ZFS Allocation Class feature it may be possible to use one pool with disks based vdevs and SSD based vdevs where you can use the SSD vdevs for metadata, dedup tables or small io ex for dedicated filesystems for VMs, see https://forums.servethehome.com/ind...ses-performance-benchmarks.26111/#post-241295
Are you saying SuperMicro storage cases like SC826, SC836, SC846, SC847, etc. are very loud? Any JBOD cases you recommend?

I am starting to read up on SAS expanders. The 6X SSDs I have are HGST HUSMR1650ASS204 (aka Ultrastar SSD1600MR) drives. If I have 6 SSDs (12Gb/s SAS) and 10 HDDs (SATA3) in a JBOD chassis on a single 12G SAS expander connection, will performance be severely crippled (by the SSDs)?

The HBAs I have are Dell Perc H330 from my R330s. I am planning on flashing them to HBA IT mode for ZFS. Will these work if I go JBOD with expander route?
 

BeTeP

Well-Known Member
Mar 23, 2019
489
306
63
If your R330 would have been 8xSFF instead your your 4xLFF - the obvious way to go would be keeping your 12G SAS SSDs in the R330 connected to H330 and getting a separate cheap 6G SAS HBA for the external JBOD.

Personally I prefer HP and I have replaced front drive cages in HP servers with more suitable config (from LFF to SFF and vice versa) many times before. The previous generation parts are easily available and rather inexpensive. I am not sure if that's an option for Dell.
 
Last edited:
  • Like
Reactions: Thomas H

i386

Well-Known Member
Mar 18, 2016
1,888
491
83
31
Are you saying SuperMicro storage cases like SC826, SC836, SC846, SC847, etc. are very loud?
It depends :D
There are different versions of each chassis with different fans and psus.

The older chassis had the lowest rpm system and rear fans (5k rpm) but psus with fans at 13k rpm running even when the server was shut down. Replace the psus and the pdb and you're good to go.

Revision B chassis have slightly faster system and rear fans 6.3~7k rpm fans (these fans don't have that annoying whine as the chassis with the slower 5k rpm fans) and sq psus that turned off when the system was shut down. I would look for such a chassis. (They also have space on the back for two additional sata/sas storage devices)
(~41 dba measured from ~10cm behind the rear fans with a nexus 5x @ 18° C ambient and 30-35° C system temperature)

The newest versions of these chassis are designed for the scalabe/epyc cpus/tesla accelerators and have system fans that can go up to 11k rpm. You don't want these for a media server :D
 
  • Like
Reactions: Thomas H

gea

Well-Known Member
Dec 31, 2010
2,472
834
113
DE
SM cases are prepared to be filled up with high performance disks under steady load. Then the cooling is adequate. With less or low power disks outside a datacenter you may want different (more silent) fans.

about SAS and expander
Expanders are connected by one or two miniSAS cables with an HBA. Such a cable combines 4 SAS ports and allows to connect a full jbod case with 24 disks or more via the expander. Performance wise, you can combine slower and faster disks without problem but as all data must go through miniSAS, this can limit performance.

ex.
The combined performance of 4 x 6G SAS is 24G. If you count 3G for a mechanical disk, the SAS cable allows up to 8 disks to perform with full performance. This doubles with 12G SAS HBAs and expanders or when you connect the expander with 2 x miniSAS cables.

If you connect Sata SSDs with around 6G performance, you reach the limit with half the disks. If you use 12G SAS SSDs like WD Ultrastar SS530 you use 12G per disk or even 24G when using both SAS ports. In such cases the expander can be a bottleneck.

As SAS offers a more robust protocol and data transmission, I always prefer SAS disks over Sata disks with an expander. With a 16 (or 24) disk case I would always prefer 2 x 8 port or 1 x 16 port HBA and avoid the expander. Faster and more robust with propably less costs.