ZFS mass storage server (no virtualization) recommendations

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

lpallard

Member
Aug 17, 2013
276
11
18
Hi all!

I'm glad the forum is finally back!

I am searching for ZFS gurus' advices regarding a server that will be used to store terabytes of data in a long-term fashion. This machine will be connected to my SoHo LAN and will

-store my proxmox node's VM backups
-Store my LAN clients backups
-store several hundred's GB's of personal data
-store a OpenKM datastore
and many other data storage uses

The goal here is to ensure data integrity and reliability in the long run. My current proxmox node has a lot of horsepower for VM's but is not really tailored toward long term storage (doesnt use ZFS, uses proprietary raid controllers, etc).

Once this future ZFS storage server is setup and running fine, I will perhaps move my VM's images to it. We'll see.

In the meantime, I must say that I already have 9x 2TB SATA drives and plan to buy more. The server you will recommend will have to be able to host at least 12x 3.5" hard drives for future expansion.

Hotswap is a big plus.

I have in mind the DL180 G6 but maybe you can recommend a better machine??? Used or not. I am however limited to about $700 so a second hand unbeaten server is perhaps the way to go??

Waiting for the storage Guru's to reply!

Cheers and again congrats to the servethehome admins and people who helped bringing the forums back online!!!!!!
 

lpallard

Member
Aug 17, 2013
276
11
18
Forgot to mention before I hit "post" button! I will (as the title says) use ZFS on a JBOD config so NO raid necessary (if your recommended server comes with a RAID controller I will remove it and use a M1015 flashed to IT mode as a dumb HBA)..
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
One thing I would at least think about is moving to the new 5TB and 6TB drives which are being produced these days. Still expensive but in a few months they will be relatively reasonable. You could move from 9 drives to 3-4 which gives you many more options.
 

capn_pineapple

Active Member
Aug 28, 2013
356
80
28
what patrick said.

I've just shifted from 2TB drives in raid 5 up to 4TB drives in raid 6 increasing both density and redundancy. You can do the same thing albeit with larger drives and the additional benefits of ZFS
 

lpallard

Member
Aug 17, 2013
276
11
18
Thats good recommendations but what about the actual machine?

The only server I ever built is my SoHo server and II must say it works very well but was kinda expensive (around $2,800) which now leaves me little budget for another critical machine... I am thinking about second hand servers.

Which model/brand would serve well as a ZFS machine with JBOD?

DL180 G6??
 

capn_pineapple

Active Member
Aug 28, 2013
356
80
28
You can grab HP microservers 2nd hand off ebay for super cheap, N54L or N40L are the models you'd be looking for.
 

lpallard

Member
Aug 17, 2013
276
11
18
Im more looking for a 2U rackmount machine... I already have a supermicro server that currently uses a full tower but I will move it to a 4U chassis soon..

I guess reading the ZFS handbook and several websites I will need about 16 to 24GB of RAM since I will be storing about 12TB in a zpool(s). Im also looking into L2ARC or ZIL but the last one is not sure yet..
 

bds1904

Active Member
Aug 30, 2013
271
76
28
1u rackmount server, pick your brand. Personally I like tbe half depth rackable servers.

Sas2008 controller with external sff8088 ports

Rackable se3016 16 drive expander.

Hands down, cheap and simple.
 

lpallard

Member
Aug 17, 2013
276
11
18
2U Supermicro 12 Bay Server X7DBN 2X Intel XEONL5420 16GB RAM AOC SAT2 MB8 Simlc | eBay

Can get a whole unit for ~300 bucks...

Just really depends what you want OP...

There are tons of deals out there.
Yeah these Supermicro 12 bay servers are pretty interesting.

I agree with you, it depends on what I want, but the problem is that I dont know what I need!

For a reliable ZFS storage, I believe I need at least 9 drive bays with caddies for hotswap, min 16GB RAM and a couple PCIE slots for M1015.

So a few questions :

  • Since I am using a M5016 & M1015 on a Supermicro H8DCL-iF with NO issues at all, can I expect to install the M1015 in the Supermicro server and it works?
  • Supermicro specs says "Hot-swap drive trays SAS or enterprise SATA HDD only recommended" so can I assume these servers can take standard consumer grade SATA drives??? If I need to buy SAS drives then here's my budget going to hell!

I will email the sellers to ask about the backplane (direct 1-1 or expanders)
 

lpallard

Member
Aug 17, 2013
276
11
18
Forgot: do I really need a dual socket server for a good ZFS experience ir should I trade a CPU socket for extra RAM?
 

ehorn

Active Member
Jun 21, 2012
342
52
28
For a soho you surely will not need one of these xeons, let alone 2! Lol.. You can saturate gigabit Ethernet with atom procs... Memory is more important for zfs performance .... 16gb min (IMHO).

The m1015 will not give problems with any of these MB's. It is well established and well documented. Sata drives are fine. But not all sata drives play nice in a nas configuration. There is lots of info on why... which do and which don't. What drives do you currently own?

If you have a budget of 700, here is another nice turn key option:

HP Proliant DL180 G6 Storage Server 2 x QC L5520 2 26GHz 24GB Trays Rails P410 008835859909 | eBay

Patrick has a nice review of this box here:

http://www.servethehome.com/Server-...-g6-dual-intel-xeon-2u-storage-server-review/
 

lpallard

Member
Aug 17, 2013
276
11
18
The Proliant DL180G6 with 2x L5520 is VERY interesting although a bit over my budget... I'd have to weight the pros & cons vs my needs... As I gathered over 6TB of important data in 5 years or so, I want a reliable platform to store as well as store my VM's via iScsi (maybe)?

I may eventually connect this storage server with my existing proxmox server through a high speed connection (TBD). Would I be OKto install a 3rd party LAN card on these supermicro & HP servers or do I have to stay proprietary?

Im not too comfortable with the idea of popping a M1015 into a Proliant server.. somehow I feel Im trying to install a Toyota master cylinder onto a Nissan sentra :)
 

Diavuno

Active Member
get more ram for ZFS any modernish xeon will be ample power.

the m1015 is fine in supermicro boards. (its just a rebranded lsi) (feel bad putting a nissan master cyl into an infinity?)


the DL180's are good boxes. but get VERY loud when you put in any non HP card.... aftermarket NIC's included!

oh, and for ZFS you want a HBA... the P410's? not prefered but can be made to work.
 

lpallard

Member
Aug 17, 2013
276
11
18
Im kinda interested in this one... on a bit expensive side but its a supermicro, has 32GB RAM (possibility to upgrade up to 64GB), has 3 PCIE slots for a infiniband NIC a M1015 and maybe another card.. and 2x Quand core xeon..

If I could find the same machine with a less powerful CPU but with more RAM!

Other than that, this is the list of the best choices Ive found todayon ebay... Any gotchas you guys know regarding these models, please let me know before I hit "Buy it now!"

http://www.ebay.ca/itm/3U-Supermicro-16Bay-Server-X7DBE-2x-Intel-Xeon-L5420-16GB-RAM-CSE-836TQ-R800B-/141318475681?pt=COMP_EN_Servers&hash=item20e73ccfa1#ht_2963wt_1036

http://www.ebay.ca/itm/2U-Supermicro-X7DBE-2xIntel-QC-E5430-2-6Ghz-32GB-8x-4GB-12-bays-SAS-w-Rail-KIT-/131215461572?pt=COMP_EN_Servers&hash=item1e8d0d0cc4#ht_2234wt_970

http://www.ebay.ca/itm/3U-Supermicro-16Bay-Server-X7DBE-2x-Intel-QC-L5320-1-86ghz-16GB-RAM-W-Rail-Kit-/151327451336?pt=COMP_EN_Servers&hash=item233bd1a8c8#ht_2551wt_1036

http://www.ebay.ca/itm/2U-Supermicro-Server-X7DBN-2x-Intel-Quad-Core-E5310-1-6Ghz-16GB-12-HD-Bays-SIMLC-/151324879678?pt=COMP_EN_Servers&hash=item233baa6b3e#ht_2708wt_970

http://www.ebay.ca/itm/2U-Supermicro-X7DBN-2x-Intel-Quad-Core-E5310-1-6Ghz-16GB-12-bays-SAS-w-Rail-KIT-/151324879622?pt=COMP_EN_Servers&hash=item233baa6b06#ht_2207wt_970

http://www.ebay.ca/itm/2U-Supermicro-X7DBE-2x-Intel-Quad-Core-E5310-1-6Ghz-16GB-12-bays-SAS-w-Rail-KIT-/131213736534?pt=COMP_EN_Servers&hash=item1e8cf2ba56#ht_2422wt_970

http://www.ebay.ca/itm/HP-ProLiant-DL180-G6-2U-2X-QC-L5520-2-26GHZ-12xTRAYS-0GB-MEM-P410-512MB-BATTERY-/251500581576?pt=COMP_EN_Servers&hash=item3a8e9a52c8#ht_2260wt_1207

http://www.ebay.ca/itm/HP-PROLIANT-DL180-G6-STORAGE-SERVER-2-x-QC-L5520-2-26GHz-8GB-TRAYS-RAILS-P410-/131193837159?pt=COMP_EN_Servers&hash=item1e8bc31667#ht_2421wt_970

http://www.ebay.ca/itm/HP-PROLIANT-DL180-G6-2-x-QC-E5520-2-26GHz-8GB-TRAYS-RAILS-/131212699451?pt=COMP_EN_Servers&hash=item1e8ce2e73b#ht_2391wt_970

http://www.ebay.com/itm/HP-PROLIANT-DL180-G6-STORAGE-SERVER-2-x-QC-L5520-2-26GHz-24GB-TRAYS-RAILS-P410-/111357841751?pt=COMP_EN_Servers&hash=item19ed71d157#ht_2421wt_970
 

Diavuno

Active Member
keep in mind a couple things:

the 5000-5400 series chips are hot and power hungry. the vast majority of them ONLY take FB DDR2 its way cheaper NOW but still kind of pricy... and again hot/power hungry.

the 55xx and 56xx chips seem to offer a great bang for your buck, run cooler and draw less juice.
they also take up to 24GB of unbuffered desktop memory per cpu. (48GB DIRT CHEAP) or up to something silly like 144gb per socket when using reg+ecc

those DL180's are good boxes, i run a pair myself....
BUT
the exander in the 14 bay models is less than awesome.
the 14 bay "storage" model has NO pci slots free, you get the one 16x slot (i think it's actually 8x electrical. that kind of kills the idea of additional nics.
you MIGHT be able to use a ribbon expander and some insulating material to move the P410 then use the controllers slot for "A" nic.

the DL 180's i have had 2 16x slots.. i replaced one riser with a 2 8x slot. My boxes now have 2 8x slots and a 16x (those risers are cheap]
and the 8 bay units actually have 2 slots you can install trys, but you need longer cables and, not fully hot swap, but works for me (they plug into the main board]
I also have a couple of SSD'S mounted behind where the optical drive would go. so 10x 3.5 2 ssd (room for more ssd] but only 8 of the 3.5 are truly hot swaps!
all with 2x additional nics!
it fits my current needs very well though!
 

lpallard

Member
Aug 17, 2013
276
11
18
starshooter, since you seems to be way ahead of me, let me ask you this simple question!

For up to 600$ Canadian, what would you get if you were me??.
 

lpallard

Member
Aug 17, 2013
276
11
18
Re-reading your comments starshooter, looks like there is a fair amount of hacking involved in making a simple storage server with the DL180 G6 servers... Am I missing something?

I need nothing complicated. No Raid, no special adapter cards, nothing. Just a mobo, CPU's RAM and a **load of SATA ports with hotswap capabilities.

With the supermicro servers it seems that you know what you get. To me they look simpler. Kinda like a standard desktop jammed into a server format.
 

Diavuno

Active Member
starshooter, since you seems to be way ahead of me, let me ask you this simple question!

For up to 600$ Canadian, what would you get if you were me??.
Re-reading your comments starshooter, looks like there is a fair amount of hacking involved in making a simple storage server with the DL180 G6 servers... Am I missing something?

I need nothing complicated. No Raid, no special adapter cards, nothing. Just a mobo, CPU's RAM and a **load of SATA ports with hotswap capabilities.

With the supermicro servers it seems that you know what you get. To me they look simpler. Kinda like a standard desktop jammed into a server format.
I chopped up my DL180 to suit MY needs...


lets start with what you want and what you NEED.

HP is kind of doing their own thing, and locking down... super micro has always been friendly and tends to play well with others... but depending on your requirements dont rule out asus or gigabytes server lineups