Nas Build

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Looking to put most of my storage in to one box, soo looking for options/opinions.

What will go in:

What I have:
HDD that I currently have planeed:
3x10TB WD RED sata - this should get expanded to 5-7drives
12x6TB HGST 4kn sas (with flash cache)
1x8TB encrypted seagate enterprise archive

Now the build:
The case was thinking on :
4U 4129-N - Inter-Tech GmbH

Good fan positioning, and can be swapped easily, space for 14 HDD on the front , and enough space for another 5x caddy behind PSU.
Bassicaly the best I fount for around the 100E mark, disk spaece important, hot swap bays not so much.

Now for the interesting bit...
Motherboard should be something with at least 3PCIE 3.0 x8(2 for HP H240 HBA , or adaptec ASR-71605 that I´m planning and one for the 10G network possibly 4 for an option to hook it up to my infiniband network as wel)l, 2 1g eth would be a plus, onboard sata not so important 5-6 would be appreciated for the WD array

Open to sugestions here... old server board, new ryzen one, new intel one, chinese 2011 knock offs


CPU low power INTEL or AMD which will best fit this(this will be for roughly 100-150TB of storage)?

RAM... well guess 32-64 should cover it.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE
First select operating system (Free-BSD, Solaris/OmniOS or Linux) and your fileserver concept and filesystem then hardware at last because the first determines hardware.
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Guess OS will be freenas as is the only native ISCSI, it´ll provide ESX datastore among other stuff and get pretty hammered.
NVME cache guess it´s not necessary, from my testing hinders performance rather then adding a benefit.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Looking on solaris forks now.
Really had no idea they are still alive and communities behind it your reply sparked my interest.
I rarely touch solaris these days I think we still have a couple of clusters at work, I think it's been around 7-10 years since I have worked with solaris actively.
I mainly work with linux and BSD, so I'm more drawn to it as i know better the ecosystem.

How is hardware support o solaris distros (had to manually compile drivers for network cards on BSD for a nas box to get it to work, rather curios as it's rather old gear)? Has anyone of it finally managed to implement online vdev expansion?

Which distro is stable with a strong development community behind?

As for hardware should I go enterprise hw or a intel x299 is ok?
MSI X299 SLI Plus Reacondicionado
Intel Core i5-7640X 4.0Ghz BOX
The jump to i7 seems a litle steep:
Intel Core i7-7800X 3.5Ghz BOX

on the cheap enterprise side:
would be some e5-2430l or e5-2630l

I could re-purpose my Threadripper build but it's kind of power hungry(200W idle) and doubt support on BSD/Solaris will be very good.
Also, does Solaris support Infiniband, and convergent 10g cards?
 
Last edited:

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Well hats down:
Omnios+napp-it seems way cleaner and better build then any other NAS os I have seen thus far, it´s more enterprise oriented, and for me seems more intuitive by far than any other NAS os I have seen lately(freenas, openmediavault, Unraid, xigmanas).

Will have to reserve some time to play with it, but seems a better choice for a storage box.

How is hardware support for consumer grade hardware? is ECC a must on this OS?
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Well, just got the parts, so far x470 motherboard with 2x8 pcie 3.0 and 1x4 pcie2.0 and 3x1 pcie2.0 1 dedicated x4 3.0 nvne slot and 1 aditional x4 2.0 shared with last pcie. I tested with 16gb of ram for post and a ryzen 1600af(actually a 2600 underneath), right now waiting for case.
Looking on hba/expander options, what would be mu best bet? 1xhp h220(lsi2309) and an ibm expander?
1xhp h220 and 1 9211-8i
2xhp h220?

Plan is to use motherboard sata for potencial ssd, and the rest will be:
12x6tb hitachi hybrid sas drives 4kn
5x10tb wd RED
1x8tb seagate enterprise archive.

1x8 slot goes to 10g card
1x8 to controller
1x4 to the other controller
If expander, guess it'll use x1 slot as no data is necessary(correct me if i'm wrong)
Open to sugestions here as well as the best way to connect.
Think expansion opens the option for 2 intel hybrid deives (1tb qlc+32gb optane) as 2nd nvme is shared with x4 slot, but does cache+l2arc make senae in this build?

How much RAM will be a ballpark for this build 16? 32? 64?
 

ari2asem

Active Member
Dec 26, 2018
745
128
43
The Netherlands, Groningen
my choice of expander would be chenbro. it has sas and sata hdd support at 6gbps, 1 small cooling fan for chipset. it doesn't have pci-power contacts, so you can put it in any free slot inside your case. not nessesarly into mainboard.

1 disadvantage: to power chenbro expanders you need 4pin molex power connector

Chenbro - Products

Chenbro - Products

intel expanders, disadvantage is no fan cooling for chipset

Intel® RAID Expander RES2SV240 Product Specifications


i would stay away from hp expanders, for sata drives is the speed 3gbps. only sas-drives are for 6gbps supported on hp expanders
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Thanks for the reply.

Never mentined hp expanders, only lsi cards and expanders(hp h220 is based of lsi 2308), read mixed opinions on compatibity of pmc and freenas, that is why i stay away from hp h240 which is based of custom pmc chip, although it has better compativility with my g8 servers, I am not considering then for this build.

But on the other hand heard good stufd of ibm lsi based doughter card, and would be cleaner as ni cables needes other than sas cables and pci power.

Optionally I could go with AOC-S2308L-L8i as a card.
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Hello,

Just got the build in a working condition, not all pieces are in but they keep comming.
For now:
Msi X470
Amd 1600af
Lsi 9211-8i
8x6tb sas - raidz2 volume- main array
3x10tb wd red - raidz1volume- secondary array
1xseagate enterprise archive standalone back-up diak

Thing is the mobo only has 2 nvme slots, if i go with sas card + expander Ican use both.
But the only way I can get l2ark and zil on nvme is to get 2 hybrid nvne(intel h10) which has 256-1tb qlc flash and 32gb of optane on the same physical stick, to zfs they will show up as 4 ssd's thing is... Is qlc worth the hassle as l2arc? Optane should do nicely as zil from my knowledge.
Anyway will get a 1tb and partion it off as a 256disk so drive will actually work as slc disk.
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
quick question:

HP H220 or Supermicro AOC-S2308L-L8i?
Need to buy today so please let me know your thoughts....
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Hello,

They are both lsi2308 based cards
Just ordered HP and an IBM expander, we´ll se how they play when they arrive.

But just for the sake of documentation:
Supermicro AOC-S2308L-L8i:
RAID / HBA | Supermicro
** AOC-S2308L-L8i Broadcom SAS 2308 8 ports, 6Gb/s per port, 8 Internal, Low-profile, 63 SATA/SAS drives RAID 0, 1, 10
SuperMicro AOC-S2308L-L8i 8 puertos SAS-600 PCIe 6Gb/s tarjeta controladora de servidor | eBay

HP
HP H220 (HP P/N 660088-001 or 638834-001)
Is mostly based off lsi2308, and is a standard LSI 9207-8i as it has pcie 3.ox8 and is marked on the board, although on HP stickers is marked as 9205-8i(which acordind to LSI is old lsi 2008chipset which is PCIE 2.0 x8).
Nuevo HP H220 9205-8I que PCI-E 3.0 x8 adaptador de bus de host 660088-001 638834-001 | eBay

As for the expander, went with IBM FRU:46m0997 which is based on LSI as well and heard it plays nice with FreeBSD, as it´s only 17EUR it worth the risk to play with it.

IBM (46m0997) ServeRAID expansión expansor SAS de 16 Puertos Adaptador | eBay
 

Attachments

Last edited:

dreamsin

Active Member
Oct 31, 2018
137
77
28
I have the same case.
It is nice, but not too high quality and not the best noise isolation.
If you have it in a livingroom you definetely need to change the fans.
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Actually I could not find the case available, it was available in a couple of months and went with another similar case, with no fans, so the fans I added myself:
3Xarcftic f12 in the fron for HDD, 1X140 aerocool shark fin in the back of the first disk cages blowing over PCI cards, 1x140 aerocool normal fan next to it the blows over the CPU(no need for it, but just in case) 1x60 coller master as exhaust(had it lying around, have not bought the 60mm exahaust ones yet).


Noise is not that big of a deal as it sits on top of my Home lab 3xDL380P G8, 1XML360 G6,a 28 port 1G switch and one Mikrotik CRS317-1G-16S+RM, so the other part generates significantly more noise, especially the 2U DL380G8(stripped them of addon cards to bring noise down, had to remove infiniband and the 4 port 1G cards to ramp down fans to about 17% so they can be bearable)
Case is UNYKAch UK 4129.
UNYKAch UK 4129 Caja Rack 19" 4U Negra sin Fuente | PcComponentes.com
 
Last edited:

Spartacus

Well-Known Member
May 27, 2019
788
328
63
Austin, TX
Whats your budget?
This thread has been ongoing, I grabbed one of the boards for $135 with tax after making an offer: https://forums.servethehome.com/index.php?threads/supermicro-x10srh-cf.28009/

The board has 10 sata, 8 SAS (flashable to IT Mode), 3 - 8x Pcie (one in a 16x slot), comes with dedicated IPMI port and 2 - 1gb intel ports.
The v3 cpus are very reasonably priced, I like the 2660 v3 10c/20t for under $100, it does require a narrow ILM cooler though.
(if this is solely for NAS you can find solid sub $50 2620/2630 v3 chips too)
The DDR4 rdimm ram is likely the most expensive part of it if you want 64gb (about 160-200$ depending on stick size)
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
Hello Spartacus,
Do not have a fixed budget.
So far have bought the following:
MB: MSI x470 gaming plus max- MSI X470 Gaming Plus Max | PcComponentes.com

RAM: 64G (4x16 kit)of Crucial Ballistix 2400

CPU: Ryzen 5 1600AF(actually a ryzen 2600)

HBA: I currently have an LSI 9211.8i
On th eway HP H220 (Nuevo HP H220 9205-8I 660088-001 638834-001) and an IBM sas expander (IBM (46m0997) and a couple of sas cables to connect it.

Network: 1xHP 530SFP+ dual 10G card

PSU 600W Silencer MKIII

CASE:UNYKAch UK 4129

Fans: 3XArctic p120(one for each hdd bay) 2X140 in the middle and 1x80 on the exhaust

HDD: 12XHGST 4kn sas12 , 3X10TB WD red, 1x8TB Seagate enterprise encrypted drive(for back-ups)

The only missing part is NVME cache disks, that I was thinking of buying a couple of hibrid intel H10 1TB(32GB of optane on each for intent log, +1TB of qlc which i was thinking to overovision as 256G in order to work as SLC nand).


If anyone has any sugestions on how to improve on this I am open to suggestions.
 

CrimsonMars

New Member
Feb 14, 2019
14
0
1
So the build is finally ready after a long time, some pieces took forever due to COVID situation...

Got the HP220 and the HBA and a IBM SAS expander delivered not long ago.

Just installed Truenas as OS, and am a little puzzled to what SSD I vould need for best performance.
I have 2 4x NVME ports on the MB free right now and some SATA ports where I can put around 4 more SSD´s.
But as opposed to cache and LOG now it seems that I can add Metadata disks and Dedup disks, but have no guidelines for sizing...

Right now I have the following 3x10TB in Raidz1(planning to expand with a couple of disks in the future)
10X6TB raidz2 with 1 hot spare.
Another 8TB disk just for VM back-ups and stuff(I have back-ups on another NAS as well that is why I am not concerned very much).

Would be interested to turn on DEDUP on the 5x10TB volume and turn on always sync on the 10x6TB one.
Dedup would be nice on the standalone disk, but I guess will not get much out of it as Veeam already does dedup...

LOG I know it should be roughly 2 GB/volume on really fast and reliable disk.
Cache well the more the merrier but I guess 3-4 500G SSD´s should be fast enough for 20GB/s.


Metadata... well unknown, I thing it sould be between 0.03 and 0.3 of the total net size someone confirm or correct me.
Dedup.... well have no ideea how big a dedup disk should be if I rember correctly is 10% of the disk but it seems kind of big.

What kind of SSD/NVME disks would be advisable in my case?

Was thinking on INTEL 2x 16GB M10 or 2x 32G/512G H10 or some 512 MLC samsung drives inside NVEM slots (think optane is the best for log).