FreeNAS based on Asrock C2750D4I

rune.hogh

New Member
Feb 18, 2014
12
14
3
Hi All,

New to the forum and thought I'd post some experience here.

5 years ago I built a server for use in a dorm, but now it is time for an upgrade.
Like the previous server, the new one had to be as low-powered as I dared to go without sacrificing network speed. I opted to go for the new Avoton and hoped it was for the best. The specs are:

MB & CPU: Asrock C2750D4I
RAM: 4 x 8GB Kingston KVR16LE11/8EF, ECC
PSU: Seasonic Platinum Fanless Series - 520W
HDD's: 6 x WD Red WD30EFRX 3 TB
System drive: SanDisk Cruzer Fit USB 16GB
Cabinet: Cooler Master Centurion 590 (from the old server)
OS: FreeNAS 9.2.1 x64

The system is right now running, but thought I'd share some more.

So before starting any OS installation, I updated the BIOS on the MB to the latest (v1.80) from Asrock - I had heard of people having trouble installing FreeNAS on this MB, so thought I'd better do some upgrading.
The OS was installed over IPMI through a browser (you need JAVA installed for that). One thing i noticed was it took a long time for it to boot up the installation media - maybe because it was over IPMI, dunno. But running through the installation progress, I found no particular issues. I just pointed the installer to the SanDisk USB flash and it installed without much issues.



As this is my first experience with FreeBSD, FreeNAS and ZFS, I still have a lot to learn. I don't really do in benchmarks, but if you guys want me to (and tell me how) I would be glad to try something out.

The setup:
  • HDD's configured in RAIDZ2 giving 11,4TB
  • System writing speeds is around 80MB/s from my 1Gbit wired network through AFP
  • Average power consumption over the last 48 hrs was 55W albeit most of the time in idle.

Im sure a lot of improvements can be done over time, but I still have a lot to learn. But for those thinking of getting an Avoton for NAS, it seems to be doing a good job. And for those doubting FreeNAS could boot on this MB, it has just been proven. I'll keep updates coming in.
 
Last edited:

odditory

Moderator
Dec 23, 2010
376
53
28
Nice post and I'm working on a similarly spec'd system for a review. A few questions:

1) What made you decide to go with the ASRock board over, say, one of the Supermicro boards like A1SRi-2758F with quad ethernet?

2) Did you consider any other software solutions before deciding on FreeNAS? Example a roll-your-own Synology/Xpenology? Just curious.
 
Last edited:

rune.hogh

New Member
Feb 18, 2014
12
14
3
1) What made you decide to go with the ASRock board over, say, one of the Supermicro boards like A1SRi-2758F with quad ethernet?
Well, I like the Asrock's 12 onboard SATA ports and it looked like a sure thing for NAS. I realize the Asrock might not be the same server quality as the SuperMicro, but weighing out the differences I cam to the conclusion that I'd rather have 12 SATA ports than 4 Gbit ports.

2) Did you consider any other software solutions before deciding on FreeNAS? Example a roll-your-own Synology/Xpenology? Just curious.
Well, so my old server was a Windows Server 2003 which is really not that home-server-friendly as other solutions. Mostly I would have preferred a linux box but for a server like this, I dont really see that happening before Chris Mason builds BtrFS' raid capabilities more stable. So again and again, the researching on the internet led me back to one of the BSD-solutions; FreeNAS, NAS4Free and ZFSGuru. These variants have the ZFS-filesystem which BtrFS is trying to catch up on. Besides, a friend of mine already is using FreeNAS, so thats the first one I tried and worked out so far.

I will say though, in all my searching the internet, I never came across Xpenology - I'm guessing it the original firmware for Synology modded to run on a wider range of hardware?
 
Last edited:

DolphinsDan

Member
Sep 17, 2013
90
6
8
Yes it is. From what I have seen on the Xpenology forums and here it seems like that is a daunting installation and there are some features that do not work correctly.

You know, if you did get all of this working, that's great.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,520
238
63
80MB/s with 32GB ram? hmm, I would expect a bit higher transfer speeds
Why would that matter?

System writing speeds is around 80MB/s from my 1Gbit wired network through AFP
This is write speed to L2ARC does not help you. If you are writing to RAM you start having a huge risk to data loss so really 125MBps or so would be the maximum you would see with a single gigabit connection. 80MBps without knowing the host and network config, plus it is parity writes in BSD makes me think 80MBps is not bad.
 

rune.hogh

New Member
Feb 18, 2014
12
14
3
80MB/s with 32GB ram? hmm, I would expect a bit higher transfer speeds
I would expect more too, BUT - I should mention that all HDD's are connected through the Marvell SATA chips and not the internal SATA port of Avoton. This was done because I ordered some SATA cable not quite long enough the reach the other side of the board. Being based off FreeBSD and the whole delayed driver support, I don't know whether the Marvell chip drivers are good enough for 125MBps speeds or not.

I do not know if FreeNAS used the RAM as a write buffer, but I would think not. Opening a shell with "top" running, I can see that during the 80MBps write speeds, none of the cores are loaded fully.
I'll do a read test when I get home.
 

rune.hogh

New Member
Feb 18, 2014
12
14
3
So came home and tried some more focused testing. Basically I just transferred files back and forth while observing "top" and transfer rates. One thing about "top" - when it measures % cpu usage, 100% means 1 core fully loaded. That means as the Avoton is an octo-core, it can go to 800% in "top" - so the high % cpu usage below is in reality not that high.

Reading a 30 GB file from the server over 1Gbit wired network:
Speed = 102-110MBps
CPU = 65-73%

Writing a 4 GB file to the server over 1Gbit wired network:
Speed = 98-100MBps
CPU = 60-90%

So as you see, during maximum transfer speeds a single core is not even maxed out - that leaves the 7 other cores to do something else :D

Some pics of the build (if I can get the link to work)


Took the side and front off, so that you can see what it looks like.


In addition to the 3 x 120mm in the front, another 2 x 120mm fans are in the top sucking out the air. All fans are Scythe Gentle Typhoon 500rpm which are virtually silent - only 5dB.


I chose to do something extra when it comes to wiring. All the computers I ever built, I always hated the wiring because it look like s**t. So for this job and to preserve some sort of flow through the cabinet, I decided to pay a little extra to get some nice adapters.

  • The blue wires are 3-pin expanders - 3 fans controlled by one output of the MoBo. In this setup, the MoBo can control all fan through PWM.
  • I bought short 50cm SATA cables in order to have as little excess wire as possible. Turns out the were so short that they just exactly couldn't reach the SATA ports on the far side of the MoBo. Over there is the SATA ports connected directly to the Avoton - the one I'm using is the Marvell-chip expanded ports. I was doubting if they were supported in FreeNAS (FreeBSD), but turns out they work fine.
  • Lastly the power cables powering the WD RED SATA drives are also expanders. Means 1 SATA power plug from the PSU can now power 4 drives through the expander. The PSU is a fanless 520W Platinum80 PSU, so there should be plenty of power to give.


The inside is relatively clean from wire clusters. Also the PSU is modular so I only plug in the cables I need (which is very few).


The MoBo has a PCI-e x8 socket for expansion and is is open-ended. Dunno what I need it for, but it there.


On the left side of the MoBo, the 6 x SATA ports from the Avoton.

Also the server is equipped with 32GB ECC RAM (as recommended by FreeNAS community). These blocks were a b**th to get it - not because of the RAM block itself, but the socket. Normally a socket would have those flips in each end that squeezes in on the RAM when you push it down. For some reason, Asrock chose to go the sockets that only had flips in one end meaning I kind of had to pivot the RAM into the socket. And not all the sockets are the same shape - weird, but it works.


The FreeNAS system is installed on the SanDisk Cruzer Fit, but the is no on-board USB A-type plug, so it had to go in the back. It doesn't really matter because it is so damn small anyways.

Up until now I had run it without fan on the heat sink. Don't noticed it being crazy hot, but it was semi hot which is expected. On the other hand, the server is mostly idle. BIOS has an option though to active only 2, 4 or all cores if you wanna keep down the heat - I haven't tried it though.
Now I put a fan on there to see if it stays a silent machine. If not, I'll take the heatsink fan off again and rely on my superman skills to create a vortex in the cabinet only with the 5 x 120mm fans.
 
Last edited:

Hrast

Member
Oct 5, 2013
40
13
8
The speed that they're seeing is completely consistant with a single VDEV RAID-Z2, a single disk's throughput/IOPS. Want more speed, create more VDEVs and stripe the zpool across them.
 

rune.hogh

New Member
Feb 18, 2014
12
14
3
Go and get longer sata cables and use the intel ports, instead of the marvell ones.
^ Seconded.
The speed that they're seeing is completely consistant with a single VDEV RAID-Z2, a single disk's throughput/IOPS. Want more speed, create more VDEVs and stripe the zpool across them.
You see guys, I'm not posting a problem here. The 100+ MB/s is consistent with my 1Gbit wired network and that is what I was aiming for. This was meant as a build-story for others, who like me, was searching for any info on FreeNAS on a Avoton CPU and especially the one found on Asrock C2750D4I. What I found was vague and many said FreeNAS wouldn't boot on the Asrock. I would have loved to find something like this before I started the build - now others can ;)

When or if I eventually decide on getting the last 6 available slots in the cabinet filled with 6 more RED drives, then I'll be sure to buy longer SATA cables.

Right now, the server is working fine. Sure I had some issues with configuration of FreeNAS, but that is for a different thread.
 

odditory

Moderator
Dec 23, 2010
376
53
28
I hope you didn't wrongly interpret friendly suggestions as criticism? :)

It is simply best practice to for example use Intel ports before resorting to the add-on third party controller chips. Regardless of whether or not the Marvell controller seems working fine in this case, that's just the best practice and that was the point.

Regardless, great that the server is working for you, and thanks for taking the time to post the photos and write-up.
 
Last edited:

Masta123

New Member
Jun 29, 2015
1
0
1
35
Hi, I just joined this forum and I am pretty new to NAS builds. Do you know if I can do a raid z2 on this board across different RAID controllers. i.e. 8 hard drives across the intel & Marvell controllers.

Furthermore, It doesn't seem like you've had any issues with the Marvell Controller either. How has it been for you so far?

Why did you post that you were getting 80MB/s write speed in your first post and then later post that you were getting 98-100MBps?

I would like a system to be able to get 100 MB/s read/write as your earlier post seem to point out

"Reading a 30 GB file from the server over 1Gbit wired network:
Speed = 102-110MBps
CPU = 65-73%

Writing a 4 GB file to the server over 1Gbit wired network:
Speed = 98-100MBps
CPU = 60-90%"
 

rune.hogh

New Member
Feb 18, 2014
12
14
3
Sorry, I've been away from this forum for so long and replying now might do you any good, but for good measure I'm gonna do it anyways.

Hi, I just joined this forum and I am pretty new to NAS builds. Do you know if I can do a raid z2 on this board across different RAID controllers. i.e. 8 hard drives across the intel & Marvell controllers.

Furthermore, It doesn't seem like you've had any issues with the Marvell Controller either. How has it been for you so far?

Why did you post that you were getting 80MB/s write speed in your first post and then later post that you were getting 98-100MBps?

I would like a system to be able to get 100 MB/s read/write as your earlier post seem to point out

"Reading a 30 GB file from the server over 1Gbit wired network:
Speed = 102-110MBps
CPU = 65-73%

Writing a 4 GB file to the server over 1Gbit wired network:
Speed = 98-100MBps
CPU = 60-90%"
I'm sure you'd be able to do RAIDZ across Intel & Marvell no problem. The controllers don't do anything themselves other than making SATA ports available.

The Marvell controller worked but sometimes I would see a disk going bad. Restarting the server would help, but it was very irritating constantly thinking a disk or 2 went bad. I then did what other here posted - got longer SATA cables and connected it on the Intel ports. Since then I've had no problems at all (except the MB recently died).

In the beginning I had not always consistent transfer speeds - but since I moved the disks to the Intel controller, they have been consistently 100+MB. I haven't tried to Marvell ports since FreeNAS 9.2 - maybe today the Marvell ports have better support, I would not know.