[Feedback sought] Enthusiastic NAS build [4.5y usage update 2019-12]

[CreoleLakerFan]

• Which ATX power supply to get? Should be very silent and energy efficient. (Eyeing an 80 Plus Gold or Platinum model) Will 450W be ok-ish, even if I decide to add more HDs later up to a maximum of 10 (LFF) and 2 (SFF) which fit in the chassis? (ATX PSU up to 260mm fits.) I also need to get power to all the spindles.

I have a FreeNAS 9.3 based system running 8x4TB green desktop drives, and 32nm Pentium/Celeron G550. The system is powered by a 300W unit. Before enabling powerd and turning on drive APM it drew a constant ~83W. I have a couple of SM 2750 boards (miniITX variety) and I can tell you that they idle much lower than this board/cpu combo. 450W of PSU is definitely enough for your use-case, you probably would realize savings in both CapEx and OpEx with a lower wattage unit.

CPU: The Atom C2750 has 8 cores (8 threads, no hyperthreading) and provides plenty of multithreading speed yet has very low power consumption and support for up-to 64GB RAM. (ZFS and VMs looove RAM, though I'll start with 32GB.)

<snipped>

• Xeon E3-1230 v3: Significantly higher power consumption. Limited to 32GB RAM. (I intend to upgrade to 64GB RAM within a year.) E3-1230 vs. C2750

How many jails are you planning on running on this box? You also stated that the use-case for this platform is home use/media serving. I recently ripped out 16GB of RAM in my FreeNAS unit and replaced it with 8GB ... I do not run any jails, but I have no issues saturating Gbe links to multiple PCs from this server. 32GB is likely overkill in your scenario. 64GB is massive overkill ...

Also consider whether the 2550-based board is a better fit. There is no power savings, but $100 price difference. I have no idea whether jails and/or Plex can run multi-threaded, but it's something to consider. The 8-cores of the 2750 bring more processing power than you would think.

 

[NeverDie]

...VMs for mixed software/OS services and testing. (CPU is VT-x/EPT capable.)

Are you planning to do "mixed software/OS services and testing" (not really sure what that means) within FreeNAS, or were you planning to run ESXi or Hyper-V as the host and creating an all-in-one? The answer might affect the feedback you get here.

 

[roberth58]

The rough guide for Plex HD transcoding seems to be a CPU passmark of 1500-2000 per stream. The C2750 is 3929, You may need a little more CPU for dual Plex streams along with your other requirements.

 

[Philbar715]

Hold that thought for a moment, because I really do want to get to the bottom of this once and for all. When I compare e3-1220L v3 (ARK | Intel Xeon Processor E3-1220L v3 (4M Cache, 1.10 GHz) ) with e3-1220 V3 (ARK | Intel Xeon Processor E3-1220 v3 (8M Cache, 3.10 GHz) they look completely different. The e3-1220Lv3 (13 watts TDP) has half the cores and 1/3 the base frequency of the e3-1220v3 (80 watts TDP). Yet you're saying that at idle they're really no different? At least on the face of it, that just doesn't sound right.

@MacLemon: v2 and v3 are still "launch phase." v1 is EOL, though I'm not sure why that alone would matter.

I have an E3-1230 V3, it idles at around 8-12W (CPU only) Under 100% load it does hit its TDP of 80W or so, but 95% of the time my server is idle. It also streams plex to a few different devices. While transcoding a 1080P stream to my phone it uses up to about 45W, but thats only in small bursts of about a second or two, then its back to idle.

 

[NeverDie]

I have an E3-1230 V3, it idles at around 8-12W (CPU only) Under 100% load it does hit its TDP of 80W or so, but 95% of the time my server is idle. It also streams plex to a few different devices. While transcoding a 1080P stream to my phone it uses up to about 45W, but thats only in small bursts of about a second or two, then its back to idle.

How were you able to measure the wattage to the CPU? Is there a bios measurement or something that reports it?

I also have the E3-1230V3. With 16GB of memory, the CPU, stock inte CPU cooler fan that came with the CPU, motherboard, memory, an SSD, and the picopsu power supply and the 12v brick that feeds it all together consume about 31 watts (as measured on a kill-a-watt type device) when FreeNAS is idling.

 

[Philbar715]

How were you able to measure the wattage to the CPU? Is there a bios measurement or something that reports it?

I also have the E3-1230V3. With 16GB of memory, the CPU, stock inte CPU cooler fan that came with the CPU, motherboard, memory, an SSD, and the picopsu power supply and the 12v brick that feeds it all together consume about 31 watts (as measured on a kill-a-watt type device) when FreeNAS is idling.

HW monitor reports the power usage if your motherboard supports it. Heres a screenshot that I posted in my thread about the idle temps.

 

[CreoleLakerFan]

How were you able to measure the wattage to the CPU? Is there a bios measurement or something that reports it?

I also have the E3-1230V3. With 16GB of memory, the CPU, stock inte CPU cooler fan that came with the CPU, motherboard, memory, an SSD, and the picopsu power supply and the 12v brick that feeds it all together consume about 31 watts (as measured on a kill-a-watt type device) when FreeNAS is idling.

How many drives?

 

[NeverDie]

How many drives?

The test setup I've been doing measurements on has just a single 250GB Samsung EVO 850.

 

[HellDiverUK]

With my Xeon E3-1225v3 on a MSI B85M-Eco board, two DDR3L DIMMs, Intel stock cooler and one SSD, I've seen Windows idling as low as 22W using a FSP 150W 1U PSU. Typical draw is in line with NeverDie's 30-odd watts when the machine is actively doing stuff with an additional 5TB WD Red running.

The p9D-M uses a little more power due to the ASpeed chip, and the two i210AT NICs use a bit more than the single i218-V on the MSI.

 

[CreoleLakerFan]

I really need to get the data on my FreeNAS backed up so I can see how Windows idles on the same hardware.

 

[NeverDie]

With my Xeon E3-1225v3 on a MSI B85M-Eco board, two DDR3L DIMMs, Intel stock cooler and one SSD, I've seen Windows idling as low as 22W using a FSP 150W 1U PSU. Typical draw is in line with NeverDie's 30-odd watts when the machine is actively doing stuff with an additional 5TB WD Red running.

I wonder whether or not Windows is due some credit for that. I recently tested a very different board with a Baytrail CPU, and straight out of the box Windows saved about 5 watts while idling as compared to just Debian while idling. I didn't attempt to tune either one.

 

[EffrafaxOfWug]

Drivers notwithstanding, debian has very little in the way of on-by-default power management that some other ditros use as a matter of course (as does windows). Try adding the following udev rules and see what the parity is like; on my kit brought debian usage to within margin of error for windows on the same hardware:

ACTION=="add", SUBSYSTEM=="pci", ATTR{power/control}="auto"
ACTION=="add", SUBSYSTEM=="usb", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="add", SUBSYSTEM=="scsi_host", KERNEL=="host*", ATTR{link_power_management_policy}="min_power"

 

[MacLemon]

Preamble: Sorry for the pause, had a few very busy days preparing the local CryptoParty event/talk. Oh, and customers.

[…] HDDs being different sizes […] Most hardware RAID cards do rounding to get past this problem. The LSI cards do this rounding so that you can mix/match different vendors. You can do the same with software raid just by not using the drive to the full potential.

Will the rounding be applied in IT mode as well? I'd doubt that, though I can't verify that myself. After all the point of IT mode is that the controller does not interfere between the disks and the OS for ZFS use.

Would be interesting how ZFS deals with drive sizes when they only marginally differ. Anyone got an insight on that or could test that?

[…]450W of PSU is definitely enough for your use-case, you probably would realize savings in both CapEx and OpEx with a lower wattage unit.

How many jails are you planning on running on this box? You also stated that the use-case for this platform is home use/media serving. I recently ripped out 16GB of RAM in my FreeNAS unit and replaced it with 8GB ... I do not run any jails, but I have no issues saturating Gbe links to multiple PCs from this server. 32GB is likely overkill in your scenario. 64GB is massive overkill ...

Also consider whether the 2550-based board is a better fit. There is no power savings, but $100 price difference. I have no idea whether jails and/or Plex can run multi-threaded, but it's something to consider. The 8-cores of the 2750 bring more processing power than you would think.

I'd expect there to accumulate about 20 lightweight jails. I like to have some room to grow should I run into unexpected hosting ideas. As I said, I plan on starting with 32GB and see how that works out. If that turns out to be plenty, I'll skip the 64GB upgrade.

Plex definitely can use multiple cores/threads which I have verified. I guess it doesn't make use of available SSE extensions though. I've also given my thoughts on C2550 vs. 2750.

Are you planning to do "mixed software/OS services and testing" (not really sure what that means) within FreeNAS, or were you planning to run ESXi or Hyper-V as the host and creating an all-in-one? The answer might affect the feedback you get here.

My plan is not to run a Type 1 Hypervisor like ESXi. I have considered SmartOS but FreeBSD/FreeNAS seem a much more suited choice for my needs. Also considering that SmartOS has really narrow hardware/driver support and very few packages available in comparison.

Currently I'm leaning towards FreeNAS and Jails plus the occasional BHyve or maybe Virtualbox VM. That way I hope to get all the comfort from FreeNAS, all the flexibility for additional ideas from FreeBSD and I wanted to look into BHyve anyway. It's unlikely that I need a permanent(ly running) *tux or Windows VM.

The rough guide for Plex HD transcoding seems to be a CPU passmark of 1500-2000 per stream. The C2750 is 3929, You may need a little more CPU for dual Plex streams along with your other requirements.

Could you please share a source link to that info? I've got a similar board with an intel C2758 here for the moment so I can do some rough estimations on the performance.


-------

And now for something completely different…

New infos on the cases.

As you may already know, I've contacted Fractal-Design about the mounting possibilities of the SuperMicro A1SA7-2750F. After some back and forth they came to the conclusion that they have no clue if it could fit with a tendency for “no”.

So extrapolating from the boards measurements, I expect it to mechanically fit into the NODE 804, but I'd need to build my own custom mounting solution/adapter. Given the large gaps in the compartment separation frame I expect this to be a problem.


As NeverDie suggested a Nanoxia case, I had a closer look at their range and also watched a few Case reviews on YouTube if which there are plenty of varying quality. (Most people seem to build gaming rigs or liquid-cooled show-off boxes. (Personal opinion/observation))

The Nanoxia Deep Silence 1 or Nanoxia Deep Silence 6 (Rev. B) seem like a very attractive case for my build opposed to the Fractal Design NODE 804. Especially given that this box will operate stashed away out of sight.

I've also contacted Nanoxia about fitting the aforementioned SuperMicro motherboard. Their support was very friendly and immediately seemed to understand my question. \o/ They clearly stated that none of their cases had native mounting support for the proprietary-form-factor board (which is expected). Though their frame lends itself a lot better to either drilling custom mounting holes for the standoffs or a self-built adapter plate/frame.

Feature comparison of the cases:
*Dang* this forum engine does not support BBCode for table layouts.

Fractal-Design NODE 804
3.5" HDD bays in carrier: 8+2
2.5" SSD bays: 2
Motherboard standards: Micro ATX, Mini ITX
H: 307mm
W: 344mm
D: 389mm
Weight: 6kg
Total 3.5" HDDs that can be fit by reusing the 5.25" bays for 3.5" drives: 10


Nanoxia Deep Silence 1
5.25" Bays, externally accessible: 3
3.5" HDD bays in carrier: 8
2.5" SSD bays: -
Motherboard standards: ATX, XL-ATX, Micro-ATX, Mini-ITX
H: 517mm
W: 220mm
D: 532mm
Weight: 11.34kg
Total 3.5" HDDs that can be fit by reusing the 5.25" bays for 3.5" drives.: 11

Deep Silence 6 Rev. B
5.25" Bays, externally accessible: 4
3.5" HDD bays in carrier: 10+3
2.5" SSD bay: 6
Motherboard standards: HPTX, E-ATX, XL-ATX, ATX, Micro-ATX, Mini-ITX
H: 644mm (672 mm)
W: 250mm
D: 655mm[/td]
Weight: 20.8kg
Total 3.5" HDDs that can be fit by reusing the 5.25" bays for 3.5" drives: 17

---

Since BlueLineSwinger pointed out that the RAMs I originally posted would not fit the C2750 CPU/board:

According to SuperMicro there are 16GB Modules by Intelligent Memory that are tested by SuperMicro. They don't seem to be easily available though. DDR3 1.35V-1333 ECC (No 1600/ECC modules seem to be tested at 16GB.)


Best regards
Pepi

 

[Chuckleb]

If you are using IT mode, the drives are passed through so no, the card won't do any rounding. It's up to whatever creates the arrays to decide on the rounding factor. For example, if you wanted to be safe, you could call all 4TB drives 3.95TB drives, which lets you use any drive in the array. You can easily do this in mdadm by building out the partition that will become a member of the array to be only 3.95 and leaving the rest unused. This results in the safest way to be able to mix and match drives between different vendors and models. Yes you lose a small part of the array, but it's pretty small for the convenience.

 

[NeverDie]

Though their frame lends itself a lot better to either drilling custom mounting holes for the standoffs or a self-built adapter plate/frame.

Maybe it's just because I haven't done it before, but I would would worry that drilling a case might be "easier said than done." Getting the holes to properly register, wandering drill bits, finding a properly sized cobalt drill bit, metal shavings finding their way into the case as a short-circuit hazard, etc. An adapter plate sounds like it would be easier to fit on a drill-press.

 

[MacLemon]

[…] call all 4TB drives 3.95TB drives, which lets you use any drive in the array. You can easily do this in mdadm by building out the partition that will become a member of the array to be only 3.95 and leaving the rest unused. […]

Of course this makes setting up a RAID Z2 a completely manual process. Seems to be the only technical solution I could think of either.

[…]drilling a case might be "easier said than done."

I likely do have access to tools, equipment and people with the necessary know-how to do this at my local hackspace. But it's certainly not a recommended approach.

After digging through a lot of the details in the user manual I came to the conclusion that SuperMicro actually doesn't want customers to use that motherboard anywhere but in a single SuperMicro chassis. So I'm afraid I have to rule out the A1SA7 2750F and go different route.

So I'm still looking at the SuperMicro A1SAM 2750F, despite its small CPU fan.


Sadly there are not many C2750 boards out there. These are the only ones I could find, excluding those that are known to be impossible for such a build.

• http://www.supermicro.nl/products/motherboard/Atom/X10/A1SAM-2750F.cfmA1SAM-2750F
• SuperMicro A1SAi-2750F
• GIGABYTE GA-9SISK
• ASRock C2750DI4 - with the problematic Marvell controllers…

Best regards
MacLemon

 

[Spotswood]

Its not that hard to drill new standoff locations. The key is a good template/tracing of the board and selecting a case with enough/flat material to drill into, such as a Lian Li or CaseLabs.

 

[Patrick]

When are you planning to purchase the hardware? That may have an impact on my recommendation for a CPU/ motherboard.

I think the C2750/ C2550 are great for low power.

For the E3 series, my recommendation today may be different than in a few weeks.

 

[pgh5278]

When are you planning to purchase the hardware? That may have an impact on my recommendation for a CPU/ motherboard.

I think the C2750/ C2550 are great for low power.

For the E3 series, my recommendation today may be different than in a few weeks.

That is good as a nudge nudge wink wink , something interesting coming along>>>

 

[MacLemon]

Its not that hard to drill new standoff locations[…]

There's also no I/O shield available at all for the A1SA7 2750F, so even if I were able to get exact technical plans for the mounting holes that motherboard, while having excellent features, is sadly only made for fitting SuperMicro's own chassis.

Therefore, after thinking this through, I've decided to skip this motherboard for the reasons mentioned.

When are you planning to purchase the hardware?[…]

I hope to start within the next 2 weeks. From the piblic intel roadmap (PDF) I don't see anything to be released by intel in the Atom CPU space that may affect my favoring for the C2750.
If one were to look at an E3 CPU, you may want to wait for the v4 generation provided you want to/can wait until its release.

--

So at the moment' I'm looking at this updated component list:
Bits that have changed from the original list are shown in italics.

Build’s Name: Work in progress
Operating System/ Storage Platform: FreeNAS/FreeBSD
CPU: Intel® Atom™ C2750
Motherboard: SuperMicro A1SAM-2750F
Chassis: Nanoxia Deep Silence 1
Drives:

• 6 × HGST HDS5C4040ALE630 PDF Datasheet
• 3 × Existing Seagate Spindles as secondary RAID Z1 for backup purposes only.

RAM: 2 × Intelligent Memory IMM2G72D3LDVD8AG-B15E 16GB modules. (Something less pricey would be preferred)
Add-in Cards: 1 × IBM ServeRAID M1015 (2 × 4 Port SATA/SAS HBA)
Power Supply: TBD Please, see below!
Other Bits: unchanged

Reasoning for the changed HDs and chassis have been given before.

I'm not yet sure about the power supply and power estimation.

For around 10 LFF HDDs I'd estimate 10 × 10W = 100W
Motherboard and CPU: 30W
IBM M1015: 13W
RAM: 8W
Fans, other small bits: 10W

Total: < 200W


This PSU was suggested:

[…]Rosewill SilentNight Series SilentNight-500 500W,80 PLUS PLATINUM Certified,Fan-less,Modular Design,Single +12V Rail,ATX12V v2.3/EPS12V v2.92,SLI Ready,CrossFire Ready,Active PFC Power Supply - Retail - Newegg.com[…]

500W seems oversized to me for this build, even with a large reserve margin. The fanless design though strikes a nerve.

I guess I'd be very fine with somewhere between 300W - 400W siding on the smaller wattage.

Best regards
MacLemon