Building a new unRAID/NAS server, based on a CSE-826.

[UnbentTulip]

Don't really have any questions regarding this build, but I thought it'd be fun to post a follow-along. I am very much open to comments and suggestions, though!

Currently I have unRAID running on a HP Z620, with an E5-2660 V2, and 32GB of ram. Currently also maxxed out on the internal drive bays, which led me to starting this project. Sort-of.

Backstory? Studying to get certs, I've been falling in love with enterprise equipment. Rack chassis, the crazy hardware, all that fun stuff. I also like making franken computers/servers. So, thus started my project.

Issues I have with my current box, is lack of 3.5" drive bays. There's only 3 internal. I could add in a 5.25 adapter, but there's also a heat issue. I see idle temps of my drives in the 40C range at times in the warmer months. Definitely more than I like.

Then I stumbled upon the Supermicro 826 chassis.. Then I stumbled upon a good deal on one. Now the work begins. Obviously with those stock 80mm fans aren't exactly quiet, nor are the PSU's. And it came with some antiquated hardware. Came with an X8DTN+ motherboard with 2x Xeon X5620 processors.

With my goal being quiet, and low power consumption, running that configuration was a bit out of the question. I don't run any VM's, and mostly just the arr dockers. I also have adguard going, but that's planned to be moved once I build a new firewall.

I did some research and it seems unRAID prefers faster cores vs more cores, which led me down the path of the E3 lineup of intel processors. I found a good deal on a X11SSH-F motherboard, complete with an E3-1245 V5, perfect.

As it stands, I have the CSE-826 chassis, with a "TQ" backplane, stuck the X11 in it today, which introduced my first issue - the power cables are too short. So, I have extenders coming this week. Currently I have 32GB of ram in unRAID and see usage up to 50%, which isn't too bad, but I currently have 16GB on the X11 board.

Plans for the project:
I put 920SQ psu's in it, but I read somewhere there is a ~500W psu that is about 27-28Db, so I may switch to that if I can find it and it fits, as the 920's will be way under utilized.
I may swap the 1245 V5 for a 1270-1280 V6 for more single core power.
I'm looking at a 9300 HBA for SAS3 speeds. Even though I'm using spinning rust for drives, I'd like a fast cache.
Put in a mellanox Connectx-3 NIC, dual qsfp port.
Upgrade to at least 32GB of ram, may max it out at the 64 if I can find a decent deal on 2133 DDR4 16GB ECC UDIMM sticks.
I have the quieter San Ace fans on order, coming from China unfortunately, so those will be a while.
Maybe an M.2 cache drive? The motherboard supports it.
upgrade to the Supermicro passive cooler

Overkill for an unRAID NAS? Maybe, but I'm also attempting a little bit of "future proofing" and if I run out of drive slots, I'd like to build a JBOD to attach to it.
So, we'll see how it goes. Hoping I can get to same, or less wattage as my Z620, time will tell.

Photos are before and after motherboard transplant.

 

[kpfleming]

I have a pair of the 500W PSUs in my CSE-826 and they work fine and are reasonably quiet (enough so that I haven't been tempted to replace their fans with Noctua fans). If you look at my post history here on the forum you can see the thread were I assembled a complete system from various used parts purchased on eBay... I've been quite pleased with the result.

 

[nexox]

The 500W that fits is the PWS-501P-1R, I'm just using one, with a 902-sq sitting in the other bay pulled out an inch to keep the buzzer from complaining. I can tolerate the downtime of going to plug in the other PSU if the first one dies and this saves a few watts of power.

If you cut the indentation off the fan carriers you can install whatever 80mm fans, I have Arctic P8 Max in mine, they go quiet enough but you have to monitor each individual component to ensure everything is getting enough airflow, and that will make a passive cooler for the CPU a little questionable, I don't know if there's an active 2U cooler for that socket, but they usually work with a Noctua fan for lower power processors. For fans that spin down below 700 RPM you'll likely have to adjust the warning thresholds in IPMI, pretty easy. I have my 826 (with one Xeon 4114) just about whisper quiet at idle, though I still probably need to do some work to cool my NIC, HBA, and NVMe adequately, there's plenty of room for more fans.

 

[UnbentTulip]

I have a pair of the 500W PSUs in my CSE-826 and they work fine and are reasonably quiet (enough so that I haven't been tempted to replace their fans with Noctua fans). If you look at my post history here on the forum you can see the thread were I assembled a complete system from various used parts purchased on eBay... I've been quite pleased with the result.

Thanks! I'll have to take a look at it. I wish I could find the post I saw where the person got the dB specs from supermicro themselves. My main worry with the 920's is that they'll use a lot more electricity from not being run in an efficient range

The 500W that fits is the PWS-501P-1R, I'm just using one, with a 902-sq sitting in the other bay pulled out an inch to keep the buzzer from complaining. I can tolerate the downtime of going to plug in the other PSU if the first one dies and this saves a few watts of power.

If you cut the indentation off the fan carriers you can install whatever 80mm fans, I have Arctic P8 Max in mine, they go quiet enough but you have to monitor each individual component to ensure everything is getting enough airflow, and that will make a passive cooler for the CPU a little questionable, I don't know if there's an active 2U cooler for that socket, but they usually work with a Noctua fan for lower power processors. For fans that spin down below 700 RPM you'll likely have to adjust the warning thresholds in IPMI, pretty easy. I have my 826 (with one Xeon 4114) just about whisper quiet at idle, though I still probably need to do some work to cool my NIC, HBA, and NVMe adequately, there's plenty of room for more fans.

I'll have to take a look at those PSU"s. I got the 902's for less than $20/ea, so if anything they'll be fine sitting on a shelf until needed.

And I'm a dummy, I meant an active CPU cooler. The board came with a stock Intel one, but I'm sure supermicro is a little better at cooling. I know Noctua also makes a small one for the socket as well.

 

[nexox]

The board came with a stock Intel one, but I'm sure supermicro is a little better at cooling. I know Noctua also makes a small one for the socket as well.

Anything that's designed to move air front to back is probably an improvement over that desktop style stock cooler. If you have a 3D printer or just a proficiency with zipties you can also probably add a 60mm fan to a passive 2U cooler to make a decent active cooler, it'll probably come out cheaper since you will likely want to buy a fan to replace whatever comes with an active 2U cooler.

 

[UnbentTulip]

Anything that's designed to move air front to back is probably an improvement over that desktop style stock cooler. If you have a 3D printer or just a proficiency with zipties you can also probably add a 60mm fan to a passive 2U cooler to make a decent active cooler, it'll probably come out cheaper since you will likely want to buy a fan to replace whatever comes with an active 2U cooler.

Noctua makes a couple active coolers for the lga1151 that are as short, or shorter than the supermicro one. Unfortunately they're top-down airflow. But, if they're angled right they have horizontal fins and not the circular style, so they could get some passive cooling from the fan wall also.

I figure if I can keep the CPU cool, the fan wall will stay pretty low unless the drives need cooling.

 

[nexox]

I would probably go with something designed for a 2U chassis, but the top-down type may be good enough. Also don't expect the fan wall to spin up to cool your drives (or NIC,) supermicro boards pretty much just modulate fan speed based on CPU temp, if you need them to do something else there are scripts you can use that should allow control from userspace, but you only have two independent zones usually, the fan ports with a number at the end are all grouped up, then the fan ports with a letter at the end are in the second zone.

You may also want to block off air flow on the PSU side to make more of it go past the RAM and CPU, and potentially on the other side to direct air over the PCIe cards. Cardboard and tape work if you don't want to get too fancy.

 

[UnbentTulip]

I would probably go with something designed for a 2U chassis, but the top-down type may be good enough. Also don't expect the fan wall to spin up to cool your drives (or NIC,) supermicro boards pretty much just modulate fan speed based on CPU temp, if you need them to do something else there are scripts you can use that should allow control from userspace, but you only have two independent zones usually, the fan ports with a number at the end are all grouped up, then the fan ports with a letter at the end are in the second zone.

You may also want to block off air flow on the PSU side to make more of it go past the RAM and CPU, and potentially on the other side to direct air over the PCIe cards. Cardboard and tape work if you don't want to get too fancy.

The stock fan shroud fits over the board/cpu, so it's directing two of the 80mm fans directly over the CPU and RAM, and then that leaves one fan blowing directly at the PCIe cards.

Even if the fan wall doesn't spin up much because of a cooler CPU, I'm sure it'll be more cooling than the drives currently see in the 620 case, they don't even have a dedicated fan. The front fan sits below the drives, so all they get is whatever excess the rear fans pull through the grille. I get a high temp warning (45C) sometimes in warmer months.

 

[UnbentTulip]

Got the 24-pin and 8-pin extender for it, today so I was finally able to boot the system up!

The SQ power supplies are definitely a lot quieter than the ones that came in. I have some of the 500W versions coming.

The 80mm fan wall fans aren't too bad on optimum, just a little louder than I would like, so I am eagerly awaiting the replacements. Everything was idling really cool (mid 20°C) EXCEPT the CPU. It kept hovering in the 40°C range at idle, and would quickly spike up under any load, resulting in the fan wall fans to jump as well. Looking into a better cooler for that.

Definitely a quick little box, just playing around in Ubuntu right now to check firmwares and such. Power usage at idle is 70W with no drives, the smaller power supply should help that. So it seems like my "Goals" are achievable.

If I didn't mention those goals in my OP, it's to make a faster machine with more bays, at same or lower wattage. I sit about 120 at idle with my current unRAID box and 3 drives.

 

[UnbentTulip]

Update time. I think I have all of the parts for the completion of this on the way. Still waiting for fans to ship, unfortunately.

Received the 500W PSU's in today, and here's some wattage numbers courtesy of IPMI view

Single PWS-920P-SQ at idle - 38W
Single PWS-501P-1R at idle - 30W
Dual PWS-501P-1R at idle - 22 + 23, for a total of 45W

Seemed a little odd to me that it's splitting the power usage between the two PSU's, and not having one as idle and one using power? Is there a setting that needs to be changed for that? I'm fine using the two if one of them is sitting idle at a couple of watts while the other one is doing all of the work. But as of right now it seems it's attempting to split the load evenly between the two.

 

[BlueFox]

Seemed a little odd to me that it's splitting the power usage between the two PSU's, and not having one as idle and one using power? Is there a setting that needs to be changed for that? I'm fine using the two if one of them is sitting idle at a couple of watts while the other one is doing all of the work. But as of right now it seems it's attempting to split the load evenly between the two.

That is normal and expected. It's not something you can change.

 

[ShutterBC]

70W idle seems to be a pretty decent improvement over the original. I have an SC826TQ with an X8DTN+ with dual L5520s and it idles well over 120W, though having 6 7200RPM drives probably contributes a bit to that.

Been thinking about going away from rackmount to a 3 node cluster of mini PCs, personally. However I haven't decided how I'd migrate the storage away from a single large ZFS array yet.

 

[UnbentTulip]

That is normal and expected. It's not something you can change.

I'm finding this out! This is my first time really digging into this kind of hardware. I have another 1U server, but the most I've done with it is turn around and play with it for a few minutes due to noise.

It seems some have a BIOS setting to change how it balances the PSU's, but I think I'll just leave one unplugged for now. Don't really need the redundancy for when I'm at home. And even then, to do it "proper" they should be on separate UPS', circuits, etc.

70W idle seems to be a pretty decent improvement over the original. I have an SC826TQ with an X8DTN+ with dual L5520s and it idles well over 120W, though having 6 7200RPM drives probably contributes a bit to that.

Been thinking about going away from rackmount to a 3 node cluster of mini PCs, personally. However I haven't decided how I'd migrate the storage away from a single large ZFS array yet.

I have 3 7200 drives in the server that this one is going to replace (16, 14, and 12tb). So that will add a few watts to it overall. But I agree. Having a lower core CPU definitely seems to help.

I'm trying to do the opposite, here. I'm wanting to migrate to rack mount, and utilize more features like IPMI. Next build will be moving my firewall from an industrial fanless into something like a CSE-512 case, and probably another x11 m-atx board.

I had a fun thought of putting a couple micro optiplex's into one of those cases. But then looking at how to try and power all 4 from one power source was going to cost more than doing new builds.

 

[UnbentTulip]

Been a minute since I did an update.

Finally got everything up and running. Was going to use the TQ backplane that came in it, but by the time I priced cables, an HBA, AND an expander, it ended up being cheaper (for now) to run the SAS3 backplane with a 9300 HBA.

Bought a cooler for it that supposedly worked really well in MicroATX builds, but when the server was "idling" it was struggling to get down to low 40C, would often sit high 40 to mid 50C temps, which I didn't like.

Found a 2U compatible "Cooljag" active cooler for 115x sockets, and the CPU is hanging out at 30C while the server is "idle" which I am comfortable with.

The low noise fan wall fans I ended up getting were NOT PWM, despite the ebay description saying they were, so they haven't been changed yet. But I have the active CPU cooler on "FAN1" plug, and then the fan wall fans on "FANA" plug, and unRAID has an IPMI plugin with fan control, so I have them turned down. As they're on separate zones.

Overall, it is QUIETER than my Z620, and with 5 spinning discs IPMI says my 24hr average wattage is 84W.

Attached is the inside of it under it's "final" iteration, until the PWM fans come in.