SC846 system gifted to me - A full overview with questions. Replacing DVD drive with SSDs? Ideas for upgrades or keep what I got?

[UhClem]

... Thanks, I think based on all discussed here I'll just grab the E5-2667v4 to start and see how it goes. $30. Easy. ...

For that same ~$30, you can get E5-2690v4.
E5-2667v4:

E5-2690v4 (6 more cores! ; and still a good tight speed range):

And, for @itronin, E5-2680v4:

Yes, the 2680 is only ~$15 now, but, the 2690, with a 10+% boost across the full 14-core range is worth it. ??

 

[nabsltd]

Since no one has answered this, I would say that is something of a mistake replacing the BPN-SAS-846TQ with a BPN-SAS2-846EL1.

Especially when the BPN-SAS3-846EL1 is only $270.

This gives you the advantages of fewer HBAs, reduced cabling (2x mini-SAS HD cables is all you need), and SAS3. And, you will still get full speed if you are using spinning disks. As long as (SSD * 6) + (HDD * 2.5) is less than about 80, you'll get full speed on every drive.

 

[Koop]

Especially when the BPN-SAS3-846EL1 is only $270.

If you are really trying to cut on cable clutter, you should have been looking for the BPN-SAS-846A

I got my BPN-SAS2-846EL1 for $40 so that is a big part of my decision making here as well. I've decided a SAS3 backplane can be a future upgrade if/when I feel I need it. I went out of my way to get a new HBA, the AOC-S3008L-L8e ($30) with 2x SFF-8643 to SFF-8087 cables to connect to the BPN-SAS2-846EL1. If I'm not mistaken, I believe I should be good to use this same HBA if I were to one day pick up the BPN-SAS3-846EL1 down the road. (I've double checked this so many times but feel free to correct me if I've made a blunder here)

Also one caveat about the BPN-SAS3-846EL1- apparently it may not be a 1:1 fit inside my chassis? I know that sounds silly based on the name but I believe I was reading some threads elsewhere on this being an issue for someone. Maybe they were trying to fit it into a different chassis though and my memory and vision is getting blurry from tracking so many things, haha.

For that same ~$30, you can get E5-2690v4.

Appreciate the input/feedback here. It's definitely something I should consider.

FWIW if the use case changes, OP gets a bigger HBA, check gets a whole 'nother server - they'll still have that 846 TQ backplane and can always swap it back in.


My opinion (which is worth $0USD): OP made a wise call starting with the SAS2-EL1. To be blunt OP is doing their home work, articulating the pros and cons, considering their use cases, and adapting their decisions to match what they are learning as theiruse case transitions a bit.

Yep I am completely on board here. I don't plan to discard or resell anything- will keep for potentially future builds or just for overall flexibility if I change my mind or as I experiment- as you see! I would love to get a whole second chassis and build out a replication target one day for example. Everything initially given to me was a gift and I plan to keep it as such.

Also I'd like to say your opinion is at least worth $1 ("I'd buy THAT for a dollar!") and I've appreciated all the feedback you've provided. Thanks again.


And with that said of course thanks again to everyone who has provided input. I've learned a ton since I initially started this thread and I continue to do so.

Edit: btw itronin I've not had much luck with finding a top panel from a partner or SuperMicro directly sadly. I see it on ebay but seems a bit expensive. I'll keep up the effort to find something though. The blank for the DVD location eludes me, nobody has a clue it seems lol.

 

[nabsltd]

I got my BPN-SAS2-846EL1 for $40 so that is a big part of my decision making here as well.

I had not seen them that cheap. It's definitely far better than having to run wires to each drive, or even a cable for every 4 drives.

I have a pair of 846 chassis and both have the BPN-SAS2-846EL1 with SAS3 HBAs, so yes, your HBA will be fine if you upgrade the backplane. I only run about 12 older spinners in each chassis, so I'm not coming close to any limit of the backplane or cables.

You've done the right thing getting an "enterprise" chassis. It will last longer than you can imagine. I have a total of 6 SuperMicro chassis that I bought brand new about 8 years ago, all with redundant power supplies, and I have had 1 power supply fail in that time...nothing else has broken.

 

[Koop]

I had not seen them that cheap. It's definitely far better than having to run wires to each drive, or even a cable for every 4 drives.

I have a pair of 846 chassis and both have the BPN-SAS2-846EL1 with SAS3 HBAs, so yes, your HBA will be fine if you upgrade the backplane. I only run about 12 older spinners in each chassis, so I'm not coming close to any limit of the backplane or cables.

Yeah it was a good deal for the backplane. My only worry is that there is something wrong with it which is why it was so cheap- but at least from physical inspection nothing seems off. I should be getting my cables to connect it today so I can swap out my current backplane and test it out with my original hardware and new HBA.

Thanks for confirmation on the HBA and my thought process there. I'll personally have 24 drives (actually 26, I got two extras in case of failures) which I think may actually be a lot. If I end up not utilizing all the space (I feel like I'll find a way lol) I would love to do exactly what you're doing and run less drives across multiple chassis. I wish I could pick up another SuperMicro 4U chassis at cheaper price- the shipping costs online are killer. Buying brand new would be cool but expensive. I'm not in a huge rush obviously as you an see by the slow pace I'm working on this current build so we'll see what I can figure out. One thing I did get was a list of local resellers in my areas but I suppose I need to cold call them and find out what they can offer. I am sure they are only used to dealing with small to enterprise businesses though so getting a call from me might be odd, haha.

I figure I'll be keeping this chassis for as long as humanly possible since it's so versatile. There's a lot of flexibility in a chassis like this so I'm extremely lucky I was able to get one for free. My only major nagging issue is not being able to close the top panel all the way. I took some time and mnanpower and was able to bend the metal on the top panel enough to at least get it mostly close though so there's been progress there. I'll need to test it out to ensure no air is getting out of the top though. What is puzzling to me is that when I try to slide the panel on it's not the corner witth the issues that has trouble closing all the way but instead the opposite side. Here's some photos of how far I got it to being as close as possible to the original shape and how it fits when trying to slide it onto the chassis:





You can see it'll start closing on the top side there (which has the messed up back corner) before it can't fully close on the bottom side which has the undamaged back corner. I feel like I'm missing something here but yea, no matter what I can't get it to close all the way. At least it was better than before where I couldn't get it to stay on at all. It seems closed enough to not let air through though but I'll have to test it out. It's more of a nitpick at this point I feel haha.

You've done the right thing getting an "enterprise" chassis. It will last longer than you can imagine. I have a total of 6 SuperMicro chassis that I bought brand new about 8 years ago, all with redundant power supplies, and I have had 1 power supply fail in that time...nothing else has broken.

Absolutely. There is a night and day difference between enterprise and anything below it. Also wow that is a good number of chassis you picked up at brand new, I'm envious for sure! The versatility is the best part and there's plenty of novel solutions out there such as 3D printing mounts and mods- though I would prefer to keep things as stock as possible. Main things I've entertained so far was trying to get some front fans mounted to push air over my drives as opposed to pulling air using the stock fans due to noise. There's also 3D prints to replace the stock internal fans which I'm interested in- but I have no access to a 3D printer- need to do more research into how I can print things test some of those solutions out. Also novel ways to also mount more internal SSD drives which would be very cool to try out but the dang DVD drive is in the way hahaha. Also not sure how I'd find the best way to power SSDs using only my single available molex connection. All things to research and suss out really. I post it here openly in case any wants to chime in with ideas though, hah. Obviously an even easier route is to just use PCIe and NVMe.

Just as a side note, I did enterprise NAS installs for quite some time... Certified technical consultant for EMC back in the day. It's been a while now. I worked a lot with Isilon Systems which as you may know is eerily similar to all this SuperMicro hardware (hint: they used SuperMicro hardware lol). I believe much different chassis decisions down the road but I know for a long time they used SM boards at a minimum. So that's why I'm so fixated on sticking with the enterprise gear and SuperMicro in general- it's what I touched in data centers every single day.

Also totally spot on with parts and replacements. So far MOST parts like PSUs, fans, etc have been easy enough to find- and well some others can you see much harder haha.

Also general update that I believe I may have mentioned but:
I got a brand new unopened 42U open StarTech rack for a deal via FB Marketplace. Company surplus. Still haven't set it up yet but thankfully since it came packaged up it was easy (but heavy) to carry to my final planned racking location which is a 3rd floor. Don't ask me how I plan to ever move. I don't want to think about that hahaha.

UPS Pickups
I also ordered for pickup 2x battery refurbished Tripp Lite SMART1500RM2U UPS systems. I know they are an older but with brand new batteries local pickup they seemed like a great price. I asked the seller to please take pictures of the units and show proof that everything is functioning and the new batteries look good before I go out of my way to pick them up. They should have USB connectivity and I can dig deeping into the whole NUT server thing for graceful shutdowns which I obviously think is important. Finding a "right" UPS has been an adventure on it's own and required a ton of research that I'm still not 100% on- so many different types but from a standpoint of affordability I think I've hit a sweet spot hopefully. I'm hoping with 2x of them rated at 1350w that should be able to hopefully cover all my needs for my 846 and future equipment. I didn't test full wattage use with all my drives yet (don't have them all yet, backplane and HBA swap pending, etc) so I know I'm winging it a bit but 1350w seemed like the best I could do while still only using 5-15P plugs- I don't have any other available currently to me in my current space. Looking to go beyond that seemed like it would start to require F-20R at higher ampere or going into the twist locking connections- which I am familiar with but hell if I have ways to connect anything like that.

Anyone feel free to chime in on the UPS choice here and if I missed anything crucial. I plan to pick them up on Sunday so I have time to perhaps back out if I royally messed up!

I suppose my topic here has evolved into a bit of a whole home lab build procress. If suggested I could start a new topic in a more appropaite place on the forums to garner more feedback.

 

[nexox]

Those UPSes look fine, if a bit small for much runtime in an extended outage. I may or may not have just installed 5-15P plugs on a couple of mine to utilize larger models, it's fiiiiine (because I'm not loading them heavily and they don't draw that much while recharging.)

I personally think that case cover is closed enough to just leave it as is, they tend to be annoying to fit even when they're not bent, just a lot of sheet metal with tight tolerances, if you get them at a bit of an angle they jam up

 

[Koop]

Those UPSes look fine, if a bit small for much runtime in an extended outage. I may or may not have just installed 5-15P plugs on a couple of mine to utilize larger models, it's fiiiiine (because I'm not loading them heavily and they don't draw that much while recharging.)

Haha I didn't want to start a house fire and play with higher amps. I'm not an electrician by any means. Though I am pretty sure most homes are wired up for 20amp to begin with, but ehh.

And yeah my situation is obviously to initiate immediate safe shutdown within my runtime window and not to just keep things running. Mostly from a NAS perspective it would be extremely dumb to have so many spinning disks not shut down gracefully with potentially data mid flight. That just feels super icky and I can't ignore such an obvious best practice from my previous enterprise experience haha. Career wise I've only worked with enterprise systems that had integrated battery backups and safe shutdown baked into the systems themselves so I am unfamiliar with how an external UPS handles it. I know I should be able tot connect these units via USB and set up a NUT server though so seems equivilent by doing that. Not sure if I can also figure a way to also have power up once the UPS detects power and enough battery again though- that would be a huge bonus.

I can't even remember the last time I lost power but I don't want to murphy law myself and have the one time it does happen cause a mess.

 

[nexox]

Just keep up with UPS battery replacement, eventually they will be the cause of your downtime, not the cure. Every time a tree hits a power line around here I end up buying more UPSes, but now I have like 40 or 60 (too many, either way) smallish 12V lead acid batteries that will only last me 3-5 years, then I need to figure out something less terrible.

 

[Koop]

Just keep up with UPS battery replacement, eventually they will be the cause of your downtime, not the cure. Every time a tree hits a power line around here I end up buying more UPSes, but now I have like 40 or 60 (too many, either way) smallish 12V lead acid batteries that will only last me 3-5 years, then I need to figure out something less terrible.

Yeah totally fair. If the NUT server works as I understand it then it should be a great way to monitor battery replacement needs.

 

[Koop]

Also I'd like to bring up another topic of discussion to get feedback on if everyone is willing to chime in

I may actually be getting a X11SPI-TF kit with a CPU and memory. The one I previously mentioned- I was able to work out an even better deal with the seller. The CPU included in his kit is a Platinum 8153 and memory is SK Hynix 4x32GB PC4-2666 (HMA84GR7MFR4N).

I began doing research into LGA3647 and the extremely long list of Xeon scalable processors and ooooh boy is that a whole new can of worms. Would love to hear suggestions and recommendations and any insight if I end up with this X11SPI-TF board (I never consider it mine until it's in my hands and functional, hah but it's hopefully on the way to me soon).

Would love to overall discuss ideas and options there! From a CPU perspective things seem to be either extremely cheap or extremely expensive haha. The memory included seems fine enough and would allow me to expand further with the same sticks easy enough.

Also if anyone can give me the TLDR/101 class on Xeon scalable Gen 1 and 2 I would really appreciate it. Googling only ended me up on a lot of cruddy marketing.

 

[mrpasc]

Also if anyone can give me the TLDR/101 class on Xeon scalable Gen 1 and 2 I would really appreciate it. Googling only ended me up on a lot of cruddy marketing.

Good starting point for Gen1:

https://www.servethehome.com/intel-xeon-scalable-processor-family-skus-value-analysis/

 

[nexox]

Second gen scalable CPUs are going to be expensive, aside from a couple QS models that are getting kind of hard to find (not that I look often, maybe I missed some.) First gen CPUs are getting retired, so they're priced more by commonality than performance, generally mid-power CPUs and higher core/lower clock parts are going to be the least expensive, because that's what a lot of people bought however long ago. My favorite way to browse the part options is to go to Ark and sort by TDP, then look at cores and clocks to see what seems good, then jump on ebay to see if they're worth the price. You'll see that there are basically a few different power levels and then within each there's a trade between cores and clocks.

I find the 4114 is pretty good for lower loads, the 6132 is nice if you want more performance, maybe a 6138 if you want more cores at lower power with the trade-off of a lower base clock. For the most part the suffix letters don't mean much important, but avoid the F models, they're for network hardware and I believe they're incompatible with regular boards.

 

[Koop]

Good starting point for Gen1:

https://www.servethehome.com/intel-xeon-scalable-processor-family-skus-value-analysis/

Good info here. Will dig in soon, thank you.

Second gen scalable CPUs are going to be expensive

Yes I noticed this right away haha. Is there any feature differences between gen 1 and gen 2 that would be applicable in my case? I couldn't find anything that seemed like it made much sense other than a lot of marketing jargon. Just curious to understand and learn.

First gen CPUs are getting retired, so they're priced more by commonality than performance, generally mid-power CPUs and higher core/lower clock parts are going to be the least expensive, because that's what a lot of people bought however long ago. My favorite way to browse the part options is to go to Ark and sort by TDP, then look at cores and clocks to see what seems good, then jump on ebay to see if they're worth the price. You'll see that there are basically a few different power levels and then within each there's a trade between cores and clocks.

I find the 4114 is pretty good for lower loads, the 6132 is nice if you want more performance, maybe a 6138 if you want more cores at lower power with the trade-off of a lower base clock. For the most part the suffix letters don't mean much important, but avoid the F models, they're for network hardware and I believe they're incompatible with regular boards.

Awesome I'm glad I was on the same path. I was looking at Ark and sorting and searching. I noticed while searching on ebay I'd come up with a lot of sellers from china with pre-production CPUs. I assume it's best to avoid those.

 

[nexox]

Is there any feature differences between gen 1 and gen 2 that would be applicable in my case?

The kinda interesting difference is DCPMM (Optane Dimm) support on Cascade Lake, but for whatever reason the X11SPI doesn't support those so I guess that's not very applicable.

I noticed while searching on ebay I'd come up with a lot of sellers from china with pre-production CPUs. I assume it's best to avoid those.

There's a megathread about Xeon ES/QS processors on here, some of them (QS) are basically the same as production versions, but most of what I have found lately on ebay is the earlier stuff that's not compatible with most boards (yours included) and which may have some legit bugs. Even if you find one that should work according to the megathread, there's no guarantee that's what actually arrives, and of course returns from overseas are not always easy.

 

[UnbentTulip]

Thought on your top cover. Have you considered taking it to an auto body shop? I have some sheet metal hammers/dollies and I'm pretty sure it'd be able to be straightened out with "old school" auto body techniques. It doesn't look too bad. But they're definitely too thick to bend easily with pliers.

 

[nabsltd]

Just as a side note, I did enterprise NAS installs for quite some time... Certified technical consultant for EMC back in the day. It's been a while now. I worked a lot with Isilon Systems which as you may know is eerily similar to all this SuperMicro hardware (hint: they used SuperMicro hardware lol).

I debugged speed issues on an Isilon where I used to work. Single node speed wasn't very good, so we had to force mounting the same file system on different nodes. The application was processing hundreds of TB of data, and reading and writing on the same node wasn't as good as reading from one and writing to another.

The fact that they used SuperMicro hardware is what led me to build a 6-node gluster storage system for that job. We needed cheap temp storage (working area for data, where everything could be recreated in case of total failure of hardware) of about 500TB. Isilon was good at protecting data, but too expensive for data that literally didn't even need backup. It cost about $90K to get 576TB of useable storage across the 6 nodes, with SC846 as the chassis. I had 20x 6TB drives in each node, plus 2x 1TB SSDs for cache/tiering.

I also ordered for pickup 2x battery refurbished Tripp Lite SMART1500RM2U UPS systems.

I have 4x CyberPower 1500VA 2U rack mount UPS to feed the servers. I have two dedicated 20A circuits, and 2x of the UPS are plugged into one, with 2x into the other, and the servers have their power supplies plugged into separate UPS systems so that they get fed from different circuits. This is more than enough for the whole rack, as it only needs to run for 30 seconds until the generator kicks in.

Even so, my whole rack only uses about 800W total, so you'll have plenty of time to let a shutdown script take care of it. Look at the "half load runtime" for the UPS, and that should be the bottom limit to your real-world time.

Don't worry too much about a non-graceful shutdown. I've had multiple times when everything just died because UPS batteries weren't really able to keep up when the power went out. I now am more proactive about testing the UPS for real (the test button doesn't really tell you anything) and replacing the batteries. But, I've never lost any data because of this. All the writes go through a RAID controller with battery, and all the SSDs have power loss protection. Once I bring the system up again, the data on the disk is the data that the app was able to send before the power went out. Sure, this may be inconsistent as far as the app is concerned, because it couldn't finish the entire write, but the filesystem is fine. Other than something like a true transactional write system (where you have to close the transaction to have the metadata reflect the newly added data), you can't really know what problems an individual app might see in the data.

 

[nexox]

Indeed, unexpected power loss isn't very bad on Linux with HDDs or PLP-equipped SSDs, I went decades without a UPS at home, and at work we toggle the PDU ports for machines constantly during tests, sometimes an entire drive dies but I've never had any filesystem corruption at home, and I don't recall any at work.

Cheap SSDs, however, lose plenty of data and corruption should be expected.

 

[Koop]

Isilon was good at protecting data, but too expensive

And don't even get me started on licensing cost haha. I know when they used SM hardware that was way back in the day- I think they were a solo company. Not sure what they used down the road when they were acquired by EMC. Later on being owned by Dell everything is on their hardware now. I once solo deployed a mutli-petabyte cluster for a customer- probably 8 years ago. They had production in NYC and DR in Seattle. Flew us across the country to make a 1:1 DR site setup. Yeah from your example no reason to have an expensive isilon gear.

Isn't that sort of what iXsystems is trying to do now with TrueNAS Scale? Scale-out NAS? As soon as I read that I can only think of isilon haha. Isilon (Now Powerscale because all dell prodcuts have to start with 'power' I guess? heh) is still BSD based though as far as I know. I wonder if Dell will pivot the same way. Been a while since I've read up on how Powerscale is progressing as a product offer. Pretty sure there's a virtual version you can deploy on your own hardware... As long as you pay for plenty of licenses.

And yeah I'm not too crazy worried about real data loss but I do like the idea of actually being able to shut down equipment properly in the face of a power event just on the basis that it's the right thing to do! Hah.

 

[Koop]

I find the 4114 is pretty good for lower loads, the 6132 is nice if you want more performance, maybe a 6138 if you want more cores at lower power with the trade-off of a lower base clock.

I think what I'll do is sit on the 6153 that's going to come with my board anyway and judge from there.

I think more importantly now I should consider what I want my layout to look like. I have 24 10TB drives total that I need to divvy up. At first I was thinking I'd just divide all my disks into single pool...Maybe like three 8 wide RAIDZ2 or four 6 wide RAIDZ2. I guess two 12 wide RAIDZ2 is an option as well for raw capacity but that seems like a lot per vdev from my limited knowledge and research.

 

[Koop]

Been doing reading on those intel optane drives- they seem really cool to use specifically for random IO which would make sense for a VM pool, no?

A bit about SSD perfomance and Optane SSDs, when you're planning your next SSD....

NOTE: I'll be referring in this page, to a type of SSD developed by Intel and Micron, called 3D X-Point (pronounced "crosspoint"). It's most widely sold as Intel's Optane. The Optane devices I mean are things like the 900p, 905p, and P48xx...

www.truenas.com

I'm a bit confused by all the different models though truthfully and prices seem all over the place. Is there any guide or info on all the options and sizes?

Everything here?

Intel® Optane™ DC SSD Series Product Specifications

Intel® Optane™ DC SSD Series product listing with links to detailed product features and specifications.

ark.intel.com