Supermicro NAS Upgrade

[fatmcgav]

So the time has finally come to increase the storage on my current ESXi/NAS AIO.

Current specs are:

• Fractal Node 804 case
• Supermicro X10SL7 MB, onboard LSI2308 flashed to IT mode.
• Intel Xeon E3-1220 v3
• 32GB RAM
• 6x 3TB Toshiba DT01ACA3 SATA HDDs - General storage in RAIDZ2
• 2x 250GB Samsung 850 SSDs - Mirrored VM Storage

I'm currently leaning towards either a 3U or 4U Supermicro chassis to give me either 16 or 24 drive bays. I can then transplant the existing hardware over and add an additional 6x 4TB HDD's, exact model TBC, to expand my general storage.

One of the SC386 chassis I'm interested in has the SAS836TQ backplane, which I believe is just a straight pass-through with 16 SATA ports.
Before I make an offer, do I need to factor in swapping the backplane for a SAS2 version? Other than making cabling a lot simpler and cleaner with my onboard LSI2308, are there any other benefits or drawbacks?
If I was to keep the TQ backplane, I'm assuming I'd just need a SAS expander card and suitable 8087 fan-out cables to connect to the backplane?

Anything I'm missing?

Thanks in advance.

Gavin

 

[itronin]

How you are going to use/grow this server, how you plan to use the two slots (x4/x8 pcie 2.0 and x8/x16 pcie 3.0 slot) in the future will be impacted by the route you go down for the case you choose.

Simplest is just to get the chassis appropriate SM Expander backplane (836/846), sas2 is good enough... any other expander and you'll have to figure out how to mount it, power it etc. etc. etc.

I'd avoid the cheap HPE expander, you'll either have to burn a slot on your motherboard or add a mining adapter to power it... Better expanders will cost more money you'll also have to go reverse sata breakout cables from your motherboard to the expander...

If you are willing to burn a slot on your motherboard just get a used/refurb 2308 card and use the x4/x8 slot with a loss of performance (which you may not even notice depending on your use case). Then wire everything direct to a TQ backplane... Fewer fiddly bits IMO. 2308 card, 2 forward SFF-8087 to SATA and 8 SATA to SATA cables (which you might already have in the appropriate length).

Personally I'm a fan of the 836 (and TQ) - its just big enough, 846 really is huge/cavernous and weighs a lot more than the 836 (IMO).
TQ gives you SAS3 down the road but trickier cabling and more limited because you may need 2 or 3 controller cards for the 846, 1 or 2 cards for the 836...


more detail and random ideas:

836 TQ backplane:

in theory... since the 2308 is wired as SATA ports on the motherboard, if you were to use an expander card you'd need a reverse SATA breakout SFF-8087 cable to go into the expander and then forward breakout SFF-8087 cables to come out and go to the TQ backplane.

Assuming you don't have the right cables and a chap expander handy (HP thing-a-mob that is typically $15.00 USD or so). figure each cable at an average of about $10.00USD - so 6 cables, cheap HP expander, mining adapter to power it - you're at about $100.00USD more or less

NB: the cheap HP expander will only clock your SATA drives at SATA 2 speed, and you will need to make sure you get one that supports the last firmware update for them ( I believe the PCB is green not yellow ). If you went with the HP expander you will either need to burn your 2.0 slot to power the HPE expander or getting a mining adapter installed in the chassis to power the expander.... If you don't do the cheap HP expander you are definitely getting lose (USD) to the cost of just getting an 836 SAS2 expander backplane.

NB: SM does make some snazzy staggered SFF-8087 SATA cables to make your cabling a bit cleaner on the TQ backplane.

836 options...

SAS2 expander backplane
Not sure what part costs are like in the UK but you may end up spending close to the cost of getting a SAS2 expander backplane and the reverse breakout cables ... IMO This is simplest configuration, doesn't burn a port, require a mining adapter (cheap HP expander).

Go Direct TQ options:
You may also find it cheaper to simply add a cheap 2008/2308 ($20-$40USD), 2 SFF-8087 breakout cables and plain old sata cables to go from the motherboard to 8 of the TQ ports. You can use that card in the x8(x4 data) 2.0 slot on the motherboard (potential loss of drive performance) keeping the x16(x8 data) free for 10Gbe or a GPU if you are looking for a server that does transcoding... Personally I think this is the winner solution on a TQ backplane and your motherboard as compared to the cheap HPE expander.

You could spend the money and get something like a 9400-16i and SFF-8643 breakout cables and then everything would be SAS3 - you'll burn your 3.0 slot though it might work (at a performance loss) in the 2.0 slot.

846 options...
Get the SM expander backplane and the reverse breakout cables... IMO just too much hassle cabling and fiddly bit stuff to do anything else.

FWIW
I'm about to "downgrade" a pair of 836 to their original TQ backplanes so I can upgrade the lot to SAS3... as well as a pair of 836 TQ to SFF-8643 breakout cables - so I will have both a SAS2 expander backplane as well as SFF-8087 forward breakout cables available - will sell things inexpensively as I got my use out of them however I am in the US and no idea what the shipping would be - nor how much paperwork is required. My down/up grades will happen in the next couple of weeks.

 

[abq]

if fatmcgav passes on sas2 backplane & cables, i am interested in upgrading my 836tq. thank you!

 

[itronin]

if fatmcgav passes on sas2 backplane & cables, i am interested in upgrading my 836tq. thank you!

feel free to PM me.

 

[fatmcgav]

Thanks for all the info @itronin. Lots of bits to think about.

Correct that the current 2308 is broken out into 8 sata ports, so I would need to combine into a sas expander.
I hadn't even considered buying another 2308 to run the other 8 bays though. That does sound like the simplest and most performant option...

I've got no specific expansion plans in mind for the future. 10gbe is probably a way off yet. So using the x16 slot for now would work.

Sounds like a plan though.

Thanks for all the advice.

Gavin

 

[i386]

About 836 vs 846:
I have a 836 and a 846 and if I get another chassis it will be a 846 again because it's "stoage density" is better (24 devices/4u vs 16 devices/3u). Another point for the 846: there are more (sas2 AND sas3) expanders available on ebay than for the 836.

Edit: "A" (sas multilane port) backplanes > tq backplanes
Look for the "A" backplanes, they make the cabling easier.

 

[fatmcgav]

One more thought I've just had is around the re-use of my existing MB.

A lot of the 846/847 chassis I'm seeing have an X9 dual-cpu board built in, and some have one or two CPU's included.

So is there any merit in swapping out the bundled MB for my mATX X10SL7-F board?
Other than my board being a generation newer, is there any pro's/con's either way?

Thanks

 

[itronin]

One more thought I've just had is around the re-use of my existing MB.

A lot of the 846/847 chassis I'm seeing have an X9 dual-cpu board built in, and some have one or two CPU's included.

So is there any merit in swapping out the bundled MB for my mATX X10SL7-F board?
Other than my board being a generation newer, is there any pro's/con's either way?

Thanks

Something to think about for any solution - Cost of Electricity - I hear in the UK its expensive but have no facts on that.

Pros to swapping in your X10SL-F
Probably less power usage by the server as a whole vs. a 2011 board with 1 or 2 CPUs.
This may also translate to less heat.

Cons to using an X10SL-F
udimms are more expensive than rdimms - if you already have the memory then this is probably a non-issue.
If this is only NAS then may not matter if you decide to spin up lots of VM's then you may find this board a limiting factor.

Pros to 847
Lots of drive trays - do you need that many drives?
Con to 847
Noise - lots of fans in there which are needed to keep the drives cool.
Fully loaded the thing will weigh a LOT (by my standards). Do you change living location often? Stairs?

Salient points:

cooling:
In the 846 you may need to add auxillary cooling to your 2308 chip - either attaching a fan to the controller heatsink or adding in a bracket based fan on the pcie brackets... You may not - depends on how well the chassis fan well cools the 2308. I don't think this will be an issue with the 847 because of the number of fans (unless you start removing fans to try and quiet the server).

Add-in cards:
846 can take full height cards
847 can take HH cards.
you have limited slots on the x10SL7 so this may be a non-issue but may not be if you think you might upgrade the guts down the road.

What I would do if it were me:

If I had the X10SL7 with cpu and memory I'd be looking at the 836/846 - see which one I can get a deal on and use it in there.

You can upgrade the guts down the road should you desire.

If you invest in the 847 that's a 2U motherboard tray which *may* limit you down the road on what you can do with the server if you want to upgrade the guts.

IMO drive costs for used enterprise drives will continue to be in this cycle of enterprise lifecycle dumps to the marketplace every so often which means you can get 3-5 year old higher capacity drives and perform upgrades every so often. Challenge is of course if you fill up your server with drives and they're filled up with data you have to have some place to put it while you swap in a new drive constellation - of course you should be backing this up anyway...

 

[fatmcgav]

Thanks again @itronin

So Yeh, I've currently got an x10sl7-f board which is fully populated with 32Gb of 1600mhz ecc mem...
I currently run a handful of vms in addition to the nas vm; however I'm looking to move a lot of that off to a 4 node rpi4 k3s cluster I've recently built... So that will free up ram that I can put back into the nas...

Current performance is fine, the only issue is that I'm running out of space...

I'm kinda swaying towards an 846 right now, as gives a bit more headroom over an 836.
And Yeh, I think transplant for now, and then can always upgrade the board down the road if I need to. And by selling the x9 gubbins, can recoup some of the outlay...

 

[i386]

In the 846 you may need to add auxillary cooling to your 2308 chip

All the different official fans in the 846 provide enough airflow and static pressure to cool pcie devices including raid controllers and hbas.

 

[fatmcgav]

So I've been and done it now...

Managed to find a decent deal on a 36 bay CSE-847, loaded with a Supermicro X9DRD-7LN4F motherboard, a pair of E5 2660v2 CPUs and 144GB of DDR3 ECC memory.

Only bit I need to do is swap out the stock PSU's for a 920W SuperQuiet version, and possibly swap out some of the fans for quieter versions...

Now to order some extra hard-drives

 

[itronin]

So I've been and done it now...

Managed to find a decent deal on a 36 bay CSE-847, loaded with a Supermicro X9DRD-7LN4F motherboard, a pair of E5 2660v2 CPUs and 144GB of DDR3 ECC memory.

Only bit I need to do is swap out the stock PSU's for a 920W SuperQuiet version, and possibly swap out some of the fans for quieter versions...

Now to order some extra hard-drives

Awesome! that's a good board, lots of 6x8 slots + the 2308 and most of the slots support bifurcation if you wanted to add nvme storage.

How many drives will you be installing? You may want to do the math on the drive's power load at power-on unless you are doing delayed spin-up. 920 could be kinda tight with 36 drives... assuming 20 watts average at spin up, = 720 watts, plus 2xcpu, motherboard, HBA etc.

If you don't fill up the chassis or are using delayed spin-up then probably no big deal but I'm kind of a believer in "if you build it you will fill it" ...

good luck on finding well priced 920-SQ's - in the US the prices have skyrocketed seems to be about 75-100+ USD on the bay. You might want to put a WTB post in the WTS/WTTWTB forum and see if someone has what you need.

 

[fatmcgav]

I'm starting out with 12 SATA HDDs plus a couple of SSDs, so hopefully a 900w psu should be enough to start with. I can always add a 2nd if start adding more drives...

Lucokly I managed to find one on ebay for £50, which was a bit of a bargain.