Maximum # harddisk a motherboard can handle with 2 SAS expanders

denywinarto

Member
Aug 11, 2016
193
22
18
36
I'm new to this JBOD thing so bear with me..
Planning to build gigantic Media server running Ubuntu.
I just bought 1 HP SAS expander and an IBM M1015 SAS Card which has 2 SFF8087 ports, meaning it can hold up to 2 SAS expanders...
I've measured the mobo and it looks like 2 SAS expander could fit in it.. and because of certain scripts i want to run, i'm limiting my library to 1 computer.

Here's the specs :

Motherboad : Asrock Z87
Memory : 16GB 1600MhZ DDR3 G-skill
Processor : Intel® Core™ i3-4130 Processor (3M Cache, 3.40 GHz)
PSU: Plan to use 2-3 PSUs connected with Add2Psu..


So technically that's 64 HDD from SAS expander + 10 from motherboard internal SATA ports.

Questions:

1. Is there any known limitation from motherboard or other factors that i'm not aware of? Is there anything that could prevent 74 HDD from working properly with this setup? (It will be filled with 8TB and 6TB of HDDs btw)..
2. Is it Ok to run SATA from SAS expander and from internal SATA port together?
3. Is non-ECC memory viable enough?

I don't mind a little performace drop as long as i can stream 1080p content
I'm asking this because i want to build acrylic custom casing, and of course i need to know the max capacity to detemine the size of the casing

thanks
 

i386

Well-Known Member
Mar 18, 2016
1,936
504
113
31
Germany

i386

Well-Known Member
Mar 18, 2016
1,936
504
113
31
Germany
Thanks for pointing this out! :)
I guess I'll stick to 42 drives then (+10 mobo slots).. too bad cause it has 2 slots
I forgot to ask what os you want to use.
If you want to go the zfs route you could flash the controller with lsi it/ir firmware, it should then work as an hba and support more drives.
 

Sleyk

Well-Known Member
Mar 25, 2016
1,007
393
83
Stamford, CT
SAS expanders allow for a butt load of connected drives. You would pretty much run out of room to fit drives before you approach the limit of what a hba card can handle/connect to. (At least in most single chassis setups) I think the number runs in the hundreds I believe when I last read it awhile ago.

Bandwidth, on the other hand will take a hit as you keep adding drives as that, as you know, has a smaller, fixed limit based on the pcie bus/slots.

Not too sure, but doesn't Ubuntu have zfs capability? Don't quote me! Heheh.

Should be fine my peep or peepette!
 

denywinarto

Member
Aug 11, 2016
193
22
18
36
I forgot to ask what os you want to use.
If you want to go the zfs route you could flash the controller with lsi it/ir firmware, it should then work as an hba and support more drives.
I'm using ubuntu 16.04 with ext4 filesystem.. already have over 40 TB of data in 10 HDDs..
Are there any limitations for ext4?

SAS expanders allow for a butt load of connected drives. You would pretty much run out of room to fit drives before you approach the limit of what a hba card can handle/connect to. (At least in most single chassis setups) I think the number runs in the hundreds I believe when I last read it awhile ago.

Bandwidth, on the other hand will take a hit as you keep adding drives as that, as you know, has a smaller, fixed limit based on the pcie bus/slots.

Not too sure, but doesn't Ubuntu have zfs capability? Don't quote me! Heheh.

Should be fine my peep or peepette!
Any links for the thread? perhaps i could PM him..
Ubuntu does have zfs.. problem is i already have over 40TB of data in ext4
 

whitey

Moderator
Jun 30, 2014
2,770
865
113
38
Pretty sure LSI 2008 HBA ctrls in IT mode (which a M1015 is in the family of) support 256 devices max.
 
  • Like
Reactions: denywinarto

denywinarto

Member
Aug 11, 2016
193
22
18
36
Pretty sure LSI 2008 HBA ctrls in IT mode (which a M1015 is in the family of) support 256 devices max.
I see.. i bought it from ebay and i think its chinese made.. do i need to flash it to get past 32 drives limit?
I tested it yesterday with hp sas expander and 1 sata drive and it works...

But then again 256 kinda hard to achieve.. i havent seen any expanders that is capable of 16 sff 8087 slots to achieve 64 hdd per expanders...

Got another question about psu..
Is it safe to use 4 splitter on 6x6 sata slots like this to achieve to 24 hdd per psu ?
PRIME 850 W Titanium | Seasonic PRIME

I will then pair the PSu's with add2psu
 

denywinarto

Member
Aug 11, 2016
193
22
18
36
If you want to make it simple, you can use a SAS storage array such as a HP MDS600 : HP MDS600 Storage Array (Configure-To-Order)
It directly provides PSUs, cooling and you have hotswap trays and LEDs on the front panel for every disks.

We bought one of these with 70*2Tb, plugged to a P812, it has been working fine for the last 2 or 3 months.
Now thats interesting !
Unfortunately i dont live in US so the import cost is gonna be quite alot, since it's 160 lbs.. probably could double the original cost... i'll have to figure out a way to get this to a sea freight maybe..
A question..
Can we use our own motherboard + sas card + sas expander on this case?
 

KioskAdmin

Active Member
Jan 20, 2015
156
32
28
49
I'd get a sas expander chassis with everything built-in. You're building something with many failure points. SAS cards don't always play well with consumer mobos.
 

TuxDude

Well-Known Member
Sep 17, 2011
615
336
63
The limit someone posted earlier of 256 devices for the M1015 HBA sounds accurate, assuming the card is flashed to IT-mode. In IR-mode (raid mode), that limit will be much much lower (32 is probably right). I've only taken my M1015 as high as 28 drives, though I also have a LSI-9201-16e here (same generation of SAS controller) with 34 drives connected, which at least pushes over that 32-drive limit.

As for physically connecting them all, you can daisy-chain expanders (that is what that HP MDS600 linked above does), which should easily get you to as many drives as you could possibly want. As far as I know the biggest expander chips out are the LSI 36-port ones, so anything with more than around 30 drives (you need ports to uplink to HBAs, and more ports to link between expanders if there are multiple) is going to be based on multiple expanders..
 

denywinarto

Member
Aug 11, 2016
193
22
18
36
The limit someone posted earlier of 256 devices for the M1015 HBA sounds accurate, assuming the card is flashed to IT-mode. In IR-mode (raid mode), that limit will be much much lower (32 is probably right). I've only taken my M1015 as high as 28 drives, though I also have a LSI-9201-16e here (same generation of SAS controller) with 34 drives connected, which at least pushes over that 32-drive limit.

As for physically connecting them all, you can daisy-chain expanders (that is what that HP MDS600 linked above does), which should easily get you to as many drives as you could possibly want. As far as I know the biggest expander chips out are the LSI 36-port ones, so anything with more than around 30 drives (you need ports to uplink to HBAs, and more ports to link between expanders if there are multiple) is going to be based on multiple expanders..
I'd get a sas expander chassis with everything built-in. You're building something with many failure points. SAS cards don't always play well with consumer mobos.
Thanks for the inputs.. are there any alternatives for MDS600? At least a lighter alternative?
Importing that chassis will triple the cost (bc of the weight) from my calculations :(

Hmm also after some googling i learnt that this thing isn't as flexible as norco in terms of spareparts.. seems like norco is more universal... meaning i can use mobo sas expander etc from another brand.. But the max norco chassis is 24 HDD
Is there any other universal storage rack like this?
 
Last edited:

TuxDude

Well-Known Member
Sep 17, 2011
615
336
63
Thanks for the inputs.. are there any alternatives for MDS600? At least a lighter alternative?
Importing that chassis will triple the cost (bc of the weight) from my calculations :(
There are lots of high density disk expansion shelves on the market, I think pretty much every brand has their own version. And they're probably all fairly heavy - weight is not one of the factors I consider when looking at gear so I don't have any advice there.

Hmm also after some googling i learnt that this thing isn't as flexible as norco in terms of spareparts.. seems like norco is more universal... meaning i can use mobo sas expander etc from another brand.. But the max norco chassis is 24 HDD
Is there any other universal storage rack like this?
That is the trade-off with having the SAS expander chip be integrated into the drive backplane. You lose flexibility in part selection - you have to use the backplane that fits your chassis, but you have fewer parts (don't need to fit an expander card elsewhere, potentially using expansion slots for power, etc.), and get better integration (eg. being able to tell the backplane to turn on the locate LED on the bay associated with port 5). My preference is to have the expander be part of the backplane.

As for a "universal" storage chassis - with a few little parts you can turn any chassis into an add-on disk chassis. All you really need is a way to turn the PSU(s) on to power drives/expander(s), and a SFF8087/8088 bracket to be able to connect an external SAS cable from your compute node to the expander inside the storage shelf. It could be as simple as an old desktop case with a paperclip to short green to black in the ATX connector to force the PSU to always-on, or as fancy as a 4U supermicro case with up to 90 hotswap drive bays, and a CSE-PTJBOD-CB3 board in it to control power, fans, remote-management, etc.
 
  • Like
Reactions: denywinarto

denywinarto

Member
Aug 11, 2016
193
22
18
36
There are lots of high density disk expansion shelves on the market, I think pretty much every brand has their own version. And they're probably all fairly heavy - weight is not one of the factors I consider when looking at gear so I don't have any advice there.



That is the trade-off with having the SAS expander chip be integrated into the drive backplane. You lose flexibility in part selection - you have to use the backplane that fits your chassis, but you have fewer parts (don't need to fit an expander card elsewhere, potentially using expansion slots for power, etc.), and get better integration (eg. being able to tell the backplane to turn on the locate LED on the bay associated with port 5). My preference is to have the expander be part of the backplane.

As for a "universal" storage chassis - with a few little parts you can turn any chassis into an add-on disk chassis. All you really need is a way to turn the PSU(s) on to power drives/expander(s), and a SFF8087/8088 bracket to be able to connect an external SAS cable from your compute node to the expander inside the storage shelf. It could be as simple as an old desktop case with a paperclip to short green to black in the ATX connector to force the PSU to always-on, or as fancy as a 4U supermicro case with up to 90 hotswap drive bays, and a CSE-PTJBOD-CB3 board in it to control power, fans, remote-management, etc.
I'm about to buy a used MDS600 with 2x I/O SAS module 455972-001/AP763A...
My main concern is the PSUs.. they age faster than other components so i'll have to replace them sooner..

So is there absolutely no way to use regular PSU on MDS600?

I could 've made a custom casing with acrylic but i kinda worried that the vibration will kill the HDDs
 

TuxDude

Well-Known Member
Sep 17, 2011
615
336
63
I'm about to buy a used MDS600 with 2x I/O SAS module 455972-001/AP763A...
My main concern is the PSUs.. they age faster than other components so i'll have to replace them sooner..

So is there absolutely no way to use regular PSU on MDS600?
Nope - only the ones specifically built for it.