High Capacity, High Speed, High Reliabilty SAS DAS

[momoceio]

Hello all, I'm new to these forums but have been reading some of the topics on here to get some ideas and pointers. I'm hoping someone here can help me out with configuring a DAS unit for backing up or virtual machines using Veeam Backup and Replication. It needs to be capable of fairly high IO (nothing crazy) and high capacity (40+ TB).

Here is our current setup:
Dell PowerEdge 2950
40GB RAM
Xeon E5420 Quad Core
PERC H800 Controller
Dell MD1200 SAS DAS (12 2TB 2.5" in RAID50)


The Dell MD1200 is partitioned into a single 14.5 TB volume where backups are written to. We're most likely going to repurpose the MD1200 as a local cache unit for a cloud backup solution.

Here is what I'm thinking of doing:
2x SuperMicro CSE-847E16-RJBOD1
90x Seagate ST1000NM0001 1TB SAS2 Drives (7200 RPM)
Use the SAS expanders in the SuperMicro cases to daisy chain the two units together into the PERC H800


I wasn't sure about the PERC H800 and 3rd party storage units so I'd also be up for purchasing a new controller to drive this beast. My goals are high capacity, fairly high performance and high reliabilty. I don't want to be messing around with this unit all the time and worrying about it flaking out on me.

Would I be better off abandoning the Dell server and making one of these units into a full blown server and then connecting the secondary unit into it?

I feel like I'm leaving out some needed parts or missing something so please help me out.

If it makes it any easier, assume you have 15-25k to spend. Is this the setup you would go with to meet the requirements?

Thanks!

 

[dba]

That's an impressive set of hard drives! I'm glad you have the budget for dual-ported drives - that'll help with reliability assuming that you select the backplanes with dual expanders.

For multipathing, you need at least two SAS HBAs or battery-backed SAS RAID cards to access those drives - one card for each SAS chain. Actually, ninety drives can drive far more IO than any two cards can handle in sequential read or write scenarios, so four cards would be better, but then it's unlikely that you'd be able to make use of more than two cards worth of IO so it's probably not necessary to further increase your disk IO capabilities.

Now the bad news. Your 2950 is not up to the task of driving those disks to anything near their maximum performance, much less serving that data up to clients. You have only three PCIe slots, and two of them are x4 speed only. I'd want at least four PCIe x8 (or PCIe 3 x4) slots - two for the HBA or RAID cards and two for 10Gbe cards.
Also, I worry about vibration in a standard chassis with 45 drives, even when using Enterprise-class drives (like yours) that are designed to be used in RAID arrays. With that many drives you'd almost certainly see a significant - and possibly very large - drop in reliability and performance, unless the chassis were designed with a great deal of isolation around each drive, isolation that seems to be very rare.

So for my $28K I'd pick up my $20K in drives and then use the other $8K to buy:

1) A new server with more PCIe slots and one or two lower-end Xeon E5 CPUs because of their fantastic PCIe3 IO.
2) Two nice LSI RAID cards - or simple LSI 9207-8e HBAs if I were using ZFS - and I would.
3) I'd then spend a bit of time researching drive chassis with specific vibration-reduction features. I'd want to run some performance tests before making an irreversible commitment to any given chassis.

It's a nice project you have. Enjoy!

 

[mobilenvidia]

90x 1TB drives, would 45x2TB or 30x 3TB drives be more economic, and lessen the risk of drive failure.

30x 3TB SAS2 drives (Seagate Constellation ES.2 ST33000650SS 3TB 7200 RPM 64MB Cache SAS 6Gb/s)
2x LSI 9202-16e HBA
or
45x 2TB SAS2 drives
3xLSI 9202-16e HBA

On, as dba said a nice low end dual E5 CPU Mobo, running ZFS

 

[dba]

You make a good point. VM backups should be a mostly sequential write workload, so you don't need that many IOPS. 45 2TB disks would save money on the disks and the chassis required or leave room for expansion - and you'd still have at least 4,500 IOPS and enough throughput to saturate three or four 10GbE links.

90x 1TB drives, would 45x2TB or 30x 3TB drives be more economic, and lessen the risk of drive failure.

30x 3TB SAS2 drives (Seagate Constellation ES.2 ST33000650SS 3TB 7200 RPM 64MB Cache SAS 6Gb/s)
2x LSI 9202-16e HBA
or
45x 2TB SAS2 drives
3xLSI 9202-16e HBA

On, as dba said a nice low end dual E5 CPU Mobo, running ZFS

 

[mobilenvidia]

If you need IOPS then a LSI 9286CV-8eCC, with Fastpath and CacheCade you'd need as many as 4x expanders to get 90 Drives and a few SSD's for caching.
Of course no ZFS as you'd be running HW RAID.

 

[momoceio]

Wow, thank you guys for the responses! Some good information here, see my responses below.

That's an impressive set of hard drives! I'm glad you have the budget for dual-ported drives - that'll help with reliability assuming that you select the backplanes with dual expanders.

For multipathing, you need at least two SAS HBAs or battery-backed SAS RAID cards to access those drives - one card for each SAS chain. Actually, ninety drives can drive far more IO than any two cards can handle in sequential read or write scenarios, so four cards would be better, but then it's unlikely that you'd be able to make use of more than two cards worth of IO so it's probably not necessary to further increase your disk IO capabilities.

Now the bad news. Your 2950 is not up to the task of driving those disks to anything near their maximum performance, much less serving that data up to clients. You have only three PCIe slots, and two of them are x4 speed only. I'd want at least four PCIe x8 (or PCIe 3 x4) slots - two for the HBA or RAID cards and two for 10Gbe cards.
Also, I worry about vibration in a standard chassis with 45 drives, even when using Enterprise-class drives (like yours) that are designed to be used in RAID arrays. With that many drives you'd almost certainly see a significant - and possibly very large - drop in reliability and performance, unless the chassis were designed with a great deal of isolation around each drive, isolation that seems to be very rare.

So for my $28K I'd pick up my $20K in drives and then use the other $8K to buy:

1) A new server with more PCIe slots and one or two lower-end Xeon E5 CPUs because of their fantastic PCIe3 IO.
2) Two nice LSI RAID cards - or simple LSI 9207-8e HBAs if I were using ZFS - and I would.
3) I'd then spend a bit of time researching drive chassis with specific vibration-reduction features. I'd want to run some performance tests before making an irreversible commitment to any given chassis.

It's a nice project you have. Enjoy!

After reading this, I realized that I left out some pertinent information.

1) This server will not be accessed by any client PCs. This is a dual homed server that resides on our main network and iSCSI network which the Veeam backup software utilizes for direct SAN access to our equallogic unit.

2) It performs change block tracking for each VMs virtual disk to pull only the changes.

3) The backups are done using a reverse incremental which is actually fairly IO intensive. Instead of doing a weekly full and 6 days of incrementals it injects the changes directly into the existing full backup. Any changes that would be deleted by the injected changes are then pushed out to a reverse incremental file. So the full backup always contains the most recent changes. So for every IO read from the VMs VMDK, three IO's are performed on the backend (write into full, read from full, write into reverse incremental).

4) Since this is all self contained, the only network access is via iSCSI to our SAN which has 3 active 1GB links.

5) I cannot utilize ZFS as this backup server must run Windows for the backup software to run on unless I separated the two servers and turned this into an iSCSI target. I assume I'd get better performance with direct SAS access though?

I had actually thought about vibration issues but assumed the purpose built SuperMicro chassis would have been designed to take that into account. You have me more concerned about it now, though, so I'm going to contact them to see if they have done any testing. I'm now convinced to just ditch the Dell PowerEdge 2950 and turn one of the 45 drive chassis into a full blown server so I can utilize a newer motherboard/cpu/ram. With that in mind, which board/cpu combo would be recommended for this setup?

I was looking at this board but it sounds like I should stick with Intel for the PCIe 3.0 spec?

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182230

90x 1TB drives, would 45x2TB or 30x 3TB drives be more economic, and lessen the risk of drive failure.

30x 3TB SAS2 drives (Seagate Constellation ES.2 ST33000650SS 3TB 7200 RPM 64MB Cache SAS 6Gb/s)
2x LSI 9202-16e HBA
or
45x 2TB SAS2 drives
3xLSI 9202-16e HBA

On, as dba said a nice low end dual E5 CPU Mobo, running ZFS

I was thinking of going the 45x 2TB route but decided to go for 90x 1TB for IO performance and because I'll be creating 3 or 4 arrays from the 90 drives which would give me the best performance per array (more spindles per array)

If you need IOPS then a LSI 9286CV-8eCC, with Fastpath and CacheCade you'd need as many as 4x expanders to get 90 Drives and a few SSD's for caching.
Of course no ZFS as you'd be running HW RAID.

The way these backups run, even though they are IO intensive, I'm not sure they would take advantage of an SSD cache since the writes/reads are typically very large files. One of the system architects from the company actually mentioned that a hybrid array wouldn't give any real world performance increases when used with their product.


Wow, sorry for such a big post and thank you all for your help so far.

 

[dba]

You are building a monster backup server. With two LSI cards on PCIe3 Xeon, I believe that you could handle at least 3Gigabytes/Second of sequential writes - I've gotten more than 4GB/Second of application throughput using two such cards so this is a conversative estimate.
In contrast, your three 1GbE cards have a theoretical maximum of around 1/10th of that - and the real world throughput is even lower given iSCSI translation and overhead. If one of your goals is improved backup speed then a faster connection to your VMs is in order. If significantly more speed is not a goal then I'd say that you are over-engineering the new backup server and could save some money by downsizing.

With your current server, what kind of MB/Second and disk IOPS are you seeing?

One last note: The Supermicro chassis usually have rubber-isolated fans, but the drive sleds are metal on metal with no rubber or plastic vibration dampening. The sleds fit very well and don't rattle like some low-end sleds, but they will transmit any vibration generated by the drives. This is not at all an unusual practice, and I don't mind one bit for any reasonable number of disks, but it's far from optimal for the mega-sized RAID chassis you are building.

 

[mobilenvidia]

What about:

88x Seagate Constellation.2 ST91000640SS Hard Disk Drive, 1000GB
Inside a
http://www.supermicro.com/products/chassis/4U/417/SC417E16-RJBOD1.cfm

Then main system with 10Gbe to allow about 800MB/s throughput, vs about 80MB/s on 1Gbe
A single LSI 9207-8e to run the lot via the onboard expander(s)

 

[momoceio]

You are building a monster backup server. With two LSI cards on PCIe3 Xeon, I believe that you could handle at least 3Gigabytes/Second of sequential writes - I've gotten more than 4GB/Second of application throughput using two such cards so this is a conversative estimate.
In contrast, your three 1GbE cards have a theoretical maximum of around 1/10th of that - and the real world throughput is even lower given iSCSI translation and overhead. If one of your goals is improved backup speed then a faster connection to your VMs is in order. If significantly more speed is not a goal then I'd say that you are over-engineering the new backup server and could save some money by downsizing.

With your current server, what kind of MB/Second and disk IOPS are you seeing?

One last note: The Supermicro chassis usually have rubber-isolated fans, but the drive sleds are metal on metal with no rubber or plastic vibration dampening. The sleds fit very well and don't rattle like some low-end sleds, but they will transmit any vibration generated by the drives. This is not at all an unusual practice, and I don't mind one bit for any reasonable number of disks, but it's far from optimal for the mega-sized RAID chassis you are building.

Thanks for the info, see my responses below.

1) Our SAN currently only has 3x 1 GbE connections however, somewhere in the not too distant future we will be adding an additional SAN to better tier our storage and to increase available IO to our VM environment. At that time, it is possible that a 10 GbE iSCSI network would be implemented. At the very least, we would have 6x 1GbE connections into our SAN environment.

2) I'm collecting some updated data for MB/Sec and IOPS

3) If you have a moment, take a look at page 4 of this PDF to get an idea of how the backups work. During a typical backup I actually see fairly low iSCSI network utilization but high disk IO as the majority of the work ends up being performed on the backend. Typically, I only see my iSCSI links saturated when I'm performing a full backup which never happens after the initial backup. Also, considering I may be running 4 - 6 jobs at any one time you can see how the IO adds up.

4) So a 45-drive supermicro chassis is probably a bit too big? If I do end up going with around 70-90 drives would it better to buy 2x 36 drive or 3x 24 drive chassis? Do any of you have performance numbers from a similar setup with fewer drives (24, 32, 45, etc)?

Thanks for the help guys, please don't take any of my responses as me disagreeing with your points. I just want to make sure my environment is understood so I don't undersize the storage setup. I'd rather spend extra now and not have to go back and ask for more money in 6-12 months when I hit my IO limits (not saying I would). It's much easier to get approved once than twice, at least where I work. We were looking at a dell MD unit that can take 60 drives (not yet available) but the price was pushing 90k when full which is hard to stomach. I want to have an end result that that doesn't make me regret going DIY

 

[momoceio]

What about:

88x Seagate Constellation.2 ST91000640SS Hard Disk Drive, 1000GB
Inside a
http://www.supermicro.com/products/chassis/4U/417/SC417E16-RJBOD1.cfm

Then main system with 10Gbe to allow about 800MB/s throughput, vs about 80MB/s on 1Gbe
A single LSI 9207-8e to run the lot via the onboard expander(s)

We actually considered the 72x drive version of that chassis but since it only takes 2.5" drives the cost goes up significantly. The 3.5" version of the Seagate Constellation drive is $100 cheaper per disk for the same RPM and capacity. Other than density, I didn't see what else we got from going to 2.5" over the 3.5". Is there more to it than that?

 

[mobilenvidia]

2.5" drives do give you less latency as the sectors can be found quicker, not a huge difference.

$100 per disk cheaper.
But how much will extra 3.5" chassis(s) cost + cabling + extra Controller(s) ?

 

[dba]

These articles discuss the real-world effect of vibration of disk subsystem performance:

http://www.zdnet.com/blog/storage/bad-bad-bad-vibrations/896

http://static.usenix.org/events/sustainit10/tech/full_papers/turner.pdf

And then there is the geeky-famous "yelling at your disk drives" video with 850K views:
http://www.youtube.com/watch?v=tDacjrSCeq4

 

[Patrick]

I know I was one of those 850K views! Classic.

 

[mobilenvidia]

I'll never shout at a JBOD again, if only I knew this earlier

But what a racket, momoceio let it be a lesson for your, many drives = much noise.

 

[momoceio]

I emailed SuperMicro support about chassis vibration:

Hello,
I am considering using the CSE-847E26-R1K28JBOD chassis for a DAS storage unit that would house backups. I am concerned that housing so many drives in a single chassis may result in poor performance due to hard drive vibration. Does SuperMicro perform vibration tests on their storage chassis?

Hi,

Super Micro does perform vibration testing on our storage chassis to ensure the chassis has adequate damping to negate HDD and fan vibration. We only recommend using enterprise drives in our servers.

Thank you,
-PA

For what that's worth.

 

[dba]

I emailed SuperMicro support about chassis vibration:
...

I say go empirical: Get a chassis and drives into your environment, test a combination of sequential and random IO on say an eight-drive RAID6 RAID volume using IOMeter with all other drives spun down, and then test again will all 45 drives spun up but still only eight of them in your RAID volume. Bonus points if the other drives are connected to a separate server that is generating a workload on them so that their heads are moving and contributing to the overall vibration but not contributing to the load on your IOMeter test machine. If there is no performance difference, or if the difference isn't enough to be worrisome, then your particular combination of equipment and workload does not have a significant vibration problem.

 

[momoceio]

Good idea, sounds like a pretty solid testing method. I will post results after I get everything setup. I also need to get a new server to run this since my PowerEdge is a bit dated. Is there a certain "go-to" board/cpu/memory combo that is recommended? Also, I see the LSI cards mentioned a lot so I'm guessing those are the favored. Any specific card that I should use to run this? I guess I would need two cards for multipathing.

 

[Mike]

On the vibration bit i would like to note that i will never buy those 10cm blower fans again

 

[momoceio]

Here's my final list:

Storage
SuperMicro Chassis CSE-847E26-RJBOD1 x2
Seagate ST1000NM0001 x90
SuperMicro SAS Cable x4
LSI Raid Controller x2

Host
SuperMicro Chassis/MB
CPU
Memory
CPU HeatSink
HDD

Look OK? Other than it being overkill and the possibility of vibration issues which will be tested before deployment.

I did forget to mention that in the event of a restore, I can actually perform an "instant on" recovery for any VM. The VM actually boots on the host and runs on the backup storage. If I were running 4 jobs and potentially had 3 or 4 VMs running then the added IO would be beneficial. As I mentioned before, the IO isn't limited by the amount of data I can pull over the 3 1GbE links as there are a lot of backend data operations happen per VM that is backed up. With just one job running, I've had the current 12 disk array be the bottleneck while the iSCSI network was only utilized 10-20%.

 

[dba]

Looks like a great setup.

Two possible problems:

1) Each disk chassis has two completely separate expander backplanes. The four SAS cables in your parts list are enough to direct-wire the four RAID card ports to four expander ports, or enough to form a single-path daisy chain, but not enough to enable multipathing. For a standard dual-ported drive wiring scheme with two cards and four dual-ported expanders, you'll need eight cables.

2) Most SAS cables, including the Supermicro cables, use a single fat cable housing. These are very stiff, and when you rig four of them from a server to a disk chassis, it can be difficult to get a cable management arm to function properly. If this might be a problem for you, ether order extra long cables so that you can make big loops or, if space inside of your rack is an issue, try to hunt down alternative cables that use two smaller cable housings instead. I can't upload photos to this forum, but here is a public URL that shows one such cable: http://www.serialcables.com/search-largeview.asp?cat=&tier=&id=1464&search=true. I have similar cables for my LSI 9202 cards (except that they are SFF-8644 connectors like the photo instead of the SFF-8088 cables you need) and it makes a dramatic difference.
If you aren't using a cable management arm - I don't even know if Supermicro offers one - then it's probably not an issue because you'll likely need to unplug all of the cables whenever you need to roll out the chassis anyway. Just make sure that they are labeled at both ends! My HP server currently has 14 SAS cables poking out the back end and without labels I would be lost.

Here's my final list:
...