Anyone Raiding current SATA III SSD drives?

[KenE]

I'm always looking of simple ways to increase my workstation/server combo power. Has anyone raided these SATA III drives in a production environment? How is performance over time effected without the use of TRIM? Is the HW garbage collection good enough to keep speeds up?

I was thinking that a a pair of SSD's running RAID 0 (with a HDD running hourly backups!) as a server data drive would be pretty 'snappy'.

 

[sotech]

I tried striping a pair of 128GB Crucial M4s as an experiment to compare to the performance of the same drives in a mirror; I found that as ZFS reads from mirrors in a round-robin fashion the read speeds weren't any lesser in the mirror than the striped pair. Since the array was largely read from rather than written to I didn't end up sticking with the stripe. It is nice seeing the Bonnie sequential read benchmark at 998MB/s for a two-drive array They worked fine for the time that I had them but it was only a few days, so that's probably not super useful to you.

Have you got sufficient network bandwidth to utilise more than one SSD worth of speed at once?

 

[Patrick]

Ken,

I've been using RAID 0 SSDs in my main workstation for about two years now. This weekend, for the second time, I lost a drive (these are OCZ Vertex 1 drives) just as I was booting to transfer everything to my new workstation with 2x Agility 3's. Speed is better (OK it is cool since I got 1GB/s reads/ writes in ATTO even with the cheapo NAND), but the big thing is the fact you get a speed boost as well as spending less to hit a given capacity. Just make sure you have backups.

 

[mobilenvidia]

I've got 4x OCZ Solid3's (SATA3) in RAID0 in my rig
Speed is king with those puppies.

I have noticed an about 5% speed decrease since I first got them.
With RAID = no TRIM and NO Firmware updates, the early OCZ Firmware was no stable in SATA3 setups, much better now.

RAID0 for DATA not normally encouraged, you'd be better off with RAID10 if speed and data integrity is needed, then backups can be done less often ?

 

[dba]

I'm always looking of simple ways to increase my workstation/server combo power. Has anyone raided these SATA III drives in a production environment? How is performance over time effected without the use of TRIM? Is the HW garbage collection good enough to keep speeds up?

I was thinking that a a pair of SSD's running RAID 0 (with a HDD running hourly backups!) as a server data drive would be pretty 'snappy'.

I started a project about six months ago called the "dirt cheap data warehouse". My theory was that the very high read speeds provided by the latest SATA3 SSD drives would provide enterprise-class database performance at a very low cost. Because data warehouses are characterized by massive read workloads but relatively low write workloads, I figured that the limitations of MLC-based commodity SSD drives would not be an issue.

I started out with eight OCZ Agility3 120GB drives. I tried hardware RAID, software RAID, and a combination of the two. I tried various RAID levels and various RAID controllers. I settled on LSI controllers as host bus adapters and software RAID1E (Oracle ASM), which provided by far the best performance. As of now I'm up to five RAID controllers and 25 SSD drives in a single RAID1E array. Testing with IOMeter, I see read performance of 10,800 MB/Second. Testing using Oracle I/O calibration, I see over 7,000 MB/Second and 375,000 IOPS. I read terabytes per day from the drives but write only a few GB per drive per day at most. So far I have had zero drive failures and zero downtime and no odd behavior whatsoever. I have seen no noticable decrease in performance over time, which isn't surprising given how little data gets written. As an additional measure, I format the "120GB" drives only to 100GB, leaving the rest of the space for additional wear leveling area.

 

[dba]

I tried striping a pair of 128GB Crucial M4s as an experiment to compare to the performance of the same drives in a mirror; I found that as ZFS reads from mirrors in a round-robin fashion the read speeds weren't any lesser in the mirror than the striped pair. Since the array was largely read from rather than written to I didn't end up sticking with the stripe...

That is interesting about ZFS. Were you testing with multiple read threads or just one? I ask because when I tested RAID1 software mirroring in Windows and then hardware RAID1 using LSI controllers, I found in both cases that with multiple read threads I saw twice the performance as a single drive - meaning that reads were happening from both drives. I didn't test with a single read thread but now I wish that I had.

 

[ehorn]

I started a project about six months ago called the "dirt cheap data warehouse"..

Sounds like a really fun project. I imagine PCIe 2.0 is holding back linear scaling of Bandwidth/IOPS...

Theoretically, one could strap (24) fast 240GB 6Gb/s SSD's onto (3) LSI 9207's to get a SPAN over 5TB usable storage delivering raw bandwidth of ~ 12GB/s with > 1M IOPS. With deals on pricing, components would put come in around ~ 6-7K USD for a SPAN solution and ~ 12.5K for R1 (additional JBOD chassis, 3 more HBA's, etc...). With that many consumer grade SSD's, I would target less capacity and more redundancy in the configuration though.

Either way, it would make a cool "lets build a hot rod" article.

You recently purchased the 16X HBA for more density. Did you purpose it for this "dirt cheap data warehouse" solution? If so, have you had time to gather data points - what type of performance are you now seeing?

peace,

 

[Patrick]

Hopefully we will be seeing more on this soon. dba stopped by this week to borrow some supplies.

 

[dba]

Hi ehorn,

I am indeed testing the LSI 9202 as part of the dirt cheap data warehouse architcture. So far I have good results and bad - more details soon. I also borrowed a nice new Intel server from Patrick to provide some AMD versus Intel compare and contrast.

Sounds like a really fun project. I imagine PCIe 2.0 is holding back linear scaling of Bandwidth/IOPS...

Theoretically, one could strap (24) fast 240GB 6Gb/s SSD's onto (3) LSI 9207's to get a SPAN over 5TB usable storage delivering raw bandwidth of ~ 12GB/s with > 1M IOPS. With deals on pricing, components would put come in around ~ 6-7K USD for a SPAN solution and ~ 12.5K for R1 (additional JBOD chassis, 3 more HBA's, etc...). With that many consumer grade SSD's, I would target less capacity and more redundancy in the configuration though.

Either way, it would make a cool "lets build a hot rod" article.

You recently purchased the 16X HBA for more density. Did you purpose it for this "dirt cheap data warehouse" solution? If so, have you had time to gather data points - what type of performance are you now seeing?

peace,

 

[Mike]

Trim is not all that required as one might think. Some SSDs/controllers do pretty active and hard garbage collection, perfect in a fileserver that's idle for hours and hours, not so great for an actively used workstation that is powered off when not in use.

What i've been wanting to do is use a few SSD's, stripe them in ZFS (did i just say zfs?!) and use them as an external "program files" for all pcs around here. With poor man's deduplication I believe this could be speedy, have a comfortable volume size and save up diskspace for stuff that is installed by multiple users. Correct me if i'm wrong but that would be pretty cool. With deduplication running at night and then turned off during the day, from what i read, data should stay deduplicated during the day yet new files will be redundant till the next dedup run, though keep the speeds at the highest possible. Ironically 2 of the latest ssd's would fill up a 10gb ethernet pipe already, better move up to infiniband right away?

 

[ehorn]

Looking forward to it guys...

peace,

 

[ehorn]

...

Theoretically, one could strap (24) fast 240GB 6Gb/s SSD's onto (3) LSI 9207's to get a SPAN over 5TB usable storage delivering raw bandwidth of ~ 12GB/s with > 1M IOPS. With deals on pricing, components would put come in around ~ 6-7K USD for a SPAN solution and ~ 12.5K for R1 (additional JBOD chassis, 3 more HBA's, etc...). With that many consumer grade SSD's, I would target less capacity and more redundancy in the configuration though.

Either way, it would make a cool "lets build a hot rod" article.

Just saw that the folks over at StorageReview are testing a similar configuration in an all-in-one, slick 2U-48 bay chassis... Looking forward to some numbers...

http://www.storagereview.com/echostreams_flachesan2_custom_flash_array_build

peace,

 

[Patrick]

Pretty cool build! Those look to be custom slim drive bays. Traditional 2U = 24 2.5" front hot swap bays. You can do this with SSDs because you have less than half of the power consumption and heat generation compared with 10K or 15K rpm drives.

 

[dba]

Now that's a RAID built with a design philosophy that makes sense to me - smart use of existing commodity components. I'd love to see the pricing and I'd really be happy if it turned out to be something reasonable.

I do have to question the 20GB/Second throughput specification. Given that the system pulls data from the SSD drives and then pushes that same data back out the Infiniband ports, achieving that number will require 50GB/Second worth of throughput from the CPUs and the PCIe3 ports - 20GB from the drives, 20GB to the Infiniband ports, and 10GB pushed from one CPU across to the other.

The LSI 9207 cards do seem to be able to handle 4GB/Second each (thessdreview.com), so the quoted five of them could be good for the 20GB/Second. The Mellanox Infiniband cards are quoted as being capable of 100Gb (call it 10GB) worth of throughput each, so three of these are up to pushing their 20GB worth of data. That said, I would be surprised if all of these components could push all that data at the same time. At full speed, you'd be nearly saturing a full 80 PCIe3 lanes and, according to a smple Excel model of data and data paths, processing 25GB/Second/CPU.

I'm doing some testing with a new dual LGA-2011 Intel system and plan to push it up to 30 SSD drives and five LSI controllers. If performance is linear with this workload then I'll be more willing to believe.

Just saw that the folks over at StorageReview are testing a similar configuration in an all-in-one, slick 2U-48 bay chassis... Looking forward to some numbers...

http://www.storagereview.com/echostreams_flachesan2_custom_flash_array_build

peace,

 

[ehorn]

Agreed...

It looks like a really slick solution.

This unit being tested (EchoStreams FlashSAN2) is configured with (8) I/O controllers (5 HBA's and 3 HCA's). EchoStreams is using a proprietary MB, cuz I am not aware of a commodity MB yet available with 8 PCIe 3.0 slots. We know Dual LGA2011 makes available 80 lanes, so it is very doable. But I have yet to see one on the market. If SuperMicro (or others vendors) makes one, it will likely require an accompanying chassis designed to support that many slots.

Even still, look at the numbers they are touting.... 24GB/s and > 2M IOPS!! Very impressive using "mostly commodity stuff".

I suspect we will be seeing more vendors release similar solutions in the near future.

peace,

 

[ehorn]

... I do have to question the 20GB/Second throughput specification....

Me too...

The 9207's have shown linear scaling (and rumor has it they have more headroom beyond) with current 6Gb/s drives.

The IB 56Gb/s HCA's show around 6.5 GB/s each port and can be teamed to aggregate BW (i.e SMB 3.0 MultiChannel).

But yeah... Lets see the CPU usage when you get this much IO being pushed through the system.

StorageReview has not yet mentioned how they will be testing. It will be one to watch.

...
I'm doing some testing with a new dual-CPU Intel system and plan to push it up to 30 SSD drives and five LSI controllers. If performance is linear with this workload then I'll be more willing to believe.

Excited to see the numbers...

peace,

 

[Patrick]

No real reason we can't build something similar. The big issue is that eventually you hit a write wall using standard SSDs. That's why places like Violin Memory have gone away from "drives". Then again, price wise, it might be a good middle ground. I really think we are going to see a ton more of these types of setups now that SSD prices are plummeting.

I'm actually surprised that setup only has 64GB of DDR3. Would think you would go to 128GB+. Then again, they may just be using it as a fat storage node but being able to cache in memory usually yields good performance gains.

 

[ehorn]

No real reason we can't build something similar. The big issue is that eventually you hit a write wall using standard SSDs. That's why places like Violin Memory have gone away from "drives". Then again, price wise, it might be a good middle ground. I really think we are going to see a ton more of these types of setups now that SSD prices are plummeting.

Agreed...

I'm actually surprised that setup only has 64GB of DDR3. Would think you would go to 128GB+. Then again, they may just be using it as a fat storage node but being able to cache in memory usually yields good performance gains.

EDIT: The board does seem to support 128GB (8x DIMMS, 4x channels of DDR3 per CPU, total 16x DIMMs)

http://www.echostreams.com/downloads/Echostreams_FlacheSAN2_SpecSheet_Rev02.pdf

I agree. Historically, we do see improvements with Ram as Cache. But with IO numbers like they are touting, the whole solution already seems like one large (and fast) Ram disk with throughput which approaches native memory speeds (even the 2133Mhz variety). But lets see what the numbers (latency, writes, IO, etc...) look like.

Either way, if the marks are close to the specs, it will be smokin' fast storage solution for the price point.

peace,

 

[dietcokefiend]

Hey guys, wanted to introduce myself as Kevin over on StorageReview. Saw the links inbound from this forum and thought I'd jump in to answer any questions that you guys had.

Testing will be done using Windows Server 2012 with SMB 3.0 support. We are actually interested where bandwidth ends up plotting out, since in some very preliminary benchmarks on the manufacturer side, they were seeing 22GB/s using 48 Micron P400s. Of course the figures are going to change switching the types of drives used... where off the bat we are using the Intel SSD 520 and hopefully adding some others along the way for comparison.

I saw some questions on the motherboard this box is using, this is what EchoStreams specs out for this platform:

http://www.intel.com/content/www/us/en/motherboards/server-motherboards/server-board-s2600ip.html

To get an idea of the types of tests we will be performing (although not limited to these) our Fusion-io ioDrive Re-review that we posted today includes our new benchmarking process.

http://www.storagereview.com/fusionio_iodrive_duo_enterprise_pcie_review

Some of that will be adapted along the way to make it more applicable for this platform (as well as multi-server testing) to hit full saturation. If any of you have questions feel more than welcome to ask in this thread and I'll make a note to visit it daily to see if anything crops up

 

[ehorn]

Hey Kevin,

Thanks for sharing some preliminary insights!

When I saw the configuration, I immediately thought to myself; that is the setup to test Romley/SMB 3.0 saturation with...

Those are some pretty impressive preliminary numbers with the P400's. I am even more anxious to see the numbers with the 520's...

That is one heck of a MB. Leave it to Intel to cram that many PCIe slots in. Now we just need some vendors to build chassis' to accommodate that beast.

I do have a few questions;

Were those preliminary BW numbers local or measured at the host?
What was the CPU utilization on the target (and initiator if SMB Direct was tested) during those peaks?

Thanks again for stopping by.

peace,