LSI / Avago CacheCade incredibly slow?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

DReffects

New Member
Apr 8, 2019
6
0
1
Hey Guys!

So I've recently upgraded from an ancient Areca ARC-1260 controller to a MegaRAID SAS 9361-8i controller with the Cache Cade option.

I am running the controller with:
  • 4x HGST Ultrastar HE12 HUH721212ALE600 12 TB drives in a RAID5 Volume
  • 2x Samsung 860 Pro 500GB SSD drives in a RAID1 Volume for CacheCade
I found my setup to be HORRIBLY slow instead, slower than the old Setup which ran in RAID6 mode on top of the old architecture. So obviously something with my configuration seems to be wrong.

My Raid5 Volume is configured like this:
upload_2019-4-8_12-59-29.png

The CacheCade Volume is configured as RAID1 with the following parameters:
upload_2019-4-8_13-0-30.png

I have had big troubles when unpacking or creating large files (5-30 GB) on the volume. Write speeds range from 45-65 MB/s WITH cache cade, 300-500 MB/s without.

Also with cacheCade enabled taskmanager shows 100% utilization of the volume even with light loads like extracting a rar archive.

I've extraced a 10GB rar file - taskmanager is showing ~50MB/s of write speed and 100% utilization. During the extraction I've disabled cachecade via the megaraid console and utilization immediately dropped to ~30% and write speeds went up to 350-400MB/s.

Sequentials benchmarks with HDTune with CacheCade enabled seem to be fine, apart from the really bad read performance:

With CachCade Volume set to "Write Through":
upload_2019-4-8_13-5-15.png
(Horrible IOPS, good Sequential Performance)

With CacheCade Volume set to "Write Back":
upload_2019-4-8_13-35-58.png
(Slightly better Read IOPS, but still bad. Good sequential write, bad sequential read performance)


Same test WITHOUT CacheCade:
upload_2019-4-8_13-21-8.png

Notice how the Write IOPS are way higher without cachecade?

For comparison: this is the same benchmark with an older samsung 850 EVO 500GB drive on the motherboards SATA connector:
upload_2019-4-8_13-23-3.png

Quite frankly those would be the IOPS Performance numbers I'd expect from the controller with CacheCade enabled.

Testing with CrystalDiskMark and CacheCade is another story:
upload_2019-4-8_13-11-23.png


Same Scenario without SSD Cache:
upload_2019-4-8_13-27-44.png


Read Access Times/Speeds with cache cade / write through:
upload_2019-4-8_13-9-50.png

Read Access Times/Speeds with cache cade / write back:
upload_2019-4-8_13-41-10.png

Read Access Times/Speeds without cache cade:
upload_2019-4-8_13-24-27.png

As you can see there's hardly any difference between ssd cache and no ssd cache. Infact without the cache the setup outperforms the ssd cache at times.

Again for comparison an older Samsung 850 EVO 500GB drive:
upload_2019-4-8_13-25-55.png

Any Idea what's going on here?

Thanks :)
 

tomaash

Active Member
Oct 11, 2016
110
53
28
Europe, Slovakia
Indeed, my though would be that the controller bypasses SSD cache on writes (does sync writes) and consumer SSDs are terribly bad at this. You are getting almost internet-level latencies on those ops...
 
  • Like
Reactions: DReffects

DReffects

New Member
Apr 8, 2019
6
0
1
Try enterprise ssds (intel s36xx/37xx, hgst husmm) that do not degrade performance under sustained workloads like the consumer/prosumer ssds.
For more information why consumer ssds are not that great for such a workload check this post from microsoft:Don’t do it: consumer-grade solid-state drives (SSD) in Storage Spaces Direct
Indeed, my though would be that the controller bypasses SSD cache on writes (does sync writes) and consumer SSDs are terribly bad at this. You are getting almost internet-level latencies on those ops...
Thanks for the swift input!

I am aware that consumer SSDs have some drawbacks. Yet when using the RAID1 CacheCade Volume Set as an ordinary Raid1 Volume through the same controller I do get these speeds:
upload_2019-4-8_16-29-7.png

You can see a clear drop in performance after the ~5GB mark but overall the performance is way better then when being used as a CacheCade volume. Especially in terms of the IOPS. Also the read speeds max out at a constant 2x500MB/s. So the drives itself do not seem to have a performance issue even when reading/writing large blocks of data.

The Rand Access values are good as well when using the two SSDs in an Raid 1 Volume:
upload_2019-4-8_16-29-51.png

I'd be happy to just swap out the SSD drives but in light of these benchmarks I don't quite see why the SSDs itself are a bottleneck when performing like this outside of CacheCade. Or am I getting this totally wrong?

Are the Access Times without CacheCade in a normal range? I do not have any past experience about this.
 

DReffects

New Member
Apr 8, 2019
6
0
1
Just did some additional testing. When configuring the SSDs in RAID 0 CacheCade Performance is way different:
upload_2019-4-8_16-50-18.png


Still sucks on the Read Access times, though.
upload_2019-4-8_16-53-16.png


Also, Raid0 ain't a configuration I am comfortable with...
 

myrison

Member
Jan 26, 2011
61
41
18
I ran across your post today while testing a new CacheCade Pro 2.0 setup. The main datastore is 11x8TB WD Whites in RAID 6, Cache is 2x200GB SanDisk Enterprise SAS SSDs (on the LSI approved list). I thought I'd post my results here for future users as well. Some nice improvements on random 4k reads/writes, but I think it's too big of hits on sequential read/write for me to keep this active for my use case. These were 2 different tests where in the top one I'm showing MB/s and the bottom one shows the IOPs view.

The second set of 4 images shows latency for all 4 tests in one view.


MB/s & IOPS view

Screen Shot 2020-04-19 at 12.23.00 PM.png


Latency View
Screen Shot 2020-04-19 at 12.27.11 PM.png
 

DReffects

New Member
Apr 8, 2019
6
0
1
I ran across your post today while testing a new CacheCade Pro 2.0 setup. The main datastore is 11x8TB WD Whites in RAID 6, Cache is 2x200GB SanDisk Enterprise SAS SSDs (on the LSI approved list). I thought I'd post my results here for future users as well. Some nice improvements on random 4k reads/writes, but I think it's too big of hits on sequential read/write for me to keep this active for my use case. These were 2 different tests where in the top one I'm showing MB/s and the bottom one shows the IOPs view.

The second set of 4 images shows latency for all 4 tests in one view.
THANK YOU for posting this. I was going nuts over the new controller about 2 years ago since it cost so much but delivered so poorly. The whole point to get a new raid card was that i wanted to saturate a 10G network link and increase performance for network based video editing.
Both of these goals cannot be accomplished with the LSI raid card :( in fact my nearly 10 year old areca raid card delivers the same sequential read/write performance. :(
 

myrison

Member
Jan 26, 2011
61
41
18
Hi @DReffects . You're welcome :). While I was not overwhelmed with the CacheCade results, I actually didn't think the card was doing that badly with these sequential read/write numbers in RAID 6 with 11 "slowish" SATA drives connected. However, I was having some other weird problems with the related datastore in vmware and just recently deleted it and started fresh. Now I'm seeing considerably increased performance (still without CacheCade enabled).

Updated results posted below.

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (UWP) (C) 2007-2019 hiyohiyo
Crystal Dew World: Crystal Dew World
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q= 8, T= 1): 1424.558 MB/s [ 1358.6 IOPS] < 5874.99 us>
Sequential 1MiB (Q= 1, T= 1): 1393.839 MB/s [ 1329.3 IOPS] < 749.12 us>
Random 4KiB (Q= 32, T=16): 14.511 MB/s [ 3542.7 IOPS] <141786.48 us>
Random 4KiB (Q= 1, T= 1): 1.159 MB/s [ 283.0 IOPS] < 3525.63 us>

[Write]
Sequential 1MiB (Q= 8, T= 1): 1451.433 MB/s [ 1384.2 IOPS] < 5751.56 us>
Sequential 1MiB (Q= 1, T= 1): 1471.464 MB/s [ 1403.3 IOPS] < 708.50 us>
Random 4KiB (Q= 32, T=16): 7.968 MB/s [ 1945.3 IOPS] <216440.68 us>
Random 4KiB (Q= 1, T= 1): 8.796 MB/s [ 2147.5 IOPS] < 462.85 us>

Profile: Default
Test: 8 GiB (x1) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Date: 2020/04/25 7:49:32
OS: Windows 10 Professional [10.0 Build 18362] (x64)
 

DReffects

New Member
Apr 8, 2019
6
0
1
and started fresh. Now I'm seeing considerably increased performance (still without CacheCade enabled).
Interesting. could this be a fragmentation issue?

Profile: Default
Test: 8 GiB (x1) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Could you please run this with a 32GB setting? :)

I am currently running the controller with 4x12 TB and can reach ~650mb/s in Raid5 (==~160MB/s per drive)
Do you have knowledge if the controller scales up rather linear, meaning ~8 Drives == 2x the performance of 4 drives?

The biggest issue i have is with the extremely bad IO performance:
Read Access Times/Speeds without cache cade:
I am currently experimenting with my cache cade drives as a read cache with a software called "primo cache". I've set up a raid1 with the ssds and use it as a level 2 read cache with primo cache. This increases read performance DRASTICALLY (up to 1100 MB/s == 2x 550 MB/s (maximum of the ssds)) for cached data with the access times of an ssd.

You can also use primo cache as a l2 write cache but that introduces the risk of data loss when having blue screens or sudden power loss. the latter of course can be circumvented by using an UPS.

I've been in communication with the developer about the risks of using the software also for write caching. the "only" risk when using a raid1 array as the write cache and having an UPS are blue screens.

Here's the full Q&A I did with the developer:

The Raid Array is currently powered by a LSI Controller with a battery backup unit and I have the option to run a SSD-based Raid 1 or even Raid 10 volume for the L2 Cache Partition.

· Will this solve any issues during power-loss?
I don't think so. When the whole computer lost power, PrimoCache has no chance to flush deferred write-data even the raid still has a backup battery. A ups is required to handle the case of sudden power-loss.

· Also, what about unexpected blue screens? Any ungraceful shutdown like blue-screen, suspending will cause possible data loss.

· If data loss should happen while using l2 defere-write caching: will this only affect the files currently being written or will this degrade the whole underlying raid array? All deferred data in cache which havent't been written to the disks will get lost. It is possible some deferred data belongs to file system meta data. So in rare cases it might cause the problem of the whole file system.

· Does your software “touch” the actual disk that it caches in any way besides delaying the write process? In other words: can I simply remove the disk I am caching and it still works or will the original disk or volume only work when connected with your software? Except defer-writing, PrimoCache doesn't "touch" the actual disk, meaning that the source data in disks will be keep intact.

· Is there any kind of fail-safe-mechanism that provides feedback if the defer-write functionality is unable to write files back to the disk so the user can initiate a restore from backup? (Error logs etc) PrimoCache will log events on defer-write failures and it will try several times (3 times). Deferred data might get lost on failure after 3 times of trying. I'm sorry that so far users cannot re-initiate the writing. We are working to improve this process, trying to save failed data to another place for restoring later.

· What happens if the l2 cache itself fails (in both read or write conditions) – are all files on the original disk or are they being moved to the l2 cache? Will read from or write to the original disk.

· What happens if the l2 cache is no longer present during reboot? It's no problme. PrimoCache will use L1 cache or the original disk.


A few follow-ups:

  • UPS is standard equipment for my servers – the only remaining issue would be bluescreens then? Yes, any ungraceful shutdown cases like bluescreens or system freeze.
  • Does PrimoCache resume to transfer the missing data from the l2 deferred-write cache to the array if power is restored / the machine reboots or does the write cache simply flush itself and start over, leaving broken fragments on the array? Sorry, but so far it cannot.
  • Where can I find the mentioned defere-write log? You can find error events in Windows Event Viewer and check the soruce "FancyCcV", if defer-write writing failure happened. No event will be logged if there is no failure.
  • Is there a way to exclude specific files and folders from a volume? Most of my customers have one large data volume with “highly important” and “less important” files mixed… PrimoCache is a disk-block-level caching program which means that it caches data based on disk sectors and has no any file information on its cached data. However, we do working on this request since many users want it.
  • Is it safe to say that only using l2 chache for reads is without any risk? Yes, if defer-write is not enabled, it is always safe whether you use L1 cache or L2 cache.
  • Are there any known incompatibilities I should know about? For the known issues, please see PrimoCache Known Issues List - Romex Software Forum
  • What about backup tasks? Will a full backup (by Acronis or Backblaze) result in a transfer of the last read file into the read cache? (which would basically mean a sort of cache reset each time a backup is being performed…) If so, is there a way to exclude certain programs (acronis, backblaze, etc) from filling the cache with data? Currently you may pause the cache when backup is starting to avoid the filling cache by backup data. See [Tool] Automatic pause and resume for backup-software - Romex Software Forum for your reference.
  • Does the l2 cache work on a file basis or on a lower level? Will primo cache cache entire files if only a certain block of the file is being accessed? PrimoCache works based on disk sectors, so only certain blocks will be accessed.
 

myrison

Member
Jan 26, 2011
61
41
18
Interesting. could this be a fragmentation issue?
Long story... but I accidentally deployed a large 60TB volume initially using thin provisioning in vmware and didn't realize it. When I did, I manually inflated the disk which took several days, and even after that, the performance was never right. It probably did have something to do with fragmentation as there were a lot of writes between creating it and

Could you please run this with a 32GB setting? :)
Here you go:
------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (UWP) (C) 2007-2019 hiyohiyo
Crystal Dew World: Crystal Dew World
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q= 8, T= 1): 1040.795 MB/s [ 992.6 IOPS] < 8046.17 us>
Sequential 1MiB (Q= 1, T= 1): 1559.435 MB/s [ 1487.2 IOPS] < 668.74 us>
Random 4KiB (Q= 32, T=16): 16.000 MB/s [ 3906.3 IOPS] <129008.40 us>
Random 4KiB (Q= 1, T= 1): 0.644 MB/s [ 157.2 IOPS] < 6346.89 us>

[Write]
Sequential 1MiB (Q= 8, T= 1): 1619.770 MB/s [ 1544.7 IOPS] < 5153.07 us>
Sequential 1MiB (Q= 1, T= 1): 1615.494 MB/s [ 1540.7 IOPS] < 645.18 us>
Random 4KiB (Q= 32, T=16): 25.547 MB/s [ 6237.1 IOPS] < 73330.72 us>
Random 4KiB (Q= 1, T= 1): 7.610 MB/s [ 1857.9 IOPS] < 535.06 us>

Profile: Default
Test: 32 GiB (x1) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Date: 2020/04/28 11:08:54
OS: Windows 10 Professional [10.0 Build 18362] (x64)

Screen Shot 2020-04-28 at 11.10.04 AM.png


I am currently running the controller with 4x12 TB and can reach ~650mb/s in Raid5 (==~160MB/s per drive)
Do you have knowledge if the controller scales up rather linear, meaning ~8 Drives == 2x the performance of 4 drives?
I do think it's relatively linear minus the RAID overhead (up to max capacity of the card) from what I've seen. You'll see above how my 11x8 compares to your 4x12 (though mine are SATA 5400 RPMs that best case seem to max around 120MB/s sequential).

I am currently experimenting with my cache cade drives as a read cache with a software called "primo cache". I've set up a raid1 with the ssds and use it as a level 2 read cache with primo cache. This increases read performance DRASTICALLY (up to 1100 MB/s == 2x 550 MB/s (maximum of the ssds)) for cached data with the access times of an ssd.
I've read about PrimoCache as well and it looks like it's fairly awesome for your use case. What are you actually doing with the server by the way? The VM that primarily uses this datastore is a media server so it is 1,000% overkill for my needs, but hey, I like overkill :)
 

Your name or

Active Member
Feb 18, 2020
281
41
28
Grüss Gott veryone
I got a Licence from Ebay (Dongle) for my good old Raid Card who support them to. What should I consider before I buy a SSD?
I want to "speed up" my 1TB Red HDD by Caching everything who going in and out. Do I need to have 2 SSDs?
Thanks
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Do I need to have 2 SSDs?
You need redundancy (raid 1 or 10 for the ssds) if you want to cache writes. I'm trying to find the pdf where I read it, but can't find it right now...

From LSI MegaRAID CacheCade Pro 2.0 Review:
LSI MegaRAID CacheCade Pro 2.0 Specs
  • Compatible with MegaRAID SAS 9260/9261/9280 series controllers
  • Supported Operating Systems – All operating systems supported by MegaRAID controller card
  • Max. Number of SSD in a CacheCade SSD Pool – 32
  • Max. Number of SSC VD Supported per Controller – Up to 64
  • Max. CacheCade Capacity per Controller – 512GB
  • RAID 1, 10 SSD Write Cache protection
 
  • Like
Reactions: Your name or

myrison

Member
Jan 26, 2011
61
41
18
Grüss Gott veryone
I got a Licence from Ebay (Dongle) for my good old Raid Card who support them to. What should I consider before I buy a SSD?
I want to "speed up" my 1TB Red HDD by Caching everything who going in and out. Do I need to have 2 SSDs?
Thanks
For only a single 1TB HDD, wouldn’t it be cheaper just to buy a 1TB SSD and move everything to it?