PCIe SSDs on older server question

[CyklonDX]

What is this video about then? What is the purpose of such mirror?
I thought it will work just fine with one disk out and all I need is to insert new disk, go to "Disk manager" and run some command to "restore"?

I guess purpose is to do it - and if a disk dies, you can still get your data. If they had os on it - one would go to recovery mode / installation fdisk and bring the dynamic disk online. (one may have to repair mbr/boot whatever)

Then how do I manage mirror? What if this controller gives up? Can I move disk to SATA port and boot from it? I mean, is it just hard bitwise copy controller?

Typically there's software (like MegaRAID) with any raid controller that gives you insights and ability to manage raid.

Also, how will it be presented to Windows? As single drive? will I see all drives in utilities like DiskInfo?
AND.. What tool would I use to monitor this mirror?

You would see single VD drive. DiskInfo won't work - but some tooling that is capable of communicating with raid controller is able to get you that data. Typically you will have access to all data info from your raid's management software like MegaRAID; you will see their state, and so on.
Do not mistake, this will be hardware accelerated raid - its not software like dynamic disk from windows.

Note: This is just a stock photo from google

EDIT: Is this correct cable for my backplane? I have 8 disk bay empty/disconnected. Can I just get 2 of those cables, connect to LSI card and have all 8 disks working?

Yes.

EDIT2: And then again, it says "HOST BUS Type: PCIe"
And I wonder if this card will be hosting bootable drives??

Yes.

What about this? For the same money I can just replace MB and it will work with all my stuff for sure..

Can't comment on that, most often its always much better solution to use pcie card rather than use motherboard provided controllers. They are always "crappier". I would always use lsi card rather than use built in sata/sas slots.

 

[katit]

Typically there's software (like MegaRAID) with any raid controller that gives you insights and ability to manage raid.

But still. RAID Controller fails. What are my options? Should I replace controller and all will work?
Can I connect single disk from mirror to my MB SATA port and it will work?

Can't comment on that, most often its always much better solution to use pcie card rather than use motherboard provided controllers. They are always "crappier". I would always use lsi card rather than use built in sata/sas slots.

Yes, this is what I was reading too when I was researching topic. If you say LSI card will work with my system (not like those PCIe disks) and make it bootable - I can totally go with it, especially that it will allow me to use all 8 bays.

What about those PCIe disks I got? I guess I will live them for windows to manage as a dynamic mirror.. Just for kicks I created and tested it - numbers look much better vs when it was a single disk.

 

[CyklonDX]

But still. RAID Controller fails. What are my options? Should I replace controller and all will work?

As per photo you posted earlier each of backplanes you have there have 2 sas cables going out of them. You can get 2 sas/raid controllers and use mpio (Each backplane would have 1 cable going to raid controller 1 and 1 cable going to controller 2; same for the other.)
So if one pcie card fails (which would be quite rare occurrence - they just don't fail) all traffic would still be present from 2nd one.


the lsi 9460 8/16i raid controller has support for 2x nvme/u.2 disks (but it will be much more expensive) - but still one of few ways to get raid1/0 on nvme/u.2 - else you would use software raids (which aren't that bad - unless you want to boot from those) ~ it also will heat up a lot more since it uses plx chip.

 

[katit]

As per photo you posted earlier each of backplanes you have there have 2 sas cables going out of them. You can get 2 sas/raid controllers and use mpio (Each backplane would have 1 cable going to raid controller 1 and 1 cable going to controller 2; same for the other.)
So if one pcie card fails (which would be quite rare occurrence - they just don't fail) all traffic would still be present from 2nd one.


the lsi 9400 16i raid/it controller has support for 2x nvme/u.2 disks (but it will be much more expensive) - but still one of few ways to get raid1/0 on nvme/u.2 - else you would use software raids (which aren't that bad - unless you want to boot from those)
View attachment 36279

Yes, I think I will be totally OK with software raid on U.2 disks. If anything - would be nice to get them out of PCI slots (on adapters) but OTOH - It's where I get some ventilation and I am not sure of where else to mount them inside a case. So, it's fine.

I recommend something of 9300-8i or 9400-8i (if you plan on nvme's)

With that said - I don't plan on NVMes, 9300-8i look much cheaper, bunch of them sold as "IT mode" which I don't want in my case. Also looks like a lot of different brands make them. Any preferences to which one?

 

[CyklonDX]

non branded just lsi are best in my opinion
- branded often have limitations in io queue depth or integrations for their management controllers like idrac, bmc etc. (or even hardware lock)

from 9300-8i raid mode cards here are some options (battery is desired)

LSI MegaRAID 9361-8i 12Gb PCIe 8-Port SAS/SATA RAID 1Gb w/BBU/CacheVault/License | eBay

Clean, used pulls from off-lease servers. Warranties do not cover any defects that are listed in the item details section above. Clean, tested pulls from off-lease servers.

www.ebay.com

LSI | MR SAS 9361-8i | MegaRAID 12g RAID Controller Card With LSI 49571-13 | eBay

Part # MR SAS 9361-8i.

www.ebay.com

LSI MegaRAID 9361-8i PCIe 3.0 x8 SATA SAS 8-Port 12Gb/s RAID + BBU & CacheVault | eBay

Eight internal ports support up to 128 SATA/SAS drives. The PCIe 3.0 x8 host interface eliminates bottlenecks to maximize the operational performance of the latest 12Gb/s SAS infrastructure. This card supports RAID 0, 1, 5, 6, 10, 50, and 60, and also supports the optional CacheVault Flash...

www.ebay.com

(Lot of people prefer IT mode - for use with zfs, and such)

 

[katit]

non branded just lsi are best in my opinion
- branded often have limitations in io queue depth or integrations for their management controllers like idrac, bmc etc. (or even hardware lock)

from 9300-8i raid mode cards here are some options (battery is desired)

LSI MegaRAID 9361-8i 12Gb PCIe 8-Port SAS/SATA RAID 1Gb w/BBU/CacheVault/License | eBay

Clean, used pulls from off-lease servers. Warranties do not cover any defects that are listed in the item details section above. Clean, tested pulls from off-lease servers.

www.ebay.com

LSI | MR SAS 9361-8i | MegaRAID 12g RAID Controller Card With LSI 49571-13 | eBay

Part # MR SAS 9361-8i.

www.ebay.com

LSI MegaRAID 9361-8i PCIe 3.0 x8 SATA SAS 8-Port 12Gb/s RAID + BBU & CacheVault | eBay

Eight internal ports support up to 128 SATA/SAS drives. The PCIe 3.0 x8 host interface eliminates bottlenecks to maximize the operational performance of the latest 12Gb/s SAS infrastructure. This card supports RAID 0, 1, 5, 6, 10, 50, and 60, and also supports the optional CacheVault Flash...

www.ebay.com

(Lot of people prefer IT mode - for use with zfs, and such)

Thanks! I wish I could buy you a beer for all your help.

I remember I was updating firmware on built in card in my Supermicro MB for FreeNAS.
Now it's different, I want RAID1

 

[katit]

Got MegaRAID, installed, everything seem to be working, took some time to find software.

I am not sure about default settings it used, is it correct for SSD? My setup is mirror + hot spare.
Idea is that if disk fails I won't have to look for replacement and will get some time.

Are those config items looking good?
What is SSD Caching? Useful or?




This is speeds am seeing which make sense to me with 3500 disks

 

[CyklonDX]

The write looks really bad for RAID1 (but i don't know which disks you are using.)
RAID1 should be 2x Read, 1x Write (of single disk)

1. Have disks finished syncing? (it can sometime take decent amount of time)
2. You should disable disk caching as in most cases its pointless with sas ssd's, and will slow you down & all connected disks to the controller once you run out of cache. (unchanged = cache is enabled in default write setting)
3. The 256K stripe should be fine, but if it offers you larger size you can use that - and potentially gain little bit of performance (few MB/s) - you need to test that, as its highly depndant on logical sector sizes; tho in raid1 it really doesn't matter much.
4. SSD Caching simply refers to ability to add SSDs as a cache for HDD array.
5. The spare functions just like that, once one of your disks dies it will be automatically replaced with the spare.
6. Data protection refers to encryption (most often SED)

7. With SAS SSD's, or SSD's in general you will have cache disabled, but

Write:
Write-back = The data is written first to cache on raid controller and will report completion; the raid controller will then try to flush the data into the disk in background.
Write-through = The controller only signals finished process once data been completely moved to disk.

Read:
Normal = there's no read ahead, when request comes in it will just load that data from the disk.
Read-ahead - It will try to load more data into cache that will potentially allow better performance.
Adaptive - if it detects random sector read it will turnoff read-ahead - so it doesn't load pointless stuff into cache.


Ensure you have proper cooling for the controller, if it overheats it will drop performance. If you are running sas ssd's ensure you have correct sas cables connected; and that the raid controller is getting all pcie lanes it needs.
If performance issue persists check if you are running latest firmware, and upgrade.

 

[Chriggel]

I think the results are decent and in the expected range with 240 GB Intel S3500 SSDs. They're not very fast.

 

[katit]

I changed disks to pass through and got this result:


Then I moved disk to SATA6 port on MB and got this result:


So, @Chriggel is right, this is all it is I guess. Interesting though that disk connected through LSI gave slightly better performance.

Firmware updated to latest.
Temp of chip looks high to me but I don't know what it should be:

 

[Chriggel]

63°C is perfectly normal, nothing to worry about.

 

[CyklonDX]

Firmware updated to latest.
Temp of chip looks high to me but I don't know what it should be:

View attachment 36455

Is this temp under load?

If those disks are slow on their own, you can turn on cache back on, and just use write-back, and adaptive read-ahead.

 

[katit]

Is this temp under load?

If those disks are slow on their own, you can turn on cache back on, and just use write-back, and adaptive read-ahead.

No but there is no such thing as "load" in our environment. And fans controlled by BMI so it should be taken care of

 

[katit]

2. You should disable disk caching as in most cases its pointless with sas ssd's, and will slow you down & all connected disks to the controller once you run out of cache. (unchanged = cache is enabled in default write setting)
3. The 256K stripe should be fine, but if it offers you larger size you can use that - and potentially gain little bit of performance (few MB/s) - you need to test that, as its highly depndant on logical sector sizes; tho in raid1 it really doesn't matter much.
4. SSD Caching simply refers to ability to add SSDs as a cache for HDD array.
5. The spare functions just like that, once one of your disks dies it will be automatically replaced with the spare.
6. Data protection refers to encryption (most often SED)

I did set largest stripe 1Mb, also changed some other stuff like Write Through, Never Read Ahead and performance so far got worse:



Simply changed to "Write Back" from "Write Through" Ran test few times - differs but about like this.
Not sure WHY read speed so high? I have 2 drives that read at 500, how do I get 1700?
And WHY changing of Write mode affected reading so much?



Setting "Always read ahead" for some reason worsened "Write" speed..

 

[CyklonDX]

Basically you are reading from your cache on raid controller; but yeah pick whatever works best for you. The 1Gig test may not be reflective to real performance (keep in mind)