LSI 9266-8i experiences

[mobilenvidia]

After my not so sucessful adventure with the LSI9202-16e, I discovered this on ebay to brighten my spirits.

LSI9266-8i with CacheCade v2.0 delivered to NZ for a mere $420
Got it from same seller as my LSI9261-8i, also with CC v2.0
I really like the SAS ports on the back of the card for ease of running cables.
Just a pity the LSI9266 takes a different Cachevault module than the LSI9265, I could have swapped the M5016 CV module.

I'll put the 7k2000's on this with RAID0 Cachecade v2 SSDs
Will do a direct comparison of Cachecade and SAS2108 and SAS2208 controllers

I will have to sell of some of my other bounty of controllers to pay for my obsession

More experiences to follow.

 

[mobilenvidia]

Card arrived today, super duper quick shipping.

Having only seen the front of these cards, I wondered how LSI handled the 1GB cache.
These are placed on the back of the card (normally on the front).

The card can take a CacheVault module which sits on the front as a daughter board.
This card is different to the IBM M5016 with a slim slot to plug it in (M5016 has a wide slot)
But it's the same as the LSI9271.

The card looks very well laid out with the SAS ports on the back makes for nice cable management.
This card replaces my LSI9261-8i which looks similar, this being the SAS2208 version of it.
The LSI9171 is the next step up again with PCIe Gen 3 support where as the 9266 is Gen 2

All the features keys are enabled permanently incl Cachecade pro V2 and Fastpath
My 9261 has everything but CacheCade Pro v2 enabled, but I have it running on a trial key.

I remembered to remove the CacheCade array on the 9261 before switching the drives to the 9266
Upon reboot the RAID6 array was available right away so it imported it with no issues.
Setup Cachecade Pro v2 and I was away again all in minutes.

Benchies to follow.

 

[mobilenvidia]

Setting my RAID6 array to No Read-ahead and Always write back, at boot time would change to Always read-ahead and Write-through.
Setting BIOS to show boot messages I see because the 9266 has no battery it sets the Cache to Write through.
Need to sort this out with MegaCLi.
Not come across this before as all my other controllers have a BBU.

 

[mobilenvidia]

D:\>megacli -LDSetProp cachedbadbbu -LAll -aAll

Set Write Cache OK if bad BBU on Adapter 0, VD 1 (target id: 1) success

Exit Code: 0x00

Should do the trick, reboot and find out

 

[mobilenvidia]

Nope, still going to Write Through, Grrrr

Did how ever get JBOD enabled

MegaCli -AdpSetProp -EnableJBOD 1 -aall

Adapter 0: Set JBOD to Enable success.

Exit Code: 0x00

Not sure why this is not enabled by default on SAS2208 controllers

 

[mobilenvidia]

I'll try this then

D:\>storcli /cx(all) set batterywarning=off

Nope, controller insists on policy change at boot to WT due to no battery.
Seems a little strange as Always Write Back was supposed to do just that


Hmm, removed the CacheCade array, made this a RAID0 array, set to Always WB
Reboot and stays in Always WB mode.

Copy 1TB data of RAID6 array to new 4TB drive, I'll leave this to copy away while I sleep.
Remove array, remake array then test again.
I wonder if deep down in the NVRAM it keeps the initial way the array was setup, although I usually setup arrays with Always WB, anyway will test.

 

[mobilenvidia]

Over the last couple of weeks all sorts of bizarre things have been happening.

I upgraded to LSI latest FW to discover that the cache was no longer visible
This lead to Cachecade key being there but not supported
Also RAID6/60 was lost

Tried all sorts of tricks and FW to get this working again.
In a last effort I flashed the Perc H710 FW
Upon boot I get a message, BBU is missing or Bad, press 'D' to disable this message, press D it did.
Further into the boot, I see there is 1GB of cache again.
The battery not being there or bad message no longer gets displayed and 'Always WB' after a boot sticks

I flashed the latest FW again and everything was broken again, so back to Dell FW

Boot into windows, check the card all the feature keys are working again.
AND now I have the ability to powerdown active drives
It looks like Dell like this feature enabled and so do I.
Now drives happily go to sleep after 30min.

I have since updated to a newer Dell FW 21.1.0007, this also works
I did have to m0flash the FW with Megacli to get it to flash at all.

One draw back there is no WebBIOS, I need to find the command to get this working.
Not a biggy as I won't be booting from this card.

But moral of story what the is going on with LSI's FW

 

[white-hot]

The Dell FW sounds interesting. I have been using the 0048 FW to retain the drive spindown on the 9261-8i I purchased from you and am wondering if this might also work for that card. While I am not currently using any of the features lost with this FW (CacheCade v2 mainly) it would be nice to have that available. Interested to see if you get the WebBIOS working as well.

 

[mobilenvidia]

The Dell FW sounds interesting. I have been using the 0048 FW to retain the drive spindown on the 9261-8i I purchased from you and am wondering if this might also work for that card. While I am not currently using any of the features lost with this FW (CacheCade v2 mainly) it would be nice to have that available. Interested to see if you get the WebBIOS working as well.

remember the LSI9266 uses a different SAS ROC than the LSI9261, so the Firmware are not interchangble.
But the idea is the the same, you could try the FW for the PercH700 (where as I'm using the H710)

LSI have just released a new FW, but still when I flash I loose the Cache again
BUT I did notice a new Feature key "Cache offload' which was enabled
But since CC Pro v2 was enabled but not supported due to lack of cache can't test it.

This is just so weird

 

[mobilenvidia]

OK, just installed the very latest SAS2208 FW from LSI (23.12.0011) in MSM and now I'm fully working
Can't figure this out, doing it via DOS with MegaCli got me no where.

But anyway the key I saw in WebBIOS (Cache offload) in MSM is called CacheVault, not sure why it's a key.

Now I'll need to put back in the IBM M5016 and see what this does as it actually has the CacheVault module built in

 

[mrkrad]

primary rom are coded keys of 1/0 for each feature like enableJBOD, or raid-6.

the FOD or physical keys then run afterwards. so they can key in enablecacheoffload=1 to allow cache vault in the main key.

The m5014/m5015 performance key enables everything including cachecade 2.0 most folks don't get cachecade 2 from physical keys.

 

[alexwill22]

I just purchase this card a couple of weeks ago and just installed it in my server over the weekend. I will push alot of data between this server and a zfs raidz 2 server. Would I benefit using cachevault 2.0 and fastpath for large and small file transfers or is this something for a environment of 20 or more people hitting this server with database apps or file sharing? I am just wondering if I should bite the bullet and pay $230 for the key. Right now I am using a trial key and haven't had time to test the difference with or without the cache drives.

I am using 6x Seagate Enterprise 1TB SATA III drives Raid 6 and 2x 128GB Samsung 830 series SSD Drives Raid 0 for the cache. Main OS is Windows 8 Pro 64bit running on Samsung 2x 128GB 840 Pro SSD Drives Raid 0. System is used for video encoding and editing.

 

[BigXor]

OK, just installed the very latest SAS2208 FW from LSI (23.12.0011) in MSM and now I'm fully working
Can't figure this out, doing it via DOS with MegaCli got me no where.

Do you still have drive spindown? I have a 9265-8i coming in next week and would like this option.

I noticed the H710 firmware rom file is about half the size of LSI's

Do you have a DOS flash procedure for the 2208?

 

[mrkrad]

can you please write the MFI to a file and post it? read the megaoem carefully - READ and WRITE are backwards.

PM me if you are uncomfortable posting your MFI string . I think I know how the physical key works.

If you have physical key, dump the MFI with and without it, take a look, and PM me a copy. I'll keep it private of course.

I can show you what to put in the mfi

 

[mobilenvidia]

Do you still have drive spindown? I have a 9265-8i coming in next week and would like this option.

I noticed the H710 firmware rom file is about half the size of LSI's

Do you have a DOS flash procedure for the 2208?

Dell FW is exactly 1/2 a LSI FW
This is due to LSI FW has a secondary built in

You cannot flash the Dell FW with MSM
You have to m0 flash it.

megacli -adpm0flash -f dell_fw.rom
This process takes ages, but not as long as flashing the twice as big LSI FW back the other way.

Now the above worked for me when I first tried it and I had active drive spin down with Dell FW.
But once I got the latest FW from LSI working, I tried it again.
m0 flash did it's thing but then the card would crash at boot time (initialiising drives is as far as it got)

BECAREFUL, don't do it if you can't disable the PCIe slot(s).
If you get the crash as I got, you need to disable the slot card is in, or enable UEFI boot (bypasses BIOS boot)
You have to be able to bypass the LSI boot or you are stuck with a non booting card.

YOU ARE WARNED !!

can you please write the MFI to a file and post it? read the megaoem carefully - READ and WRITE are backwards.

PM me if you are uncomfortable posting your MFI string . I think I know how the physical key works.

If you have physical key, dump the MFI with and without it, take a look, and PM me a copy. I'll keep it private of course.

I can show you what to put in the mfi

My keys are build into the card, no HW key needed.

MegaOEM doesn't work on SAS2208, nor does Megarec.

I've played with the MFC on the M5015, tried to enable JBOD this way but no luck.

StoreCLi has some Key commands
storcli /cx show ASO
storcli /cx set aso key=<key value> preview
storcli /cx set aso key=<key value>
storcli /cx set aso transfertovault
storcli /cx set aso rehostcomplete
storcli /cx set aso deactivatetrialkey

MegaCLi equivalents:
MegaCli -ELF -GetSafeId -aN|-a0,1,2|-aALL
MegaCli -ELF -ControllerFeatures -aN|-a0,1,2|-aALL
MegaCli -ELF -Applykey key <val> [Preview] -aN|-a0,1,2|-aALL
MegaCli -ELF -TransferToVault -aN|-a0,1,2|-aALL
MegaCli -ELF -DeactivateTrialKey -aN|-a0,1,2|-aALL
MegaCli -ELF -ReHostInfo -aN|-a0,1,2|-aALL
MegaCli -ELF -ReHostComplete -aN|-a0,1,2|-aALL

None actually do anything useful with the keys.

 

[BigXor]

Now the above worked for me when I first tried it and I had active drive spin down with Dell FW.
But once I got the latest FW from LSI working, I tried it again.
m0 flash did it's thing but then the card would crash at boot time (initialiising drives is as far as it got)

BECAREFUL, don't do it if you can't disable the PCIe slot(s).
If you get the crash as I got, you need to disable the slot card is in, or enable UEFI boot (bypasses BIOS boot)
You have to be able to bypass the LSI boot or you are stuck with a non booting card.

YOU ARE WARNED !!

Let me see if I understand this:

Use M0 to flash to Dell FW

Flashing back requires M0 flash, can't use MSM?

Why would the card crash and what did you do to rectify the condition?

 

[mobilenvidia]

Let me see if I understand this:

Use M0 to flash to Dell FW

Flashing back requires M0 flash, can't use MSM?

Why would the card crash and what did you do to rectify the condition?

i can't remember if flashing back to LSI needs m0flash, I just did it.

Don't know why it crashed, it worked fine before.
LSI may have put something in the latest FW that stops what we're attempting
I'd use the Dell FW with active driver shutdown, all my SW keys still work.

Once the card crashed, I set the BIOS to not boot or even look at non UEFI cards.
This powers up the controller but doesn't initialise it.
Then use m0flash again to get it back.

If you can boot to DOS or UEFI without the LSI splash screen then you are set.

 

[Dragon]

After my not so sucessful adventure with the LSI9202-16e

What happened? I was just thinking about using it for the next build.

 

[mobilenvidia]

Just experimenting with it now.

I can boot with Mobo in UEFi and Dell's latest FW flashed (and stopping at boot time)
But LSI9266 doesn't show in MSM

I don't know why it won't work now

 

[mobilenvidia]

OK, I got Dell FW loaded and working.
Active spindown is also there.

After numerous different LSI FW versions and then trying Dell = no luck.

I m0flashed 23.4.1-0028 an ancient BETA rom from LSI
This FW breaks the LSI9266 thinking is has no Cache
I then m0flashed the older Dell FW
This fixes the cache and other issues that lack of cache brings.
This seems to work for me.

But do this in UEFI mode, saves a lot of mucking around if it doesn't work.

Now to get the later Dell FW flashed