m1015 - feature key, firmware, and SED troubles

[craze]

About 18 months ago I picked up an m1015 based on one of the articles here speaking rather highly of them(and I've used several other serveraid cards in the past with good results), along with 4 3tb sata drives with the intent of setting them up in raid10. The article on crossflashing suggested switching from m1015 mode to IR mode was a good choice for my application, so I gave it a go.

Turns out that wasn't such a great idea after all. I successfully flashed the card into IR mode(thanks to the very well written article), however on my new raid10 array I was greeted with speeds lower than that of a single drive configured as JBOD, and I also lost out on all of the high end features I'd come to expect from the SR series of cards over the years(I've been using these since back in the days of SCSI). So that was a bust, and I proceeded to plop the stock m1015 firmware back on the card and test it out in windows. That gave me speeds much more in line with what I was expecting, with sequential read speeds in excess of 400mB/s, and write speeds not far behind that. Despite the firmware being out of date, I left it as is and all was well for a time.

Fast forward a year later and I picked up 4 new 3tb seagate enterprise SED sas drives, as my crappy consumer models were dropping out of the array constantly and overflowing their smart logs with firmware errors, eventually leading to me losing the array along with all my data, while waiting on my new sas cables to arrive from china(ebay, what can you do) so that I could use the $1200 in shiny enterprise kit that was sitting on my dresser, silently mocking me. Bloody cables showed up the next day, to add insult to injury.

At the time I also picked up an m1000 feature key so that I could make use of the SED functionality of my new drives, however attempting to actually use said feature key has proven to be an adventure. No matter what I do, if I hook this thing up to the card, something breaks, what that something is depends entirely on which firmware build I happen to have on the card at the time. On the stock 4.30.00(dated 10-26-11) the card will post and complain about not having a valid sas address programmed(usually a very bad sign), then pause, then print the exact same message again for some reason, then finish posting as if nothing was wrong, except all configurations are now detected as foreign. I can import the configuration and get into windows and pull up MSM, or hop on the web bios, but there is no sign of it detecting the feature key or any of the functions that it is supposed to unlock(in my case, access to lsi safestore key management is the only feature I care about). If I load any firmware newer than what came on the card, things get worse.

It's been 6 months since I've actively played with this, so most of this is from memory(I did however try popping the key back on earlier today to make sure it hadn't suddenly decided to start working on me for no reason, so that specific error is fresh in my mind). On newer firmware it will throw all manor of errors and then fail to complete post. I believe when I tried it on the latest(as of 6 months ago) lsi firmware, the error it gave me had something to do with a missing 'iButton'? Googling this revealed absolutely nothing useful, or even remotely relevant for that matter. I also seem to recall it not wanting to boot into windows properly on the newer firmwares(bsod's, I've got the latest drivers straight from lsi to boot), despite the card appearing to be fully functional, with the feature key removed obviously. One odd quirk I will point out, is that through all of this that I've done to the card, for some strange reason it claims to have raid5 support unlocked via license key(I'm guessing here), and is available for use. I don't actually need raid5 for anything, but I thought it odd that it had it available, as it is supposedly one of the features enabled by the m1000.

So, I'm really scratching my head on this one, did I inadvertently bugger something up when I crossflashed? Is my m1000 toast? Do I need a new card? Honestly I'm not super married to this thing at the moment, I have a new server box I just setup and the serveraid 8s I have installed in it is too old to be recognized by ESXI, and needs to be replaced. The m1015 is on the official ESXI hardware compatibility list, and the regular old raid functions still work just fine, so if it came down to it I could always just replace the card and dump this guy into my server(otherwise I was planning on just purchasing a second m1015 for it).

I'm not quite sure where to go from here, I'd kind of stuck the whole thing on the back burner after I couldn't get the firmware to accept the feature key properly. Another small conundrum I've run into, is that after consulting the lsi megaraid sas manual, it appears that enabling drive security features is done at the VD level, during creation, and so I would have to delete and recreate my raid array to take advantage of my SEDs(and thus have to backup and restore all of my data, something I'm not quite equipped to do at the moment; that's what the server was supposed to be for until it got temperamental about me trying to get away with using an older raid card...), is there anyone who's actually played with SEDs on one of these(or other comparable lsi based card) and can clarify that part for me?

Anybody have any ideas whats up with my card? I just want my SEDs working properly, as running them in the factory 'unlocked' mode rather defeats their purpose... Any help would be greatly appreciated.

 

[craze]

Anyone? Not even a confirmation on whether an m1015+m1000 will successfully integrate with SEDs(IBM redbooks says it should)? Like I said, my ESXI box needs a new raid card, and I was considering an m1015 for the job, Id be perfectly open to giving up on fixing this and just buying a whole new card and feature key for my main rig and plopping the old one in the server, if someone could vouch for it working properly... or perhaps recommend something else that would work instead?

 

[gea]

If you need Raid-5 and self-encrypting drives (have not tested myself) then a IBM 1015 is not a good controller for you.

The IBM is good if you need a simple Raid-1 controller but only if you can accept the risks due the write whole problem as you do not have a cache/BBU on the controller.

The IBM 1015 is perfect if you need a raidless HBA as you can crossflash to a LSI 9211-IT mode, one of the best of all controller for software raid, especially with ZFS. With a feature key it is a slow and not very good hardware raid-5 controller.

What you can think about with this controller (if your mainboard supports vt-d and has RAM) is a local virtualized SAN like my napp-in-one with a shared NFS datastore for ESXi. Much more features, safer and faster than a local ESXi datastore.

 

[craze]

Thanks for the reply, however I'm not running raid5 in my current setup, nor am I intending to do so on the ESXI machine.

I've got 4 3tb sas(nearline) SEDs hooked up in raid10 presently. The ESXI machine has 8 proper(non-nearline, 15krpm) 72gb sas drives also setup in raid10, however it doesn't like the serveraid 8s controller that I have installed right now, hence my need for a replacement(I've got it configured with a local datastore on an old sata hdd at the moment).

 

[gea]

If you only need the feature key for SED, you should stay with the original firmware as I would not expect that it works with another firmware.

 

[craze]

As I mentioned, I've been using the original firmware. I have no need for an HBA(thus ruling out IT mode), and IR mode delivered extremely lackluster performance vs m1015 mode. The issue I'm having is my card refuses to accept my feature key without erroring out and failing to post properly, yet seems to have raid5 already unlocked without the key attached(SED functionality is still locked out however).

 

[craze]

After doing some digging, it appears that the lsi 9240-8i does not support SED functionality under any configuration(software license, feature key, or otherwise), whilst supporting raid5/50 out of the box. However all of the IBM documentation for both the m1015 and the m1000 clearly state that card is SED capable when combined with the m1000, which grants access to LSI SafeStore(the necessary advanced software feature for interacting with SEDs). My conclusion based on this, is that the SED functionality is something IBM worked in on their own, while locking out the raid5/50 support.

The next cheapest option I could find was the lsi 9260-8i combined with the dedicated lsi00245 feature key, which specifically enables access to safestore, and thus SED capability. The downside, is the cheapest I can find one of these cards is for about $200 on ebay, and the feature key itself is roughly $75. That's over twice the cost for a bunch of extra features that I don't need, and a job that theoretically should be able to be handled by the m1015 with an m1000(which I already own).

While digging on ebay, one discouraging trend I noticed was folks selling legitimate 9240-8i's and passing them off as m1015 equivalents. While that is more or less true, I could certainly see that posing an issue if I tried to attach an IBM specific feature key onto an LSI card. My guess at this point is I'm sitting on a real 9240, and not an m1015; it would explain why I've got raid5/50 unlocked...

Its got the IBM bios on it(and I still have raid5/50 available), as that's what I loaded after deciding crossflashing wasn't suitable to my application, so simply looking at the callout during post wont cut it... Is there any clear, definitive way to ID an IBM m1015 vs an LSI 9240-8i on the card itself?

I'll admit, a lot of this is conjecture, based on what information I could find, but that's the best I've got at the moment. Any help would be appreciated.

EDIT: Just checked the card, and it is in fact bearing the IBM 46M0861 FRU sticker, which the true LSI cards lack, so that rules that out. Pretty much back at square one now...

 

[craze]

So basically I gave up, and bought an m5015+m5000(with bbu). I would have preferred a true lsi 9260-8i, but the price markup was just too extreme vs going with a branded model. All in all, it only ended up costing me $67 more than a new m1015+m1000 would have. At least with this card, safestore functionality is present on both the official and IBM models(with feature key attached), so there's absolutely no question about the card supporting it or not.

I really would have liked to figure out what was wrong with my existing card/key, but I guess it just wasn't meant to be. New kit should be here next week, at least this time I've ordered it all at once, so if I start running into bizarre problems like I had with my m1015 I can at least return the thing and try something else...

 

[craze]

So my m5015+m5000 combo has some kind of weird issue, I guess it doesn't like something in my desktop rig(I made another thread about it), and I'm not even using the write cache(it seems to exacerbate the problem). I do have it working with the SED's correctly, and lsi safestore is properly unlocked by the m5000, however I've decided the card would serve better in my server than in my desktop, and picked up yet another m1015 and m1000(under the assumption that at least one of the two had to be bad to produce the issues I was having). Plus the audible alarm is great for esxi, as I have no way to detect errors without rebooting the machine and watching the cards POST info. The write cache and bbu would also be more helpful there, I might even switch to raid50 for a little extra capacity on my datastore because of it(plus the things not on a UPS, unlike my desktop, so I could kill the on-disk cache for a little peace of mind).

The point I'm trying to make, is my new m1015 just showed up, appears to work all well and good despite having ancient firmware on it, but as soon as I strap my old m1000 on it does the exact same thing as described in the OP(complains about not having a sas address programmed), and then halts the system. Now this has me leaning toward my m1000 being a dud, which is why I ordered a replacement, however I was unable to find a US based seller of those and had to get one from china... and it wont be in for another 2-3 weeks. The card also has raid5 enabled via software license just like my old one does, which I find peculiar. Just to keep things clear, I have no intention of running raid5 on this card, and I am perfectly aware that the sas2008 chip struggles at parity and has no business with parity based raid levels. I have 4 drives in raid10, and just need my m1000 to be detected so I can use the safestore local KMS, like on my m5015(hopefully with less random array drops this time around; my old m1015 went a year without any problems caused by the card itself, yet I'm on my third m5015 and these things cant seem to go a week without flipping out(plus the bbu is bad, again), I just pray that it holds up better in a proper server board than it does in my desktop).

Anyone out there actually using a m1015 with m1000 feature key that can help? Did someone 'hack' the raid5 support in for free(like you can do with the raid6/60 support on the m5015s by flashing the sbr over to the lsi one) and its conflicting with my ability to run the feature key(and is it something that can be undone, though it is detecting itself during post as an m1015 and not a pure lsi card)? Any ideas/suggestions/whatever? I've got 2+ weeks to kill before my new(seller reports it as being actually new, as in IBM retail packaging, never opened, etc.) feature key finds its way over here via boat, or whatever else they happen to be using that takes 3-4 weeks to get stateside...

 

[craze]

New m1000 key just arrived, still sealed in its original IBM packaging. Looks like my old one was toast from the get go, that's the price you pay for buying used kit from china I guess...

Anyway I tried to install it, and its actually recognizing it as a feature key! ...however its thinking its the wrong feature key for the card. Instead of freaking out and halting the system during post with key attached, it now throws this error:

Incompatible secondary iButton present!
Please insert the correct iButton and restart the system.
Press any key to continue but OEM specific features will not be upgraded!

Tried updating the firmware on the card(as it too was ~3 years out of date) but that did nothing, still get the same error.

Anyone got any ideas? Has the card been modified in some way that's causing it to reject official IBM feature keys(hacking in raid5)? Should I just contact the ebay seller I bought the card from instead?

 

[poto]

It is possible that your raid controller has oem-specific firmware which precludes the use of feature key.

I just went through a similar situation with used lsi 9266-8i that would not accept known good advanced software key. It functioned fine otherwise, but responded with the "incompatible secondary ibutton..." message upon boot. It also had cachevault listed as advanced software feature regardless of whether flashcap was attached or not - very odd.

I was able to flash current firmware with no problem, but could not get fastpath to enable, or cachecade key recognized. LSI support was very helpful, and confirmed via the serial number that this was an oem for Hitachi, not compatible w/advanced software key, even though it looked and was stickered exactly like a "genuine" unit. Maybe you could make similar serial # inquiry via IBM?

 

[craze]

Hm, interesting. I'm contacting the ebay seller now to see what they have to say on the matter. It does have raid5 unlocked out of the box, which would support your idea. I'm not sure what IBM's policy on this sort of thing is contact wise, but it might be a moot point; from where I'm sitting right now, it looks like I wont be able to use this card either...

I guess my next step(if I wanted a 100% guarantee of success) would be to buy a card+key bundle(and return this one), but at present there is only one of those on all of ebay, and its over twice the price of a whole new card+key purchased separately. I'll have to see if the seller knows anything about this first...

 

[craze]

Ebay seller was super cool and tracked down and tested a replacement for compatibility with the m1000(poto was probably right about the OEM thing), and shipped it out to me(next day air, no less) along with letting me keep the old one(probably going to get the 9211it firmware and get used with vt-d in my esxi machine). Got that all up and running, but I've been having nonstop issues with my array on now six different controllers that I've tried('unexpected sense' errors, followed by the disks failing out, either both disks in one span, or all disks in the array, all at the exact same point in time). It got to the point where my desktop couldn't stay online for more than about 5 minutes before the array tanked on me. So I got tired of that, yanked everything out, and stuck it in my server for testing. Was able to reproduce the issue, and ended up backing all my data up off the array and shooting for a slow-init to try and clear things up. That actually caused the same thing to happen once it got about half/just shy of half way through(about the same percentage of the drive I have filled in my filesystem), under raid10, 5, and 0. Flashed over to a 9211it to update the drives firmware(didn't particularly trust jbod to handle a firmware upload), and that didn't do much of anything for stability once I flashed back. I ended up passing the drives through as jbod to do some tests with seatools, all of which said the drives were perfectly fine(I called BS here). At that point I was kind of out of ideas, and decided to run an 'overwrite erase' from the advanced tools section, figuring it would do some kind of thorough format and might fix any potential bad sectors I might have(which I found kinda hard to believe would be happening on all 4 drives at once, in the same place, plus the raid card ought to be able to handle that anyway). I'll mention that the controller was operating with drive security engaged and locked at this time.

Moral of the story, do not do this, it bricked all four of my drives($1300 in kit [metaphorically] up in smoke). After spending an extended period of time on the phone with a seagate technician in a lab somewhere in california, he too managed to brick a bunch of drives while testing a similar setup on an lsi based controller. Turns out there's a firmware error with builds 0003(what came on the drives when I bought them) and 0004(the latest build I got off the website); they ended up sending me build 0002 which did not have this issue when they tested it, and apparently does not drop drives out of raid randomly either(couldn't actually test that part yet though).

Unfortunately it was too late for my drives, as they refused the firmware upload(technically they 'accepted' it, but it completed super fast and they didn't actually do anything with it; they're stuck on 0004), and even the seagate 'factory reset'(labeled 'sed crypto erase' under seatools), which basically does an ISE without requiring unlocking, and then blows off any secure configuration you have and resets the disk to unlocked mode(and requires entering of a drive specific key printed on the disks label) couldn't get the drives working again(it succeeds, but doesn't actually do anything). The drives were completely unusable under m1015 mode by this point, and I had to flash back to IT mode just to get access to the drives under seatools(where every test now failed instantly with the exception of the crypto reset, which did nothing, and firmware upload, which said it completed but actually did nothing as well). Under m1015 mode, the controller insisted the drives were locked with an unknown security key, even after I reset them, and cleared then re-entered my security key and passphrase(and that was only from webbios, under MSM it just flat said they were incompatible). The whole time it kept spitting out intermittent errors about the disks either being unsupported+unable to access, or having a 'security subsystem problem'. The whole thing was a giant mess, and I've got to RMA my drives now.

I'll report back once my replacement drives get here and I can hit them with a slow init to make sure that this firmware does the trick(don't think I'm going to be doing an 'overwrite erase' again anytime soon though). I can upload it here for anyone else who wants it if it ends up fixing the problems.