Fun with an MD1200/MD1220 & SC200/SC220

[Cypher-]

That would have certainly helped in some cases but I'm not sure if I could have used it for everything. When I first powered all of the drives up all of them had green lights on the front, yet TrueNAS could only see 94/96 drives. Since TrueNAS couldn't see it there was no /dev/sdX. Now that I have encled setup this is an example of why it is useful:

0:0:476:0/21    sdw     fault_off       locate_off
0:0:476:0/11    ----    fault_off       locate_off
0:0:476:0/1     sdc     fault_off       locate_off

Now for this particular drive the light is out so it is blatant, but I imagine when I start having these drives just randomly disappear in TrueNAS but the lights are on I assume they will show up the same way. Flashing the error light is easy as pie.

Although thinking about it I suppose you could script the inverse and turn off the lights with sdparm for the drives that are known.

Regardless it was a bunch of fun tinkering with the firmware. Everything is flashed now so I shouldn't have to worry about it any longer.

 

[jabuzzard]

I have never seen a drive have its ready light on but not show up in the OS. That would require either a dead backplane or a SAS cable not plugged in, I would have thought. In the latter case, the whole shelf would be missing (why are you not using multipath???) and would be obvious from the back of the shelf as the link light(s) would be off. I have been spinning a few hundred drives at work in JBOD's for well over a decade now and simply never seen that sort of failure mode.

I have seen lights out on drives that have been repurposed from "enterprise" arrays, but that is quickly fixed with sdparm, and they do show up in the OS.

 

[Cypher-]

As far as I know multipath was removed from TrueNAS scale. Regardless it's added complexity I really shouldn't need in a homelab environment. For the record TrueNAS is either marking these drives as removed or unavailable, it's a mixed bag.

I very clearly have hardware issues somewhere. When I was setting up this new array, any time I had a disk issue I just yanked it and replaced it with a spare. Once I got to around a dozen drives across three shelves (as far as I can remember one shelf has been completely stable) I wanted to dig into a better way of locating the drives so I could track and replace them easier. I wasn't expecting this to be an ongoing problem when I started yanking out suspect drives so I didn't note which ones I was pulling out. Even as I have been messing with it today though, three more drives have disappeared. Two had the light out, one did not.

I've never seen anything like this either but I know the equipment did not come out of the best environment. I've also worked in our datacenters for the last decade and can assure you nothing like this has ever happened on a production system. Unfortunately I don't get to bring new stuff home, only the old stuff headed off to the recycler. The benefit with that is I get it incredibly cheap. This all might as well have been free for what I ended up paying for it.

The drives I have are from 2015 (model HUSMR1619ASS200 if you are curious) and it probably doesn't help that these enclosures I'm trying to setup have been sitting in my basement unpowered for a few years. I've got a ton of stuff to test still ('bad' drives in another server to see if they are really bad, cabling, HBAs, EMMs, hell I have a couple of spare enclosures I can swap if I want try a different backplane) but it'll take time and I'll do it as I can.

Edit: It is just the drives nuking themselves. I've now tested 11 of them in an old R730, five got stuck when the raid controller tried to initialize them and eventually faulted (orange light blinking rapidly). Five are in a weird state where the status light is green but the drives are not picked up in the BIOS, nor does the server log that a drive is being inserted or removed. One was detected by the BIOS and might be OK. I'll hit it with smart tests later. I still have another six to test but considering 10 of 11 turned up blatantly bad I think it's safe to assume the others probably will.

Anyhow I'm not trying to derail this thread with my storage issues, I just wanted to pass along that you can fix the firmware checksum and flash the firmware with sg_write_buffer since I haven't seen anyone mention it.

 

[Shortbus]

So now basically custom firmware is possible. I wonder if it would be possible to patch the default fan speed instead of needing to set a fan speed on a power cycle manually. Maybe a speed/temp table that's patchable. I may take a look, though I know just enough to be dangerous lol.

 

[Mr. GM]

I've got an EqualLogic PS4100 and in it I replaced both controllers with MD1200s. I ran into a bit of trouble with my setup, but I was able to fumble through it and get it working. Here are some additional tips based on my experiences:

• Searching eBay for "MD1200 controller" gives you results, but all of the ones I found were in the $50+ range. I bit the bullet and ordered one...
• While waiting for it to arrive, I had the brilliant idea to search "Dell MD12" instead, and there are (as of this writing) a ton of results for cheap controllers. I paid under $30 for the second controller, and if I'd thought of this before ordering the first one I could have saved a few bucks. Both controllers appear to be identical to my eyes.

This next part was likely due a lack of experience with screen and minicom, but I fumbled for a bit and I was only getting a diamond character and the occasional random letter when trying to connect. All of my previous experience with connecting via serial was using PuTTY on my desktop, but with the disk shelf in my rack in the basement that was a no-go. I set up a Debian VM on my Proxmox server and passed through a USB Serial adapter for this use case. Eventually I found that both screen and minicom work when I set the rate to 38400 instead of the default of 115200. Here's how to do that in minicom:

1. run # minicom -s
2. Go to "Serial port setup" using the arrow keys and Enter
3. Press A to change the "Serial Device" field
4. I changed mine to "/dev/ttyUSB0" as that's the correct device for my USB-Serial adapter, however you may need to use something different if yours isn't mapped to the same location.
5. Press Enter when done
6. While still in this menu, press E to select "Bps/Par/Bits"
7. Press D to select the pre-configured 38400 speed.
8. Press Enter to exit this setup screen.
9. Go to "Save setup as dfl" and press Enter to save this setup as the default.
10. Press Esc to start a session. Minicom will say "Offline" at the bottom of the screen, but that's not actually true; from some Googling apparently minicom just says this if the serial connection doesn't send specific bits, but it'll still work. If the disk shelf is already running then you'll likely just have an underscore. Press Enter and you should see "BlueDress.106.000 >" or something similar.

Finally, I was able to use minicom's runscript to run a very basic script to set the fan speed to 20. My script is saved as "setfanspeed.script" and it's just the following two lines:

send set_speed 20
exit

Running the script in minicom is simple. It's "minicom -S [script location and name]" so for my case it's "minicom -S setfanspeed.script" and it works. From there I set up a stupid-simple bash script to run this command and then I set up the VM to run the bash script at boot. There's probably a better way to do this than devoting an entire VM to connecting to the shelf's management plane and running a single command, but this works well enough for a homelab setup.

 

[turtleravioli]

I am currently running a MD1200 with a Dell R730XD and a PERC H830. Two EMMs with redundant path and TrueNAS as the OS.

Everything seems to work fine, but I see this error message randomly in the logs of TrueNAS:
May 7 03:07:46 kernel: ses 13:0:0:0: Power-on or device reset occurred

Also, about every 5 minutes there seems to be a reset of some kind within the MD1200, as if I have and LEDs set using ses, they will clear out and the amber fault light will come on for about 1-2 seconds then go back to normal. Has anyone figured out fixes for these issues?

 

[turtleravioli]

Decompiled using Ghidra. This might help us track down the front shelf periodic Amber LED issue and periodic SES Reset issue. Also, we could possibly permanently set fan speeds to be lower.