I got a decommissioned Netapp FAS3250 and some DS4246 shelves, and I'm not a Netapp fanboy so I thought, hey, let's try to get Linux installed on it?
The FAS3250 (and I assume all of the FAS3xxx models?) are interesting. There's a little 2GB internal USB drive in there. It's an "embedded USB" device, which just means its a USB flash drive (a high quality one I guess... SLC so it should be able to last years unlike a cheapo), and it plugs directly into a motherboard 10-pin USB header, not USB-A or anything like that. So that's interesting.
To start out, I took out that module and plugged it directly into my workstation in an empty USB 2.0 header. It shows up fine. It's FAT (not FAT32, just plain FAT) so you should have no problem making an image of it, copying the files, whatever. I used ImageUSB to make a backup of it in case I want to restore.
With my backup done, I then wiped it, formatted as FAT32, and copied Unraid over to it and used the "make bootable" script. But then there's a couple things you'll need to do since this is a headless system (no video, no keyboard). In the /config directory for Unraid you just copied over, edit the ident.cfg and add whatever options for NAME="netapp01" or whatever, set your timeZone if you want (or you can do that part later). Also it's very important you edit the "network.cfg" file and put in the info you'll use to connect when it boots into Unraid. Here's what I used:
# Generated network settings
USE_DHCP="no"
IPADDR="10.1.1.62"
NETMASK="255.255.255.0"
GATEWAY="10.1.1.1"
DNS_SERVER1="10.1.1.1"
BONDING="yes"
BRIDGING="yes"
Helpful hint, if you don't want to use your embedded USB module and want to keep that safe, there is a totally functional USB port on the back (above the RJ45 serial port) that you can boot from. Use any USB drive you want... I used a 2GB drive so I could make sure it would fit onto the embedded 2GB module if I want to go that route and it would all fit. Just remember that Unraid licensing is tied to your USB drive's serial # so you might try it out with an external drive and then if you're happy, copy your configs to the embedded drive and license from that for a nice "hidden" solution where you're not dealing with a dongle dangling off the back.
Now for some fun stuff... you'll need to hook a computer up to the serial port of the Netapp because you need to change the BIOS settings to tell it to boot off the USB, and configure a few other things.
The serial cable is your basic RJ45 to DB9 cable that comes with Cisco equipment, and hopefully you have some, or they're super easy to get if needed. Anyone who's been doing computer stuff for a while probably has a few of them (or a drawer full)
Get it attached to your computer and connect to it (I was using Putty on Windows) with 9600-N-1. Yeah... 9600 is going to be slow, but you're just doing a few things.
For bonus points ( you can do this later, but it'll be helpful during the initial setup in case your networking is weird), you can add settings so Unraid will work with the serial console after it boots. I didn't do this when setting up but since I wasn't sure what network interface it was going to find, I basically just plugged in everything I could see on the back of the Netapp into my switch. Turns out the primary connection it'll use for eth0 (for me anyway) was the NIC with the open wrench (I think that's the em0 port in Netapp lingo), but at least one of the other NICs was also coming up... I don't recall off hand but I think it may have been either the e0a or e0b ports. I'm not sure of the relationship between how Netapp labels the ports and what actually gets detected in Unraid, so clearly I have more figuring to do.
That being said, plug at least your em0 "wrench" port into your switch, but you can add a few more config lines to your Unraid USB to help in case that doesn't work for you.
In the /config/go file, append these lines to the end:
#setup serial console
sed -i -e "s/^#ttyS0/ttyS0/" /etc/securetty
sed -i -e "/^#s1/ i s1:12345:respawn:/sbin/agetty -L ttyS0 115200 vt102" /etc/inittab
init q
And in your /syslinux/syslinux.cfg file, you'll add a few things
- In the main section, below the line that says "timeout 50", add a new line:
SERIAL 0 115200
- In the section "label UNRAID OS" there's a line that has "append initrd=/bzroot"
Change that line by adding some stuff to the end... the end result will look like:
append initrd=/bzroot console=ttyS0,115200 console=tty0
At least on my FAS3250, the serial port is ttyS0 and that's probably a safe bet for anything similar. There does seem to be a UART for a ttyS1 detected but as far as I know there's no connectors for it anywhere, so ttyS0 is your guy.
Note that it's 115200 baud so when Unraid actually starts up, you'd have to change your console settings to 115200 instead of the 9600 that the BIOS is using. It'd be nice if the BIOS worked at 115200 as well... I don't know if there's a way to set that but I'll probably try some things later to see what I can do.
Once all attached and your USB drive is plugged into the Netapp, go ahead and fire it up. It'll take a little bit before the serial port starts sending anything, and in the meantime I was just hitting a few keys, and at this point I'm not sure which particular key will get you into the BIOS setup screen. I was hitting Escape, Delete, F2, F10, F12, over and over, and one of those worked because it brought up (slowly...remember, 9600 baud) a Phoenix BIOS setup screen. So that means it's probably DEL to get into it, but I haven't tested that specifically to see. I'll test that again as soon as my Unraid finished the initial parity on my drives.
Use your arrow keys to navigate around just like any other Phoenix BIOS. There are 2 things you really need to change:
1) Set the USB to removable mode instead of emulating a hard drive
2) Tell it NOT to boot from the internal flash - if you don't do that, there's an internal flash somewhere on the controller that boots into that netapp LOADER stuff, and when it doesn't see the FAT partition on the USB with the Netapp files, it'll just sit there at the LOADER prompt waiting for you to do something. Changing the BIOS to skip booting from that internal flash will let it boot directly from USB which is what we want.
I don't remember if I changed anything else in there... It's already set NOT to halt on error and basically run headless since that's what Netapp needs, so I think changing those two items is all you need. Save your changes and reboot.
When it boots off the USB, you'll see the BIOS boot stuff in your serial console but then Unraid will show some odd things as it boots. It tells you what files it's loading but it adds a line feed after each character so it's kind of messy, but at least you'll see what its doing. If you added the SERIAL stuff to the Unraid configs, you can switch to 115200 baud and once Unraid is done booting, you should get a login prompt.
In my case, I had a continuous ping going to the IP address I set, and eventually, just a little past where I thought "uh oh, it didn't work", it started responding.
From there, hey, you just go to the Unraid web interface, login, setup your trial key or whatever, and that's it, you're good to go.
There are a few things that I did afterwards that may be useful for someone to know.
- If you're using a Netapp drive shelf and it has Netapp SAS drives, they're going to be 520-byte formatted. Not a problem... we're running Linux now. Setup your Unraid so you can SSH into it and install sg3_utils:
wget https://sg.danny.cz/sg/p/sg3_utils-1.48.tgz
installpkg sg3_utils-1.48.tgz
Use lsblk to show drive list/sizes. 520-byte drives show space as 0B. In my case my shelf had a mix of 4TB SAS and 3TB SATA. The sata drives were all 512-byte so they showed up fine, but the SAS drives with 520, well, those all showed up as 0B but I already knew they'd be like that and was ready.
I used sg_map to correlate /dev/sd* with /dev/sg* to create the list of which 520-byte drives to reformat and then ran this command to kick off the reformat of all 12 of my SAS drives to 512 bytes. Usual warnings about being careful and knowing what you're doing... this will wipe the drives, and it will take a while so sit back and relax. But at least it's doing all of them at the same time. Modify the list of drive #'s for your own needs. That list from 5...25 are mine, do NOT copy and paste that part and expect good things, I'm just showing you an example of adding a custom list. If you needed to do all of your drives from 2-25 you can probably do it easier, I just had a mix of them scattered around (fyi, I think sg1 is the enclosure, sg0 might have been the USB drive):
for sg in 5 6 7 11 12 13 16 17 18 23 24 25; do (sg_format --quick --early --format --size=512 /dev/sg$sg ); done
If you want to check the progress of the reformatting along the way (and yeah, you'll want to know when it's done), you can use this to spit out the % done for each drive. If you started them all at once, they should finish all at once, but this way you'll know for sure:
for sg in 5 6 7 11 12 13 16 17 18 23 24 25; do (echo -n "/dev/sg$sg : ";sg_requests -p /dev/sg$sg ); done
Are you interested in a list of some vital drive stats once the reformat is complete? Here's a handy little one-liner I whipped up so I could check various details on the drives:
for sg in 2 3 4 8 9 10 14 15 19 20 21 22; do (echo " ---- /dev/sg$sg ----";smartctl --all /dev/sg$sg | grep 'Vendor:\|Product\|Revision\|Capacity\|Logical block size\|Serial\|SMART Health\|Accumulated power'); done
That example is from the other SATA drives, so those #'s are different, but you get the idea. I should mention that your /dev/sd* and /dev/sg* are likely to change between reboots so don't expect to make a list of which drives are which and it'll always be the same. That's why we have wwids or go by serial #'s if we want to make sure. Like, just now when I checked, everything had changed, the enclosure is now sg25, etc.
If you want any help on setting Unraid up for serial console access, I used this site as a reference for my setup. "dmesg" didn't say anything about my ttyS0 but I just gave it a shot anyway and that's the right one:
Setting up Unraid for serial console
What else... a few odds and ends like the FAS3250 has dual L5410 CPUs in there. Not exactly screaming, but it gets the job done. It has 16GB of DDR2 in 4 sockets, and then there's another 4GB setup as a pair of 2GB modules that I think belong to the honkin' big battery/supercap in the controller node. Unraid sees 20 GB but then says 16 GB is usable. I've got a ton of DDR2 modules lying around... they're ECC though and I didn't look closely enough to see if the Netapp is using ECC or regular modules, but I'm curious if I can bump this up to the 64GB max capacity it seems to think it can handle.
Get your networking all setup... bonds, which ports to include. I'm still figuring some of that out. Some of the ports only show as being 100Mb which isn't great, but at least one of them connected at 1Gb, but the other one like that isn't linking at all but maybe that's because I configured that bond as active/backup. I'm new to Unraid.
The initial parity is going slow, but then again, the parity drives are writing at 40-42 MB/s which probably is about what a SAS spinning hard drive can do. If I swapped these out for SSDs I would expect it to be much faster. We'll see how it performs in read/write tests after the init is complete.
Also, my particular FAS3250 has some additional PCI cards in there, in the lower module (a pair of added 4-port SAS controllers) and there's also another dual 10Gb SFP+ card installed in the main module, but I'm not sure if Unraid is seeing those or not. When I look at the devices that Unraid detected I see 5 different "PMC 8001" devices so that might be correct (I think a "single" PMC 8001 is 2 SAS ports, and I have a total of 10 including the 2 onboard the main controller itself). I just haven't attached anything to the other ports to check. I do have more drive shelves I might hook up just to see if they appear. For the extra 10Gb network ports, I'm still confused about the network interfaces that are on the controller itself.
As mentioned, I'm not a Netapp guy, so I'm still fuzzy on what ports are used for what. I know there's a pair of HA ports (c0a and c0b) if you're using dual controllers, and those, I think, are showing up, but when I tried connecting them to my switch, I wasn't getting link indicators. And it's weird that the em0 port seems to only link at 100Mb, but maybe that's normal for the Netapp? Then there's a couple of Qlogic FC ports, so ignore those. Anyway, point is, I don't know which of the ports on back correspond to what Unraid is seeing, and I kind of need to sit down at my computer out there while I unplug a cable at a time and see what Unraid says is going on. Having that serial console to the box is going to be very helpful in that case since I'll probably lose my network connection to it here and there while testing.
So... that's about it for now. Long post, but I was unable to find anyone else who'd done something like this on a Netapp. A few have mentioned something similar but no instructions or guides on what they did so I was flying blind and I thought this would be helpful to anyone else who felt like doing something a little crazy with an old Netapp. I've got another filer, and FAS2240-4 that, to me, looks like a DS4243 or DS4246 but the top IOM module is replaced with a controller. It may be the same, or totally different... I just pulled that module out to look at it closer on my desk so we'll see. That one could be fun since the drives and controller are all in one chassis so it would take up less space overall. I don't know what else differentiates it with the FAS3250... different CPU/memory, I expect, at a minimum.
The FAS3250 (and I assume all of the FAS3xxx models?) are interesting. There's a little 2GB internal USB drive in there. It's an "embedded USB" device, which just means its a USB flash drive (a high quality one I guess... SLC so it should be able to last years unlike a cheapo), and it plugs directly into a motherboard 10-pin USB header, not USB-A or anything like that. So that's interesting.
To start out, I took out that module and plugged it directly into my workstation in an empty USB 2.0 header. It shows up fine. It's FAT (not FAT32, just plain FAT) so you should have no problem making an image of it, copying the files, whatever. I used ImageUSB to make a backup of it in case I want to restore.
With my backup done, I then wiped it, formatted as FAT32, and copied Unraid over to it and used the "make bootable" script. But then there's a couple things you'll need to do since this is a headless system (no video, no keyboard). In the /config directory for Unraid you just copied over, edit the ident.cfg and add whatever options for NAME="netapp01" or whatever, set your timeZone if you want (or you can do that part later). Also it's very important you edit the "network.cfg" file and put in the info you'll use to connect when it boots into Unraid. Here's what I used:
# Generated network settings
USE_DHCP="no"
IPADDR="10.1.1.62"
NETMASK="255.255.255.0"
GATEWAY="10.1.1.1"
DNS_SERVER1="10.1.1.1"
BONDING="yes"
BRIDGING="yes"
Helpful hint, if you don't want to use your embedded USB module and want to keep that safe, there is a totally functional USB port on the back (above the RJ45 serial port) that you can boot from. Use any USB drive you want... I used a 2GB drive so I could make sure it would fit onto the embedded 2GB module if I want to go that route and it would all fit. Just remember that Unraid licensing is tied to your USB drive's serial # so you might try it out with an external drive and then if you're happy, copy your configs to the embedded drive and license from that for a nice "hidden" solution where you're not dealing with a dongle dangling off the back.
Now for some fun stuff... you'll need to hook a computer up to the serial port of the Netapp because you need to change the BIOS settings to tell it to boot off the USB, and configure a few other things.
The serial cable is your basic RJ45 to DB9 cable that comes with Cisco equipment, and hopefully you have some, or they're super easy to get if needed. Anyone who's been doing computer stuff for a while probably has a few of them (or a drawer full)
Get it attached to your computer and connect to it (I was using Putty on Windows) with 9600-N-1. Yeah... 9600 is going to be slow, but you're just doing a few things.
For bonus points ( you can do this later, but it'll be helpful during the initial setup in case your networking is weird), you can add settings so Unraid will work with the serial console after it boots. I didn't do this when setting up but since I wasn't sure what network interface it was going to find, I basically just plugged in everything I could see on the back of the Netapp into my switch. Turns out the primary connection it'll use for eth0 (for me anyway) was the NIC with the open wrench (I think that's the em0 port in Netapp lingo), but at least one of the other NICs was also coming up... I don't recall off hand but I think it may have been either the e0a or e0b ports. I'm not sure of the relationship between how Netapp labels the ports and what actually gets detected in Unraid, so clearly I have more figuring to do.
That being said, plug at least your em0 "wrench" port into your switch, but you can add a few more config lines to your Unraid USB to help in case that doesn't work for you.
In the /config/go file, append these lines to the end:
#setup serial console
sed -i -e "s/^#ttyS0/ttyS0/" /etc/securetty
sed -i -e "/^#s1/ i s1:12345:respawn:/sbin/agetty -L ttyS0 115200 vt102" /etc/inittab
init q
And in your /syslinux/syslinux.cfg file, you'll add a few things
- In the main section, below the line that says "timeout 50", add a new line:
SERIAL 0 115200
- In the section "label UNRAID OS" there's a line that has "append initrd=/bzroot"
Change that line by adding some stuff to the end... the end result will look like:
append initrd=/bzroot console=ttyS0,115200 console=tty0
At least on my FAS3250, the serial port is ttyS0 and that's probably a safe bet for anything similar. There does seem to be a UART for a ttyS1 detected but as far as I know there's no connectors for it anywhere, so ttyS0 is your guy.
Note that it's 115200 baud so when Unraid actually starts up, you'd have to change your console settings to 115200 instead of the 9600 that the BIOS is using. It'd be nice if the BIOS worked at 115200 as well... I don't know if there's a way to set that but I'll probably try some things later to see what I can do.
Once all attached and your USB drive is plugged into the Netapp, go ahead and fire it up. It'll take a little bit before the serial port starts sending anything, and in the meantime I was just hitting a few keys, and at this point I'm not sure which particular key will get you into the BIOS setup screen. I was hitting Escape, Delete, F2, F10, F12, over and over, and one of those worked because it brought up (slowly...remember, 9600 baud) a Phoenix BIOS setup screen. So that means it's probably DEL to get into it, but I haven't tested that specifically to see. I'll test that again as soon as my Unraid finished the initial parity on my drives.
Use your arrow keys to navigate around just like any other Phoenix BIOS. There are 2 things you really need to change:
1) Set the USB to removable mode instead of emulating a hard drive
2) Tell it NOT to boot from the internal flash - if you don't do that, there's an internal flash somewhere on the controller that boots into that netapp LOADER stuff, and when it doesn't see the FAT partition on the USB with the Netapp files, it'll just sit there at the LOADER prompt waiting for you to do something. Changing the BIOS to skip booting from that internal flash will let it boot directly from USB which is what we want.
I don't remember if I changed anything else in there... It's already set NOT to halt on error and basically run headless since that's what Netapp needs, so I think changing those two items is all you need. Save your changes and reboot.
When it boots off the USB, you'll see the BIOS boot stuff in your serial console but then Unraid will show some odd things as it boots. It tells you what files it's loading but it adds a line feed after each character so it's kind of messy, but at least you'll see what its doing. If you added the SERIAL stuff to the Unraid configs, you can switch to 115200 baud and once Unraid is done booting, you should get a login prompt.
In my case, I had a continuous ping going to the IP address I set, and eventually, just a little past where I thought "uh oh, it didn't work", it started responding.
From there, hey, you just go to the Unraid web interface, login, setup your trial key or whatever, and that's it, you're good to go.
There are a few things that I did afterwards that may be useful for someone to know.
- If you're using a Netapp drive shelf and it has Netapp SAS drives, they're going to be 520-byte formatted. Not a problem... we're running Linux now. Setup your Unraid so you can SSH into it and install sg3_utils:
wget https://sg.danny.cz/sg/p/sg3_utils-1.48.tgz
installpkg sg3_utils-1.48.tgz
Use lsblk to show drive list/sizes. 520-byte drives show space as 0B. In my case my shelf had a mix of 4TB SAS and 3TB SATA. The sata drives were all 512-byte so they showed up fine, but the SAS drives with 520, well, those all showed up as 0B but I already knew they'd be like that and was ready.
I used sg_map to correlate /dev/sd* with /dev/sg* to create the list of which 520-byte drives to reformat and then ran this command to kick off the reformat of all 12 of my SAS drives to 512 bytes. Usual warnings about being careful and knowing what you're doing... this will wipe the drives, and it will take a while so sit back and relax. But at least it's doing all of them at the same time. Modify the list of drive #'s for your own needs. That list from 5...25 are mine, do NOT copy and paste that part and expect good things, I'm just showing you an example of adding a custom list. If you needed to do all of your drives from 2-25 you can probably do it easier, I just had a mix of them scattered around (fyi, I think sg1 is the enclosure, sg0 might have been the USB drive):
for sg in 5 6 7 11 12 13 16 17 18 23 24 25; do (sg_format --quick --early --format --size=512 /dev/sg$sg ); done
If you want to check the progress of the reformatting along the way (and yeah, you'll want to know when it's done), you can use this to spit out the % done for each drive. If you started them all at once, they should finish all at once, but this way you'll know for sure:
for sg in 5 6 7 11 12 13 16 17 18 23 24 25; do (echo -n "/dev/sg$sg : ";sg_requests -p /dev/sg$sg ); done
Are you interested in a list of some vital drive stats once the reformat is complete? Here's a handy little one-liner I whipped up so I could check various details on the drives:
for sg in 2 3 4 8 9 10 14 15 19 20 21 22; do (echo " ---- /dev/sg$sg ----";smartctl --all /dev/sg$sg | grep 'Vendor:\|Product\|Revision\|Capacity\|Logical block size\|Serial\|SMART Health\|Accumulated power'); done
That example is from the other SATA drives, so those #'s are different, but you get the idea. I should mention that your /dev/sd* and /dev/sg* are likely to change between reboots so don't expect to make a list of which drives are which and it'll always be the same. That's why we have wwids or go by serial #'s if we want to make sure.
Like, just now when I checked, everything had changed, the enclosure is now sg25, etc.If you want any help on setting Unraid up for serial console access, I used this site as a reference for my setup. "dmesg" didn't say anything about my ttyS0 but I just gave it a shot anyway and that's the right one:
Setting up Unraid for serial console
What else... a few odds and ends like the FAS3250 has dual L5410 CPUs in there. Not exactly screaming, but it gets the job done. It has 16GB of DDR2 in 4 sockets, and then there's another 4GB setup as a pair of 2GB modules that I think belong to the honkin' big battery/supercap in the controller node. Unraid sees 20 GB but then says 16 GB is usable. I've got a ton of DDR2 modules lying around... they're ECC though and I didn't look closely enough to see if the Netapp is using ECC or regular modules, but I'm curious if I can bump this up to the 64GB max capacity it seems to think it can handle.
Get your networking all setup... bonds, which ports to include. I'm still figuring some of that out. Some of the ports only show as being 100Mb which isn't great, but at least one of them connected at 1Gb, but the other one like that isn't linking at all but maybe that's because I configured that bond as active/backup. I'm new to Unraid.
The initial parity is going slow, but then again, the parity drives are writing at 40-42 MB/s which probably is about what a SAS spinning hard drive can do. If I swapped these out for SSDs I would expect it to be much faster. We'll see how it performs in read/write tests after the init is complete.
Also, my particular FAS3250 has some additional PCI cards in there, in the lower module (a pair of added 4-port SAS controllers) and there's also another dual 10Gb SFP+ card installed in the main module, but I'm not sure if Unraid is seeing those or not. When I look at the devices that Unraid detected I see 5 different "PMC 8001" devices so that might be correct (I think a "single" PMC 8001 is 2 SAS ports, and I have a total of 10 including the 2 onboard the main controller itself). I just haven't attached anything to the other ports to check. I do have more drive shelves I might hook up just to see if they appear. For the extra 10Gb network ports, I'm still confused about the network interfaces that are on the controller itself.
As mentioned, I'm not a Netapp guy, so I'm still fuzzy on what ports are used for what. I know there's a pair of HA ports (c0a and c0b) if you're using dual controllers, and those, I think, are showing up, but when I tried connecting them to my switch, I wasn't getting link indicators. And it's weird that the em0 port seems to only link at 100Mb, but maybe that's normal for the Netapp? Then there's a couple of Qlogic FC ports, so ignore those. Anyway, point is, I don't know which of the ports on back correspond to what Unraid is seeing, and I kind of need to sit down at my computer out there while I unplug a cable at a time and see what Unraid says is going on. Having that serial console to the box is going to be very helpful in that case since I'll probably lose my network connection to it here and there while testing.
So... that's about it for now. Long post, but I was unable to find anyone else who'd done something like this on a Netapp. A few have mentioned something similar but no instructions or guides on what they did so I was flying blind and I thought this would be helpful to anyone else who felt like doing something a little crazy with an old Netapp. I've got another filer, and FAS2240-4 that, to me, looks like a DS4243 or DS4246 but the top IOM module is replaced with a controller. It may be the same, or totally different... I just pulled that module out to look at it closer on my desk so we'll see. That one could be fun since the drives and controller are all in one chassis so it would take up less space overall. I don't know what else differentiates it with the FAS3250... different CPU/memory, I expect, at a minimum.