So with a lot of other things going on in life, I kinda forgot about this thread. It didn't help at all that the drive spent far more time powered off and cooling down than it did trying to recover data. But eventually I got to the point where I decided it wasn't worth spending any more time on. Final status, I recovered all but 226KB spread across 408 errors of the problem drive.
So now we have 3 disk images, corresponding to the 3 disks that originally made up the array. Two of the images are complete, and the last one is almost complete except for 226KB of zero's where data used to be - but at least we can read those zeros without getting IO errors. Time to move on to the next step...
Step 2 - Prepare to assemble the array
Before we actually do anything, its time to make some backups. As some of the following steps could possibly make (bad) changes to the image files, and it might not even be possible to recover all (or as much) of the data again, make copies of all three image files to a safe location - I made a copy of each on my NAS just in case. On top of that backup copy I also wanted to be able to test things without having to wait to restore 1.5TB over a 1gbps network connection over and over, for that job I turned to snapshots.
First off, we need to turn the image files into block devices. A block device in Linux is just a device that you can read or write in blocks instead of files, a filesystem sits on top of a block device. A hard-drive (eg. /dev/sda) is a block device, a partition (/dev/sda1) is also a block device, a RAID array is a block device, etc. We will be using loopback devices to make our image files work like block devices. As we run these commands pay attention to the output, we will need the loopback device names later on.
$ sudo losetup -f --show <disk1>.img
/dev/loop0
$ sudo losetup -f --show <disk2>.img
/dev/loop1
$ sudo losetup -f --show <disk3>.img
/dev/loop2
Ok, so now we can treat /dev/loop0 as if it was /dev/sdc or whatever the original drives would have been assigned. If it wasn't a RAID array we could run fdisk, fsck, mount, etc. directly against /dev/loop0. Then to save time if we have to do a restore lets snapshot the three new block devices. To set that up we need to know the exact sizes of each disk, and we need another image file for each disk where writes to the snapshot will end up going. These extra image files don't need to be the full size though, only big enough to hold all of the changes that we will be making.
$ sudo blockdev --getsize /dev/loop0
976773168
$ sudo blockdev --getsize /dev/loop1
976773168
$ sudo blockdev --getsize /dev/loop2
976773168
So all of our original disks are 976773168 blocks in size. 976773168 blocks * 512 bytes per block / 1024 bytes per KiB / 1024 KiB per MiB / 1024 MiB per GiB = 465.76174 GiB. Looks good considering we started with 500GB drives. Lets create some more image files and turn them into block devices as well. The truncate commands will create 10GB sparse files - a sparse file only uses as much disk space as the content written into it, like a thin-provisioned virtual machine, so the commands below actually only use up a few KB of disk space. The '.cow' in the filename stands for Copy-on-Write which is how the snapshots we will be making soon will work. Again pay attention to the output as we will need the loopback device names.
$ truncate -s 10G <disk1>.cow.img
$ truncate -s 10G <disk2>.cow.img
$ truncate -s 10G <disk3>.cow.img
$ sudo losetup -f --show <disk1>.cow.img
/dev/loop3
$ sudo losetup -f --show <disk2>.cow.img
/dev/loop4
$ sudo losetup -f --show <disk3>.cow.img
/dev/loop5
Now we can finally create the snapshots, the following commands work directly against Linux's device-mapper subsystem. Normally users don't deal with it directly - md-raid, LVM, and other such technologies usually sit between users and device-mapper to make it more user friendly - but powerful things can be done with it. I'm not going to spend a lot of words describing it as it would take many pages, feel free to read the documentation on the 'dmsetup' command here:
dmsetup(8): low level logical volume management - Linux man page if you want to decode these commands or see what other kinds of things it can do.
$ echo 0 976773168 snapshot-origin /dev/loop0 | sudo dmsetup create tmp-<disk1>-orig
$ echo 0 976773168 snapshot-origin /dev/loop1 | sudo dmsetup create tmp-<disk2>-orig
$ echo 0 976773168 snapshot-origin /dev/loop2 | sudo dmsetup create tmp-<disk3>-orig
$ echo 0 976773168 snapshot /dev/loop0 /dev/loop3 p 128 | sudo dmsetup create tmp-<disk1>-snap
$ echo 0 976773168 snapshot /dev/loop1 /dev/loop4 p 128 | sudo dmsetup create tmp-<disk2>-snap
$ echo 0 976773168 snapshot /dev/loop2 /dev/loop5 p 128 | sudo dmsetup create tmp-<disk3>-snap
Great - we've got lots more block devices! What we've really accomplished is that we now have three block devices named '/dev/mapper/tmp-<diskx>-snap' that look and work exactly like '/dev/loop0-3' that we created first, except that if we make any changes those writes will end up in the appropriate .cow.img file and our large image files remain untouched. If we break something now we can just tear down the device-mapper devices, delete and re-create the .cow.img files as empty files, and re-run the device-mapper commands and try again - two seconds of copy-pasting commands instead of waiting for 1.5TB to come over the network. For reference, this is the little script I kept in a text editor - pasting it into the terminal window would set me back to working on untouched images again.
dmsetup remove tmp-<disk1>-snap
dmsetup remove tmp-<disk2>-snap
dmsetup remove tmp-<disk3>-snap
losetup -d /dev/loop3
losetup -d /dev/loop4
losetup -d /dev/loop5
rm -f *.cow.img
truncate -s 10G <disk1>.cow.img
truncate -s 10G <disk2>.cow.img
truncate -s 10G <disk3>.cow.img
sudo losetup -f --show <disk1>.cow.img
sudo losetup -f --show <disk2>.cow.img
sudo losetup -f --show <disk3>.cow.img
echo 0 976773168 snapshot /dev/loop0 /dev/loop3 p 128 | sudo dmsetup create tmp-<disk1>-snap
echo 0 976773168 snapshot /dev/loop1 /dev/loop4 p 128 | sudo dmsetup create tmp-<disk2>-snap
echo 0 976773168 snapshot /dev/loop2 /dev/loop5 p 128 | sudo dmsetup create tmp-<disk3>-snap