Much nashing of teeth...

[pricklypunter]

So here we go...

My laptop has been having the odd disk niggle over the last few months, mainly, I suspected, just windows being cluttered up, fragmentation etc, plus I had a whole load of stuff sitting on a partition that I really only use rarely. So I decided this morning that it would be a great idea to fire everything important to me, over onto the server for safe keeping and just to zero my laptop disk and start over, all nice and clean. Great plan, problem was in the implementation.

I disconnected my iscsi disks from my server VM so that I could use it directly with my laptop. All goes well so far. I copy all of my work, photos and everything else important over to the server ZVol. All was going well, and quickly too!

I then roll over my Ethernet cable with my chair and pull out my net connection. Don't ask why it was there to begin with. So, after swapping the cable out, I find I have the oddest of issues, I can ping stuff, I can access the Internet, but not the server. My iscsi drives are mapped correctly, but not accessible, gave me a message of not accessible/ corrupted.

I figured, damn, my laptop is screwed up, so I reboot. No dice, still the same. I decided to reboot the server and voila, everything is back to normal. Or so I thought.

I continue copying the last of my stuff over and then discover that some folders are ok, and some are giving me that same error message again. I figure the cable being yanked out has caused something odd to happen with NTFS, so I run chkdsk. It fixes it for me by deleting everything except the last folder I copied over plus a few other files!

I'm now 6hrs into a deep file scan/ recovery, after having smacked myself about for not paying attention earlier.

When I disconnected the iscsi ZVols from my DC's VM, I forgot to remove one of the "favourite targets", when I rebooted the server, iscsi re-established the iscsi connection to the same disk I was using for my backup and at some point the server must have accessed the disk at the same time I was using it with the laptop.

I lost my backup last month due to an external disk re-jiggle, so this was kind of a "two birds with one stone" type thing. I figured I would get the data off my laptop, clean it up and copy back what I needed to backup giving me two copies. Such a rookie mistake, but it may end up costing me around 900B of irreplaceable data.

Not 900MB, 900GB, I can't even type now it seems...

 

[cperalt1]

Was there any snapshot of the zvol you could use if before the data corruption. Having been in a similar position I was happy for having set up auto snapshots.

 

[pricklypunter]

I would love to say yes to that question but...unfortunately...no, nor do I have auto snapshots set up, so no chance of recovery from that direction. This was an off the cuff decision I made this morning and I just grabbed some unused disk space with fast access to stick the files on until I had cleaned up the laptop and time to sort through stuff. The intention was to clean up what files I really needed to backup and move them from the server onto my backup disks. It's my own stupid fault for not being careful, but I'm mildly hopeful, so far recuva is reporting that it has found loads of files, fingers crossed they are recoverable

It is still on stage 1, reporting another 5hrs to go...

update:

So, it completed, and was about as much use as a wet rag...basically showing me a whole load of not a lot, plus the few files that I know I can recover myself.

However, thanks to running recuva, I now know why this is proving difficult. When I rebooted my laptop I nipped away to grab a coffee, plus I didn't pay any mind when I rebooted the server, just waited till my mapped drives reappeared. At some point, either when the server rebooted, or the laptop, one of them ran chkdsk on the ZVol. I know this because about the only useful thing I learned from recuva was that there's a found.000 folder on the root of the ZVol. I have never used chkdsk manually as it's about the worst util on the planet imo. So it's a double whammy, corruption from the iscsi cock up and then chkdsk deciding it knows best.

I think I can kiss my data goodbye, but just in case, I am now running getdataback. I'm a lot less confident now, as I am seeing just about every file it identifies showing a byte size of zero


success, sorta:

GetDataBack came through for me, well, mostly. I lost a few files here and there (2000+, thanks chkdsk) but I was able to get 99% of my data back. I was fortunate in that I remembered that last month I had also copied most of that data to another Zvol and from there on to my backup disks. Although I had since deleted the files, and lost that backup disk, I hadn't really done anything with the Zvol since. So I also ran GetDataBack against that Zvol. I was able to make up most of the missing files from there.

In all I would say I recovered 99.95% of my data, the rest being transient stuff that I would not have been backing up anyway, however, lesson learned the hard way