Need some advice/opinions on "broken" hard drives i've sold

vrod

Active Member
Jan 18, 2015
233
35
28
29
Hi all,

This is a rather special question and i'll try to keep the story short.

For some time i've been selling out of old stuff that was decommed from my working place (they didn't mind). I got in contact with a guy from a huge global reselling company and we've been doing many succesfull deals so far. Our latest one however has ended up in a weird situation.

In november we agreed on a deal of 45 drives for x amount of money. Package was sent and received (there was some complications since my fiancee managed to "re-use" a label. :) However about a month and a half after it was received and put up for testing, the sales guy tells me that 30 of these drives are not working. Apparently they don't even mount in their testing server. The guy says that megaraid reports "unconfigured bad, unsupported). Here's some facts about how I prep the sale first:

- I run a complete Killdisk (always save certs and I have these for the drives as well)
- I make sure that none is predicted to fail (S.M.A.R.T.)
- I always pack the drives securely and in antistatic bags (one for each).
- In this case, the sales guy said that there was no physical damage on the shipping package.

Anyone has an idea of what this could be? I mean 30 out of 45 drives which fails is a lot and all of the drives came out of newly decommissioned production servers. Some where even a few months old. On the drives themselves I also don't see any damage.

I do however, think it's a little late to come 1-1,5 months after receiving the drives and say something is wrong with them? I know they test the drives but I cannot control what they might have been trying to do to the drives.

Anyone got some advice? I like to be a fair person but I dont know any place where you could come back after a month (given the product is sold without guarantee which it is here) and claim that the purchased product doesn't work?

I tried to mount one of the drives in my FreeNAS server and got this:

Jan 13 10:15:26 NAS da8 at mps0 bus 0 scbus3 target 23 lun 0
Jan 13 10:15:26 NAS da8: <IBM-ESXS ST9600205SS B556> Fixed Direct Access SPC-4 SCSI device
Jan 13 10:15:26 NAS da8: Serial Number 6XR1S4JP0000B225BZWD
Jan 13 10:15:26 NAS da8: 600.000MB/s transfers
Jan 13 10:15:26 NAS da8: Command Queueing enabled
Jan 13 10:15:26 NAS da8: Attempt to query device size failed: NOT READY, Logical unit not ready, cause n


On the first 15 drives I had 3 that mounted just fine, 2 which doesn't even get recognized and 10 which gives the unsupported message. Could this be firmware?

- Chris
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
I checked on one of them and they match. i did a search on my server and found the certificate from killdisk for that one as well.
 

vl1969

Active Member
Feb 5, 2014
611
69
28
it happens to the best of us.
I got 2 3TB drives from some one on one of the forums I read,
came ok, SMART and other tests OK at arrival,
worked for 6 month just peachy. in a BTRFS raid-1 on raw devices array

one day I come home and my server is down, hard.
no data exists.(I just finish moving all my data to that pool 2 days before)
no drives exist. pool them out try reading them in my main PC
one drive was not even identifiable, second made strange noise and would not mount at all.
lost 2.5TB of data, no backup. I was planning on having 2 pools with rsync but the second pool was not ready.

what are you going to do? luckily had a copy of most of the important data on my main PC and some old drives I planed to reuse but never got around to clean.
some pics and home movies had an extra copy on my inlaw computer.
but lot's and lots of media files collected over the years are gone to la la land.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Apparently they did not work at all when they tried to check the drives. That's what really is bugging me. Especially when they all worked fine when I shipped them
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,668
491
83
Canada
I agree, unless the disks are making unusual noises where physical damage might be apparent, most likely a bad backplane that corrupted the disk controller firmware in some way :)
 

whitey

Moderator
Jun 30, 2014
2,771
872
113
39
Or 520 block size on disks but if you say you tested them prior to sending out I'm sure that is not the issue probably.
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,117
568
113
New York City
www.glaver.org
Anyone has an idea of what this could be? I mean 30 out of 45 drives which fails is a lot and all of the drives came out of newly decommissioned production servers. Some where even a few months old. On the drives themselves I also don't see any damage.
When I sell bare drives or complete systems with drives, I always include a SMART printout for each drive showing both a recent long test and the full report with attributes, showing the current date and time. I keep a copy for my records as well.
I do however, think it's a little late to come 1-1,5 months after receiving the drives and say something is wrong with them? I know they test the drives but I cannot control what they might have been trying to do to the drives.
Seems like a long time to me as well, but some companies just put the drives into inventory without checking (a seller I buy from got burned by Dell that way - supposedly new drives with many hours on them, in sealed Dell antistatic bags).
I tried to mount one of the drives in my FreeNAS server and got this:

Jan 13 10:15:26 NAS da8 at mps0 bus 0 scbus3 target 23 lun 0
Jan 13 10:15:26 NAS da8: <IBM-ESXS ST9600205SS B556> Fixed Direct Access SPC-4 SCSI device
Jan 13 10:15:26 NAS da8: Serial Number 6XR1S4JP0000B225BZWD
Jan 13 10:15:26 NAS da8: 600.000MB/s transfers
Jan 13 10:15:26 NAS da8: Command Queueing enabled
Jan 13 10:15:26 NAS da8: Attempt to query device size failed: NOT READY, Logical unit not ready, cause n


On the first 15 drives I had 3 that mounted just fine, 2 which doesn't even get recognized and 10 which gives the unsupported message. Could this be firmware?
Well, these are Seagates with oddball IBM firmware, but if they worked when you wiped them, they should still work.

I can think of 3 things, offhand:

1) They didn't allow the drives to acclimate for a day or so before spinning them up, and the media was damaged due to condensation inside the HDA from cold dry outside air to warm moist inside air.
2) Somehow these got set in either a generic Seagate or IBM-specific mode like "power up in standby", where they don't spin until told to. On most LSI controllers, you can force a spin-up in the controller BIOS.
3) The drives got zapped somehow, as has happened in the past with Norco backplanes. Usually there will be some obvious damage on the drive circuit board.

The [older] generations of Seagate drives I'm familiar with store the firmware partially in flash on the logic board and partially on reserved tracks on the media. The flash has enough code to spin the drive up and read the rest of the firmware from the media into RAM on the logic board. That's the main reason why "swap the logic board" doesn't work between dissimilar firmware releases.

If you want to send me a pair of duds (one "nobody home" and one "not ready"), I'll do a failure analysis on them (note: this is destructive - the drives will never work again) and let you know what I find. PM me if interested.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Thanks for all your answers. The acclimating thing is also what I think could have something to do with it... They had swapped office space so these were laying in their old storage for a few days or so. Could be they were too quick on the drives in the "outside" part of the package and those "inside" survived because they were not having any mist or so.

The backplane could of course also be right. I have looked at the logic boards from some of them and they dont have any burn marks. The drives are 600GB SAS 2.5" (49y2004) so there's also some models from Toshiba and Hitachi.

What I just thought about... was that there's 30 failed drives. I just looked over a couple of pics the guy sent me and he's also using a x3650 M3 server to test them... There's 15 drives in the picture so could be that there was 2x15 drive batches that they somehow messed up in some way... or am i being crazy now? :D I just find the number rather interesting.

@Terry Kennedy - thanks for all your tips, I guess you live in the US? I am from Germany so it will be a long way to send those drives. :) Which program do you use to test SMART? Just smartmontools/smartctl or something else? Since I already have the killdisk cert (which includes serial# and so on), I would think i would be good but I guess having a SMART certificate helps as well.

The buyer has not tried to get the money back through paypal or anything (I don't think they can actually since it was a direct private purchase, not over ebay), but currently he needs some drives working. Whether its his fault or mine, I guess we could reach a settlement of some sort? Afterall I've been selling over 100 of those drives to him for a pretty good price (for him), just to get rid of the HW.
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,117
568
113
New York City
www.glaver.org
@Terry Kennedy - thanks for all your tips, I guess you live in the US?
Yes.
I am from Germany so it will be a long way to send those drives.
Yup, definitely not worth it for these drives.
Which program do you use to test SMART? Just smartmontools/smartctl or something else?
A variety of tools. In addition to the usual smartmontools, I have some OEM diagnostic software for various drive models. I also have access to engineers for some specific brands / product lines. Then there's a 32-channel logic analyzer and scope for looking at things on the logic board and the spindle motor drive. The last step is HDA inspection port access or complete disassembly and microscopic (if needed - head crashes are obvious) examination.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Thanks everyone for your answers. I offered a 50/50 settlement with the buyer and explained him the information I got from you guys. Unfortunately he is not willing to settle and requires either full refund or 30 working drives.

I'm still testing the first set of 8 drives and did some megacli stuff.... I have attached the PDList output, I don't even see spin information for the drives. Somebody got an idea? :)
 

Attachments

BenchZowner

Member
Nov 16, 2016
36
5
8
37
Do you have working drives of the same models ?
If yes, try swapping the PCB of a working one with one of the "failed" ones and see what happens.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
I do have some and they work fine... However I have now noticed that a lot of the drives fail to spin up (sounds like they try to spin up, then stops and tries again)... I am kind of suspecting that the buyer might have dropped the package accidentially? According to my memory it was not dropped between getting packed and delivered to the courier.
 

whitey

Moderator
Jun 30, 2014
2,771
872
113
39
I do have some and they work fine... However I have now noticed that a lot of the drives fail to spin up (sounds like they try to spin up, then stops and tries again)... I am kind of suspecting that the buyer might have dropped the package accidentially? According to my memory it was not dropped between getting packed and delivered to the courier.
LOL, sorry I had to laugh at this, it was likely dropped, kicked, crammed, jostled, dropped again by the courier/delivery guy on it's way...ya know same deal that happens to all packages I propose :-D

You did have them packed extremely well right, I mean nothing is bullet-proof by the time UPS/fedex/USPS get their grubby hands on them but it's a good start?
 

Tom5051

Active Member
Jan 18, 2017
356
76
28
44
Blame the buyer. Whatever they are doing to 'test' the drives is damaging them. Not your problem.
 

ServerSemi

Active Member
Jan 12, 2017
118
25
28
39
Blame the buyer. Whatever they are doing to 'test' the drives is damaging them. Not your problem.
Oh yeah? Really gold advice there. I've purchased thousands of items on ebay and sold several hundred and let me tell you of all the shitty sellers I've had the pleasure to work with. The ratio of bad seller vs bad buyer is like 10 to 1. Blaming the buyer with no proof of what happened is not really sound advice. Full refund or replacement is in order in this case or get hit with a chargeback that you will lose 100%.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,668
491
83
Canada
At this point, you really only have two options as I see it. 1. Return the working disks to the buyer, after all, they did buy them, and refund fully the faulty ones making it clear to the buyer that all returned disks have been thoroughly tested again and that you will only accept them back this time if they are obviously physically damaged in transit, or 2. Fully refund them and take it on the chin. If it were me, I might just refund them and move on, after all, you do lots of business with them it seems, so it's swings and roundabouts really :)
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Oh yeah? Really gold advice there. I've purchased thousands of items on ebay and sold several hundred and let me tell you of all the shitty sellers I've had the pleasure to work with. The ratio of bad seller vs bad buyer is like 10 to 1. Blaming the buyer with no proof of what happened is not really sound advice. Full refund or replacement is in order in this case or get hit with a chargeback that you will lose 100%.
I do have proof in form of a deletion certificate for every disk stating the serial number as well. I wouldn't usually blame any buyer but I can prove on paper that the disks worked when I sent them. The shipment was insured for 4000 euros so I suppose that it will cover this. Nothing explains how carefully handling the drives and packing them correctly would cause the drives to go bad. Could also be he didn't pay attention to the package and the courier did do damage.