Cleaning SAS backplane connectors

[llowrey]

I recently acquired a second hand (at least!) chassis with SAS backplanes. I've been having a ton of trouble with it. I first put in a bunch of old SATA drives but they all were constantly erroring and when the did work, they were unusually slow. I then swapped them out and put in 12g SAS drives and they are working much better... but not perfect.

I'm using the chassis as a simple JBOD enclosure and have an HP 12g SAS expander mounted in it. It's a bit non-ideal in that I'm using one of the below to expose two internal connectors as external for connection to my server. I have other chassis that I'm using as JBODs but using SAS expanders with proper external ports. I don't think the reliability is due to janky connectivity because not all drives are impacted equally, which is what I would expect if the connection from the expander to host was flaky.



I don't think it's a power supply issue since I would expect the SAS drives to put more strain on the power side of things. Given that the SAS drives are behaving better than the SATA drives makes me think that the higher signaling voltage of SAS is able to (better) overcome whatever is impacting the signal from the drives to the expander.

The chassis was filthy with dust when I got it. I was shocked by how dirty it was. I used a can-o-air to blow out the connectors but I'm thinking that there may be more grunge in the backplane connectors than just loose dust.

Any suggestions for how to go about thoroughly cleaning the SAS backplane contacts? They're quite small and it's not particularly easy to get anything in there (and get it back out). I've never tried using contact cleaner on anything that small.

 

[BlueFox]

What is the total cable length between the SAS expander and the SATA drives? If over a meter, well, that's likely going to be your problem.

 

[llowrey]

internal cables were all either 0.5m or 0.75m

 

[BlueFox]

And the external? Need the total length.

 

[llowrey]

The chassis in question is at the end of a daisy-chain with one JBOD in the middle. External cables are 2m

[hba -> 0.5m -> expander] -> 2m -> [ expander ] ->2m -> [ expander -> 0.75m -> sas/sata drive]

So, the max signal-lenghth is 2m and at SAS signaling and 0.75m at SATA signaling.

 

[BlueFox]

Gotcha. Wasn't completely clear to me if you had the expander outside the chassis for the final leg. Length shouldn't be a problem then. For cleaning, rubbing alcohol or contact cleaner, though ensure the latter won't damage the plastic portion of the connector.

 

[mattventura]

What are the specific errors you're seeing? Is there anything in SMART data that stands out?

If the issue is indeed dirty connectors, then if you were to swap drives between different slots, the errors would follow the slot rather than the drive. Also, when you say you want to clean the SAS contacts, do you mean the drive ports, or the upstream ports?

 

[mr44er]

I had once a missing/broken pin on one of the canisters/expanders. This resulted in constant error led of one disk slot and I suspected a bad disk, until I got the stupid idea to compare the pins with another canister.

I have also some of these adapters in your picture, they still fit tight and are just a passive connection -> unlikely as error source. Maybe a broken cable inside?
For better cleaning the connectors I would disassemble the case to reach the backplane directly.

I think nearly all of these rebrand/casevendors came to the point where the cumstomers had problems with SATA disks. Technically the SAS-protocol was designed to be also compatible with SATA, but in reality it didn't work well.
LSI delivered at that time interposers to circumvent stability issus and to realize dual path with SATA. But that workaround should be avoided, because it splits SATA600 to 2x300 per path and only if the interposer supports it and it's enabled in the firmware (yes, hardlock).

But anyway, if you say that SAS disks also don't behave right...the dust seems to be the problem for now.

 

[llowrey]

What are the specific errors you're seeing? Is there anything in SMART data that stands out?

If the issue is indeed dirty connectors, then if you were to swap drives between different slots, the errors would follow the slot rather than the drive. Also, when you say you want to clean the SAS contacts, do you mean the drive ports, or the upstream ports?

In the case of the SAS drives it's:

sd 33:0:155:0: [sdaa] tag#2242 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
sd 33:0:155:0: [sdaa] tag#2242 Sense Key : Aborted Command [current] [descriptor]
sd 33:0:155:0: [sdaa] tag#2242 Add. Sense: Ack/nak timeout
sd 33:0:155:0: [sdaa] tag#2242 CDB: Read(10) 28 00 05 f1 6d a9 00 00 03 00

Good suggestion about swapping drives. The sdaa drive was particularly bad so I swapped it with a neighbor and the new drive in that slot is noisy and the drive that had been in that slot is now quiet. So, that confirms a problem with the slot but still could be an issue with the backplane, the cable to the expander, or the expander.

 

[mr44er]

Do you see changes in speed negotiation if you swap the drives? If your SATA600-disks are detected sometimes and in different ports as SATA300 vs a SAS600-disk is always detected correctly, then this would be such an irregularity.

 

[llowrey]

Do you see changes in speed negotiation if you swap the drives? If your SATA600-disks are detected sometimes and in different ports as SATA300 vs a SAS600-disk is always detected correctly, then this would be such an irregularity.

Good suggestion. I put one SATA back in the chassis and it only negotiated 3Gbps (SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)) Interestingly, the SAS drives all negotiated 12Gbps even though the links are not completely reliable.

 

[mr44er]

Yeah, same effect that's often seen with SATA. Since then I have SAS-only and no problems.

SAS should be fully compatible downwards...if everything would work ideal and nice together.
Could you draw a simple illustration of how you connected all your SAS-stuff for better overview?
Also a list with exact model names of the components, firmware version, photo of the labels (cover up the serial numbers, these should not be in public), especially interesting would be the date of the backplane. That should be etched or engraved in an edge.
Have you plugged in all the power supplies? Some shelves don't power up fully, when 50% or less of all power supplies are running...
How many slots for these canisters do you have? If you only use one canister, then it should be in the first (upper) slot. That is a hint in some manuals for SATA drives.

 

[mattventura]

Are the bad slots further away from the connectors by any chance? It could be going over the 1m limit once trace length is factored in.

 

[llowrey]

After doing an absurd amount of testing to reduce variables, I'm left with my problems likely being due to the type of SAS connector on the backplanes.

This Norco chassis has 5 separate passive backplanes. All five are identical, except, two boards have SAS connectors that are different from the other three. The three boards with the connector type A operate as expected at 12Gbps for SAS and 6Gbps for SATA. No issues. The two boards with type B connectors have at least one slot which doesn't work well... or at all. (My type A/B notation is arbitrary)

In the photo below, the top connector is the 'bad' type. There are a few obvious differences but I just can't figure why/how they perform differently. The only thing I saw which seems plausible is that the pins of the 'bad' type have tabs on either side inside the main body, presumably for maintaining position/alignment. There is very little space between them. The 'good' type also has these tabs but they are smaller and there is more space between them. Perhaps the small gaps between the pins in the 'bad' type cause interference at high link rates.

Anyway, lowering the link rate for the 'bad' connectors to 6Gbps for SAS and 3Gbps for SATA solved all issues so I'm going to consider this ordeal over.