Toshiba 12Gb SAS3 SSD usually negotiates link at 6Gb/s

[Sniper_X]

I have several Toshiba PX02SMF080 800GB SSD's.

These are 12Gb/s (SAS 3) drives.

However, on my Quanta chassis, these negotiate at 6Gb most of the time, but if i change slot, or remove and re-insert them, they can negotiate at 12gb.
It's just very inconsistent and it usually only links up at 6.

This happens on all four nodes. SAME behavior.

The HBA is a QCT/Quanta 3108 (a.k.a. LSI MegaRAID SAS 9361-8i).

All firmware is latest on the server and the RAID card.

It's driving me crazy, and the server has to ship ASAP.

Here is a screenshot of one of the nodes in the BIOS for hte Raid card.
Since it's in the BIOS, this shows that this is a hardware issue and not driver related.

 

[psannz]

What speed are the other disks working at? 6G or 12G?

:edit:
Are the Toshiba Disks from Toshiba themselves, or Dell/HP/Fujitsu/etc branded?
Dell for instance have released 3 firmware updates for the PX02 series.

 

[Sniper_X]

Other spinning SAS3 drives link up at 12 and only 6 rarely.

it seems that 12gb/s SSDs are having linking issues.

linking at 6 and rarely at 12.

i have posted in the chassis section a Thread regarding the linking boards between the mid plane and SAS backplane.

I just swapped out the REV C linking boards to older versions (rev a), and I’m getting 12g solid links now.

The newer REV C boards have these DIP switches, where as the older “A” boards do not.

This is truly frustrating as now I have my last whole chassis that has these linking boards and now I know I’ll have issues with “specific drives” that might not link at proper speed.

:|

I wish I knew what to do here.

maybe I’ll try toggling these switches to see what happens.

 

[psannz]

That's an annoying bug. Probably doesn't help that the PX02SMs were release before SAS3 controllers were generally available. They were actually one of the very first SAS3 SSDs ever built....

Toshiba PX02SM Series SAS3 Enterprise SSD Review

Toshiba PX02SM Series enterprise SSD reflects the strengths of Toshiba's eMLC engineering. It performs in synthetic tests and in the MySQL environment

www.storagereview.com

 

[Sniper_X]

Agreed.

However, now I’m starting to find out that the “linking boards” between the mid plane and the SAS backplane may have been the problem.

It seems the newer mid plane boards have these dip switches on them for some strange reason.
The older initial revision of these boards had no dip switches and we’re far simpler devices.

Some thing about these revision “C” Linking boards may be part of the problem.

Granted, I also replaced the med plane at the same time and those seem different.

I have another server I’m going to build that is identical and I’m hoping I’ll get to swap out These components one by one in the future.

For now, I’m blaming the linking boards not the mid-plane.

Even though the mid-plane is also different

 

[NateS]

Possibly a dodgy cable or backplane? Do you have any other cables you can try, ideally straight from the HBA to the drive, to see if it reliably links up at 12G then? And I'd recommend double checking that every cable and the backplane are all rated for 12G.

The fact that it occasionally links up at 12G but usually at 6G most likely means the communication channel is just barely good enough sometimes to do 12G, but it's pretty marginal. Usually this sort of thing is a hardware problem, and the most likely culprit is generally cables and interconnects.

Edit since I think we were writing at the same time: it sounds like this server has a much more complicated SAS routing setup than I've encountered before. Most likely it's right at, or slightly over, the limit of the SAS spec, and the drives either don't tolerate any deviation from the spec, or they aren't happy with something right on the edge of the spec (which means they're the one not spec compliant). Figuring out exactly which device is out of spec would likely require an oscilloscope and a lot of hardware debug knowledge unfortunately, so I think you've got the right idea swapping out one device at a time.

 

[Sniper_X]

I found the issue.

i had a second thread going out in the “chassis” area.

read here and you’ll see that something is “off” with the mid plane/linking boards.

The other thread with the solution.

im pretty sure it was a signal issue with the boards with switches on them vs. the more simple versions.