Anyone used T10-PI? -> UPDATE, IT WORKS!!!!!

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

jcl333

Active Member
May 28, 2011
253
74
28
Hello, wondering if anyone has used or tried to use this feature, I am having trouble finding enough information on it.

I thought this was a feature reserved for commercial SAN vendors or other special applications, but now I am not so sure.
I kept seeing to references like needing support all the way from the disk to the application and everything in between would need to be aware of and support the feature for it to work. But, of course we use our SAN with VMware and Windows, neither of which support T10-PI that I am aware of, let alone the applications. The SAN is taking care of it in the background and just serving out virtual disks.

What I think I have established is:
- The disks themselves need to be SAS, and need to support T10-PI as well as be formatted to do so
- The controller/driver/firmware also need to support it

I have an LSI MegaRAID SAS 9361-8i
The hardware and software clearly mentions support for T10-Pi "Protection Information".
Here is a shot from Storage Authority running from Windows
protection information.png

Here are some of the more useful / helpful resources I have found explaining this feature:

This is an old article talking about the differences between SAS and SATA, but it goes into great detail about PI:
SAS vs. SATA
Interestingly, it also goes into the relative merits of ZFS and how it can and can not address some of the issues.

This is a PDF from Seagate with really good information about their support for the feature:
https://www.seagate.com/files/stati...-from-corruption-technology-paper-tp621us.pdf

Both the SAS spinners and SSDs I am using came out of an EMC SAN, and were originally 520-byte drives that I re-formatted to 512-byte so that I could use them.
I am wondering, if I were to format them back to 520-byte drives again, would that be recognized by the controller and allow me to enable this feature?
- I would have to figure out the exact command to do it, but I am close to figuring it out​

You have to have an HBA or be in JBOD mode to be able to format the drives, but that is easy enough.
It might be slightly inconvenient because regular Windows and say TrueNAS probably would not recognize the drives directly.
But, the trade-off could be being able to have check summing / silent data corruption protection in hardware, without having to wrestle with ZFS or ReFS.

Has anyone tried this?

Thanks

-JCL

P.S. - I also purchased the super-cap backed cache and the license for CacheCade 2.0 that allows me to use a pool of SSDs as cache, should be interesting to see how it performs. I am using it with 16x 4TB spinners + 4x 400Gig SSDs in a R60+R1 setup. This gives me 743GB of cache and just over 43TB usable.
 

Attachments

jcl333

Active Member
May 28, 2011
253
74
28
OK, I did some better Google-FU and I think I may have this almost figured out.
I think it will just come down to the right format command.

Here are some interesting finds:
- A careful re-read of the Seagate PDF, the section on "Formatting PI Drives" is basically what you need to know
- Seagate drives support type 1 and 2 protection, unfortunately it falls just short of what the exact command is
- Here is another excellent PDF explanation of this from Fujitsu and Q-Logic: https://www.fujitsu.com/downloads/strsys/system/dx_s3_Oracle_Linux_T10_PI_Q_en_011.pdf
- Here is an excerpt from the Ubuntu manual pages with examples:
Ubuntu Manpage: sg_format - format or resize a SCSI disk (perhaps change its block size)

The basic command is this: sg_format -v --size=512 --long --format --fmtpinfo=2 --pfu=0

A helpful post here describes some example commands: (the post is about 40% of the way down)
Expanding the HPE DAS with 42TB But Where are They - 814
commands.png

Another post I found gives almost a step-by-step specifically related to MegaRAID controllers: Replace megaraid by strocli
- It's almost at the end of the article, search for "T10 PI for megaraid" and it will take you there
- There is a curious warning about and issue with Dell PERC controllers and the Rapid Rebuild feature
- This is the kind of thing I get a little concerned about, getting into an area of proprietary issues off the beaten path
- But, if I can find what I need to stick to what is required by the Seagate disks and LSI MegaRAID, I think it should be OK

Trying to understand what the "--pfu=x" part of the command means, I found this page from Oracle:
https://docs.oracle.com/cd/E88353_01/html/E72487/sg-format-8.html
- Actually I think this is the most complete explanation on how to use the sg_format command I have seen yet
- Since the Seagate PDF indicates that only types 1 and 2 are supported, and I am using Seagate drives, I will go with that
- Note that, the Seagate PDF is from 2011, the drives I have are from 2014, so this may be out of date
- The oracle document later describes the commands needed to query your drive for the protection level it supports

So, in theory, the right command may be this:
sg_format -v --size=512 --long --format --fmtpinfo=2 --pfu=1

I am going to give this a try and report back.

-JCL
 

jcl333

Active Member
May 28, 2011
253
74
28
Whelp, the saga continues.

I have not found any combination of the command that works, they all error out.
The disks behave as though they don't support it, but according to Seagate they do:
https://www.seagate.com/www-content...d/docs/constellation-es-3-ds1769-3-1502us.pdf
seagate.png
I suppose it could be an issue with the EMC firmware being used, but since those started out as 520 drives I will assume it can be done.
Maybe I will grab a couple of 520 disks from work so that I can run some of the sg3_utils commands on them rather than just re-formatting to see if I can get some clues.

I am following some of the command examples at the end of this page:
Replace megaraid by strocli
- Even this mentions having to run the format command twice to get it to work

An IBM article here mentions some T10 capable drives may not be detected as such:
H21621: T10 Protection Information (PI) capable disk drives do not show as supporting T10 PI

From reading some other articles, it sounds like to change it back to 520/PI may require more than just the format command, such as "MODE SELECT".
Looking through the docs for sg3_utils mentions some commands like this, but I haven't found something that makes sense yet.
The sg3_utils package
This is the article that talks a little about mode select:
https://github.com/hreinecke/sg3_utils/blob/master/doc/sg_format.8

Some other things to try:
- See if it works on one of the Hitachi SSDs
- Try it on a "real" HBA rather than a RAID controller in JBOD personality mode

Part of the difficulty here is that nearly every article I search for is how to go from 520 -> 512 and remove PI, not the other way around.

If anyone has any clues, I would greatly appreciate it. But I am having a great forum discussion with myself here ;-)

-JCL
 

jcl333

Active Member
May 28, 2011
253
74
28
So, I tried the same thing with the SSDs and it works, so at least I know I am using the right commands:

sg_format --format --fmtpinfo=3 --pfu=0 <drive#> works fine.

Here is what the SSDs look like, one formatted with PI and one without:
before.pngafter.png
- Pretty cool, nice to see it work
- I did also change the block size to 4096, just playing around really, posted a separate question about that actually

But, this is probably not helpful unless I can also figure out how to do it with the Seagate disks, I don't think the controller would like mixing them.
I am using the SSDs with CachCade 2.0, which I am not even sure it supports that anyway.

Maybe I can find a Seagate specific forum that would have the answer.

-JCL
 

Whaaat

Active Member
Jan 31, 2020
301
157
43
1. You don't need to format back to 520 bps to have PI enabled
2. Why don't you use Seachest utilities for your seagate?
 

jcl333

Active Member
May 28, 2011
253
74
28
1. You don't need to format back to 520 bps to have PI enabled
2. Why don't you use Seachest utilities for your seagate?
Thanks for the reply.

I have the SeaChest utilities, that format command also refused to format with PI.

-JCL
 

jcl333

Active Member
May 28, 2011
253
74
28
OK, here is the SeaChest stuff.

The command I am using is as follows, PD5 is one of the drives in question:
SeaChest_Format -d PD5 --protectionType 2 --formatUnit current --confirm this-will-erase-data

Here is a screenshot of the output:
SeaChest_Format--format.png

Here is a drive listing, this is on my RAID controller in HBA/JBOD mode:
SeaChest_Format--scan.png

Here is the information page about PD5, and for comparison, the same for one of the SSD that does support PI:
See the red square showing the SSD does support it, but this is missing from the Seagate drive.
SeaChest_Format--i_disc.pngSeaChest_Format--i_ssd.png

Since Seagate does list these drives as supporting this, I am assuming that these EMC OEM drives just didn't implement it in the firmware they use.
But, this is odd because the EMC SAN does use T10-PI, although it could be their proprietary implementation.

I did poke around the drives with Niagara as well as try to format it. I could not see anything obvious that would indicate that PI was disabled or that it could be enabled again, but there are so many options in that program you really need a guide on what you are specifically trying to do. I managed to unlock the SSDs to be 12Gig drives for example, but only because I found some PDF screenshots that showed exactly what to do.

I have started looking at articles that talk about updating firmware on these drives, but so far I haven't seen anything that I can run with, much less that would actually put back the PI feature.

I could try it on my 9400-16i HBA but I doubt that it will make a difference, everything is showing up as it should.

It would be awesome if I was just doing the command wrong, but there is not much to this tool really.

Let me know what you think.

-JCL
 

Whaaat

Active Member
Jan 31, 2020
301
157
43
Yep, it looks like Clariion firmware with SPC-3 set of commands is reduced version that doesn't support PI. EMC implements redundancy via additional 8 bytes in 520pbs format, so they might not need PI at all. You cannot even switch firmware in seagate drives - they are protected from cross-vendor flashing.
 

jcl333

Active Member
May 28, 2011
253
74
28
Yep, it looks like Clariion firmware with SPC-3 set of commands is reduced version that doesn't support PI. EMC implements redundancy via additional 8 bytes in 520pbs format, so they might not need PI at all. You cannot even switch firmware in seagate drives - they are protected from cross-vendor flashing.
The only post I have found on this is from an older post here:
https://forums.servethehome.com/ind...ating-firmware-on-seagate-st4000nm0023.22383/
If you follow the one weblink, the person claims to have changed firmware from Seagate to Dell.
But, he was using a HEX editor to change values in the firmware files to get it to go, maybe going a little too far for me.
Also, even if I did that, it does not mean the PI feature would necessarily come back.

I think I am going to just buy some drives that I know have PI enabled from the get go. These drives are only 4TB anyways and they run hot.

-JCL
 

jcl333

Active Member
May 28, 2011
253
74
28
Stop your grinning and drop your linen!! I did it!!
I SUCCESSFULLY flashed a drive from the EMC firmware to the latest generic Seagate firmware, check it out:

Before firmware flash: :mad: ------------------------------------------------------->After firmware flash: :cool:
before.pngafter.png
And, for the money shot, check out at the bottom, SPC-4 and Protection Type 1 and 2 now supported!!!!
Beyond that, all the regular features of a generic drive are now restored.

Here is a shot of the firmware after it was flashed:
- Notice how the flashed drive is now at E007 (which is the latest 2017 firmware) and displays the SN information differently now
- And I did it with SeaTools for Windows of all things, the secret was just getting a copy of the right firmware in the right format.
- The software is looking for a file in LOD format, the files I had found up to this point were in .fla or .fwh format
flash firmware.png

I can't believe it, I was just about to give up and spend money on new generic drives, but I am really stubborn.
So I thought I would give it one more try, I was poking around over at HDDGuru, and found this download for this model drive that included a copy of the right firmware.
After it worked, I figured yeah but maybe this is some really old firmware - nope it is the latest ;-)
I can't download the firmware from Seagate's site with the serial number on the drives I have, so I put in one from another drive I saw online, and it worked.
When I go to the download page, it had all the tools and specifications, and the firmware was the same one I found earlier.

To be fair, I must have gotten lucky and found a batch of EMC drives that were not firmware locked.
You can actually see near the end of the information page "firmware download support"...
I have like 60 of these, so it will be interesting to see if they can all be flashed or only some of them. And of course, now I need to re-format them with T10-PI Protection Type 2 enabled, that will take awhile.

But, I still feel good, everywhere I looked people said it could not be done, or only by hacking the firmware with a hex editor. Whelp, hold my beer!

It would be great if someone else on this forum has some of these drives and could give it a try, see if it works for them too. No idea it this will work on other drives.
I am thinking of trying it on the SSDs I have as well, but maybe I should not push my luck since they already do what I want.

Happy Thanksgiving everybody!!

-JCL
 

Whaaat

Active Member
Jan 31, 2020
301
157
43
Great if this works. I tried the firmware update with SeaTool on 15k.7 with Oracle firmware many years ago and although SeaTools reported 100% success and standard firmware model number, it was so only until next reboot - those drives have some sort of sophisticated protection from reflashing - current firmware and saved firmware may differ. I have a couple of Clariion ES.3 working in Raid0 will also try re-flashing this way
 

jcl333

Active Member
May 28, 2011
253
74
28
Well, I just flashed 16 drives and then rebooted, and they were still at the new firmware.

It's a beautiful thing:
format with pi.png

Got all 16 drives formatting, takes about a day.

-JCL
 
  • Like
Reactions: bandit8623