SOLVED: I cannot format/use HGST Ultrastar DC SN200 SAS NVMe 7.68TB HUSMR7676BDP3Y1

rootgremlin

New Member
Jun 9, 2016
27
4
3
SAS and nvme are completely different interface protocols. You need a SAS hba, like the 9440 you ordered.
Actually yes, You need to look closely at what protocol you are talking to the Drives,
and ignore the (old known fact) SATA and SAS "Cables".
Like the new "MegaRAID 9440-8i Tri-Mode Storage Adapter is a 12Gb/s SAS/SATA/PCIe (NVMe) " controller cards enable you to connect either Sata/sas/nvme devices to the same Controller with the same "SAS" cables. While in fact this specific Drive speaks NVME over the "SAS" Cable. (While still can speak SAS and even SATA to a connected Drive)

Also all Linux partitiontools DO NOT make any changes to the partitiontable until you commit/write the new Partitiontable to disk. (which fdisk even warns you about when you start it)

Also, since it looks like you somewhat know your way around Linux, read through the full bootmessage for some kind of hint whats going on.
I one time had a similar encounter with a Hard Drive using an "hpa" (=Host protected Area) which is some kind of internal Firmware mechanism to shortstroke a Drive by only exposing a certein size of it to the operating system.

For me it looks like something similar has been done here.
 

c steff

Member
Apr 18, 2019
48
6
8
Also, since it looks like you somewhat know your way around Linux, read through the full bootmessage for some kind of hint whats going on.
I one time had a similar encounter with a Hard Drive using an "hpa" (=Host protected Area) which is some kind of internal Firmware mechanism to shortstroke a Drive by only exposing a certein size of it to the operating system.

For me it looks like something similar has been done here.
sounds like it
62% progress for the long format but I will check once done
I presume I should look at /var/log/dmesg.log but I don't know that much linux to "interpret" everything but will paste here some of it if relevant...
 

rivet

New Member
Jan 12, 2019
20
6
3
As azev pointed out, the drive is not SAS, it's NVME. That means that the sg* utilities aren't going to be useful, because those utilities are designed to work with the SCSI protocol.

It's encouraging that your HD Sentinel and Crystal Disk Info output shows the model number and firmware; that's good evidence that the OS recognizes the device.

From some of the other output you posted, I suspect that the drive has a GPT partition table, but no partitions defined. Under Ubuntu, the "fdisk" only works with MBR partition tables. For GPT, you need gdisk, parted, gparted, etc.

Would it possible for you to run the following commands under Ubuntu and show us the output?
Code:
dmesg | grep -i nvme
nvme list
lsblk
parted /dev/nvme0n1 print
I missed it before, but your "nvme list" output shows the format to be 512 B + 8 B.

When your Windows format is done, you might try in Ubuntu:
Code:
nvme id-ns /dev/nvme0n1
My NVME drive is showing 512 B + 0 B as the format.

There's a "nvme format" command in Linux that looks like it lets you specify the block size and if the end-to-end protection metadata (the extra 8 bytes) is enabled.
 
  • Like
Reactions: c steff

c steff

Member
Apr 18, 2019
48
6
8
few hours since 100% but still not done maybe it got stuck maybe it doesn't know the size of the partition
should I stop it or let it do it until I see the format summary ?!
 

c steff

Member
Apr 18, 2019
48
6
8
My NVME drive is showing 512 B + 0 B as the format.

There's a "nvme format" command in Linux that looks like it lets you specify the block size and if the end-to-end protection metadata (the extra 8 bytes) is enabled.
I stopped long format
I run
Code:
nvme id-ns /dev/nvme0n1
See below. What is the nvme format command please or what next?
Code:
ubuntu@ubuntu:~$ sudo nvme id-ns /dev/nvme0n1
NVME Identify Namespace 1:
nsze    : 0x37e3e92b0
ncap    : 0x37e3e92b0
nuse    : 0x37e3e92b0
nsfeat  : 0
nlbaf   : 3
flbas   : 0x1
mc      : 0x3
dpc     : 0x17
dps     : 0
nmic    : 0x1
rescap  : 0x7f
fpi     : 0x80
nawun   : 0
nawupf  : 0
nacwu   : 0
nabsn   : 0
nabo    : 0
nabspf  : 0
noiob   : 0
nvmcap  : 7687991459840
nguid   : 0000000000000000000cca0b01fa2180
eui64   : 000cca0b01fa2180
lbaf  0 : ms:0   lbads:9  rp:0
lbaf  1 : ms:8   lbads:9  rp:0x2 (in use)
lbaf  2 : ms:0   lbads:12 rp:0
lbaf  3 : ms:8   lbads:12 rp:0x1
 

c steff

Member
Apr 18, 2019
48
6
8
the help for nvme format below but I am not sure which options I need
Code:
Re-format a specified namespace on the given device. Can erase all data in
namespace (user data erase) or delete data encryption key if specified. Can
also be used to change LBAF to change the namespaces reported physical block
format.

Options:
  [  --namespace-id=<NUM>, -n <NUM> ]   --- identifier of desired namespace
  [  --timeout=<NUM>, -t <NUM> ]        --- timeout value, in milliseconds
  [  --lbaf=<NUM>, -l <NUM> ]           --- LBA format to apply (required)
  [  --ses=<NUM>, -s <NUM> ]            --- [0-2]: secure erase
  [  --pi=<NUM>, -i <NUM> ]             --- [0-3]: protection info off/Type
                                            1/Type 2/Type 3
  [  --pil=<NUM>, -p <NUM> ]            --- [0-1]: protection info location
                                            last/first 8 bytes of metadata
  [  --ms=<NUM>, -m <NUM> ]             --- [0-1]: extended format off/on
  [  --reset, -r ]                      --- Automatically reset the
                                            controller after successful
                                            format
 

c steff

Member
Apr 18, 2019
48
6
8
WD support was kind enough to provide me the HDM software by email. They sent me 3.4 version (online PDF manual only is for 3.2 but 3.4 includes support for SN200 my SSD and I have the 3.4 manual from them as well)
HDM for Windows does not show in scan the SSD but this is maybe it needs the HGST NVMe drivers as per manual
However in ubuntu it shows the SSD fine
See below HDM help/commands, HDM scan and HDM generate-report. Not sure what to do next, any help based on below and let me know if you want me help for any command so you can see what I should try next
Link to manual: http://www.cstefan.com/HGST.pdf

Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm
usage: hdm <command> [-o, --output-format FORMAT] [-c, --config PATH]
                     {-u, --uid UID | -p, --path PATH | -a, --alias ALIAS}
                     <command args>

COMMANDS:
   alias               Sets or clears a user supplied name for a device.
   capture-diagnostics  Captures diagnostic data from a physical device.
   configure-smart     Configures SMART thresholds and clears warnings on
                       physical devices.
   configure-trace     Displays and configures the trace settings.
   format              Performs a low-level format on devices.
   generate-report     Retrieves information and statistics for devices.
   get-capabilities    Lists commands supported by devices.
   get-feature         Retrieves feature data from devices.
   get-info            Retrieves properties from devices.
   get-inquiry-page    Retrieves inquiry pages from devices.
   get-log-page        Retrieves log pages from devices.
   get-mode-page       Retrieves mode pages from devices.
   get-smart           Retrieves SMART statistics and thresholds from devices.
   get-state           Retrieves state information from devices.
   get-statistics      Retrieves statistics from devices.
   help                Prints command help information.
   locate              Enables (default), disables, or gets the status of the
                       beacon LED for physical devices.
   manage-firmware     Displays and updates firmware on physical devices.
   manage-namespaces   Displays and manages namespaces on a capable physical
                       device.
   manage-power        Displays and controls power states of a physical device.
   manage-uefi         Displays and updates the UEFI driver on physical
                       devices.
   monitor             Continuously retrieves device information at a specified
                       interval.
   prepare-for-removal  Prepares physical devices for safe removal from the
                       system.
   reset-to-defaults   Resets physical devices to factory default
                       configuration.
   resize              Changes the user capacity of physical devices.
   sanitize            Performs a sanitize operation to erase all user data on
                       physical devices.
   scan                Lists the storage devices attached to this host.
   secure-erase        Securely erases all user data from physical devices.
   secure-purge        Performs a secure purge on physical devices.
   self-test           Executes device self tests.
   version             Displays HDM build and version information.

Run 'hdm help <command>' for more information about a particular hdm command.

For more information about HDM, refer to the HDM User Guide and Release Notes
documents.
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm scan
[1C58SDM00003375FHUSMR7676BDP3Y10023]
  Product Name        = Ultrastar SN200
  Device Type         = NVMe Controller
  Device Path         = /dev/nvme0
  UID                 = 1C58SDM00003375FHUSMR7676BDP3Y10023
  Alias               = @nvme0
  Vendor Name         = HGST
  Model Name          = HUSMR7676BDP3Y1
  Vendor ID           = 1C58
  Device ID           = 0023
  NVM Subsystem UID   = 1C58SDM00003375FHUSMR7676BDP3Y1

[000CCA0B01FA2180]
  Device Type         = NVMe Namespace
  Device Path         = /dev/nvme0n1
  UID                 = 000CCA0B01FA2180
  Alias               = @nvmens0
  Parent Product Name = Ultrastar SN200
  Parent Device Type  = NVMe Controller
  Parent Device Path  = /dev/nvme0
  Parent UID          = 1C58SDM00003375FHUSMR7676BDP3Y10023
  NVM Subsystem UID   = 1C58SDM00003375FHUSMR7676BDP3Y1
  Namespace ID        = 1
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm generate-report
[1C58SDM00003375FHUSMR7676BDP3Y10023]
  Product Name                                           = Ultrastar SN200
  Device Type                                            = NVMe Controller
  Device Path                                            = /dev/nvme0
  UID                                                    = 1C58SDM00003375FHUSMR7676BDP3Y10023
  Alias                                                  = @nvme0
  Vendor Name                                            = HGST
  Model Name                                             = HUSMR7676BDP3Y1
  Serial Number                                          = SDM00003375F
  Vendor ID                                              = 1C58
  Device ID                                              = 0023
  Firmware Version                                       = KNEGZ107
  Default Capacity                                       = 7680000000000
  Capacity                                               = 7681501126656
  NVM Subsystem UID                                      = 1C58SDM00003375FHUSMR7676BDP3Y1
  Namespace Count                                        = 1
  Multipath Support                                      = false
  Multipath Mode                                         = Single Port
  Namespace Sharing Support                              = false
  Encryption Support                                     = false
  Encryption Mode                                        = No Encryption
  UEFI Driver Version                                    = 1c58-0023-105c
  SBL Version                                            = KNEGZ107_SFRF_R01_Q32_S02_N080_D4_G08_X4
  PCI Domain Number                                      = 0000
  PCI Bus Number                                         = 01
  PCI Device Number                                      = 00
  PCI Function Number                                    = 0
  PCI Max Link Speed                                     = PCIe 3.0 (8.0 GT/s)
  PCI Max Link Width                                     = x4
  PCI Current Link Speed                                 = PCIe 2.0 (5.0 GT/s)
  PCI Current Link Width                                 = x4
  Device Status                                          = Ready
  Thermal Throttling Status                              = Enabled
  Beacon LED Enabled                                     = false
  Life Gauge                                             = 100
  System Area Percentage Used                            = 0
  Power Consumption                                      = 8.762000
  Data Units Read                                        = 726
  Data Units Written                                     = 4294987
  Host Read Commands                                     = 14335
  Host Write Commands                                    = 2098397
  Available Spare                                        = 100
  Available Spare Threshold                              = 10
  Available Spare Critical Warning                       = false
  Percentage Used                                        = 0
  Temperature                                            = 66
  Temperature Threshold                                  = 66
  Temperature Critical Warning                           = true
  Device Reliability Critical Warning                    = false
  Media Read Only Mode Critical Warning                  = false
  Volatile Memory Backup Device Failure Critical Warning = false
  Controller Busy Time                                   = 95
  Power Cycles                                           = 40
  Power On Hours                                         = 66
  Unsafe Shutdowns                                       = 17
  Media And Data Integrity Errors                        = 134
  Error Information Log Entries                          = 57
  SMART Status                                           = Critical Warning
  Total Re-Writes                                        = 0
  Total Write Bytes Processed                            = 157556789531520
  Total Uncorrected Write Errors                         = 0
  Flash Write Commands                                   = 2217860213
  Read Errors Corrected Without Delays                   = 41428073494
  Read Errors Corrected With Possible Delays             = 0
  Total Re-Reads                                         = 0
  Total Read Errors Corrected                            = 41428073494
  Total Read Bytes Processed                             = 183949128884520
  Total Soft Read Errors                                 = 0
  Total Uncorrected Read Errors                          = 0
  Flash Read Commands                                    = 41429983983
  Total XOR Recovered                                    = 0
  Total Corrected Bit Count                              = 459605774502
  Program Failure Count                                  = 0
  Normalized Program Failure Count                       = 0
  Total Uncorrected Erase Errors                         = 0
  Flash Erase Commands                                   = 8246147
  System Area Erase Failure Count                        = 0
  Normalized System Area Erase Failure Count             = 0
  User Data Erase Failure Count                          = 0
  Normalized User Data Erase Failure Count               = 0
  Manufacturers Defect Count                             = 8166
  Grown Defect Count                                     = 0
  Normalized Grown Defect Count                          = 0
  Minimum Erase Count Of User Data                       = 25
  Maximum Erase Count of User Data                       = 33
  Maximum Erase Count of System Data                     = 53
  Block Refresh Count                                    = 0
  End To End Correction Counts Detected                  = 0
  End To End Correction Counts Corrected                 = 0
  Host Read Blocks                                       = 105274413613
  Average Read Size                                      = 1830.382417
  Host Read Cache Hit Commands                           = 9026090
  Host Read Cache Hit Percentage                         = 15
  Host Read Cache Hit Blocks                             = 5156938506
  Average Read Cache Hit Size                            = 571.336925
  Host Read Commands Stalled                             = 361867
  Host Read Commands Stalled Percentage                  = 0
  Host Write Blocks                                      = 90304541110
  Average Write Size                                     = 1795.725750
  Host Write Odd Start Commands                          = 22348351
  Host Write Odd Start Commands Percentage               = 44
  Host Write Odd End Commands                            = 22742363
  Host Write Odd End Commands Percentage                 = 45
  Host Write Commands Stalled                            = 459463
  Host Write Commands Stalled Percentage                 = 0
  NAND Read Commands                                     = 10446694257
  NAND Read Blocks                                       = 8161624597456
  Average NAND Read Size                                 = 781.263852
  NAND Write Commands                                    = 53507007191
  NAND Write Blocks                                      = 6848896920448
  Average NAND Write Size                                = 128.000000
  NAND Read Before Write                                 = 31950999
  Read Error Rate                                        = 0.999954
  Processor Power Consumption                            = 7279
  NAND Power Consumption                                 = 1421
  Number of I/O Completion Queues Allocated              = 128
  Number of I/O Submission Queues Allocated              = 128
  Maximum Temperature                                    = 70
  Minimum Temperature                                    = 14
  Total Time Over Reference Temperature                  = 122
  Power On Time When At Max Temperature                  = 69
  Power On Time When At Min Temperature                  = 0
  Main Sensor Temperature                                = 66
  Inlet Sensor Temperature                               = 55
  Daughterboard 1 Sensor Temperature                     = 65
  Daughterboard 2 Sensor Temperature                     = 65506
  Thermal Throttling Event Count                         = 2
  Firmware Download Number                               = 0
  PFA Counter                                            = 0
  Total Correctable PCIe Error Count                     = 22118

[000CCA0B01FA2180]
  Device Type                                            = NVMe Namespace
  Device Path                                            = /dev/nvme0n1
  UID                                                    = 000CCA0B01FA2180
  Alias                                                  = @nvmens0
  Parent Product Name                                    = Ultrastar SN200
  Parent Device Path                                     = /dev/nvme0
  Parent UID                                             = 1C58SDM00003375FHUSMR7676BDP3Y10023
  NVM Subsystem UID                                      = 1C58SDM00003375FHUSMR7676BDP3Y1
  Namespace ID                                           = 1
  Size                                                   = 7801524581760
  Sector Count                                           = 15002931888
  Sector Size                                            = 512
  Metadata Size                                          = 8
  DIF Level                                              = None
  Shared                                                 = false
  Private                                                = false
  Device Status                                          = Ready
  Life Gauge                                             = 100
  Data Units Read                                        = 726
  Data Units Written                                     = 4294987
  Host Read Commands                                     = 14335
  Host Write Commands                                    = 2098397
  Available Spare                                        = 100
  Available Spare Threshold                              = 10
  Available Spare Critical Warning                       = false
  Percentage Used                                        = 0
  Temperature                                            = 66
  Temperature Critical Warning                           = true
  Device Reliability Critical Warning                    = false
  Media Read Only Mode Critical Warning                  = false
  Volatile Memory Backup Device Failure Critical Warning = false
  Controller Busy Time                                   = 95
  Power Cycles                                           = 40
  Power On Hours                                         = 66
  Unsafe Shutdowns                                       = 17
  Media And Data Integrity Errors                        = 134
  Error Information Log Entries                          = 57
  SMART Status                                           = Critical Warning
 

c steff

Member
Apr 18, 2019
48
6
8
get-info command
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm get-info -u 000CCA0B01FA2180
[000CCA0B01FA2180]
  Device Type                   = NVMe Namespace
  Device Path                   = /dev/nvme0n1
  UID                           = 000CCA0B01FA2180
  Alias                         = @nvmens0
  Parent Product Name           = Ultrastar SN200
  Parent Device Path            = /dev/nvme0
  Parent UID                    = 1C58SDM00003375FHUSMR7676BDP3Y10023
  NVM Subsystem UID             = 1C58SDM00003375FHUSMR7676BDP3Y1
  Namespace ID                  = 1
  Size                          = 7801524581760
  Sector Count                  = 15002931888
  Sector Size                   = 512
  Metadata Size                 = 8
  DIF Level                     = None
  Shared                        = false
  Private                       = false
get-state command
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm get-state -u 000CCA0B01FA2180
[000CCA0B01FA2180]
  Device Type                 = NVMe Namespace
  Device Path                 = /dev/nvme0n1
  UID                         = 000CCA0B01FA2180
  Alias                       = @nvmens0
  Device Status               = Ready
  Life Gauge                  = 100
get-capabilities command
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm get-capabilities -u 000CCA0B01FA2180
[000CCA0B01FA2180]
  Device Type  = NVMe Namespace
  Device Path  = /dev/nvme0n1
  UID          = 000CCA0B01FA2180
  Alias        = @nvmens0
  Command      = alias
  Command      = format
  Command      = generate-report
  Command      = get-capabilities
  Command      = get-feature
  Command      = get-info
  Command      = get-log-page
  Command      = get-smart
  Command      = get-state
  Command      = monitor
 

c steff

Member
Apr 18, 2019
48
6
8
after a lot of testing with all commands only those in get-capabilities work with this SSD and below is help format command
anything I can try with the format command?
format simple does a 1 second format but I still cannot do much with the SSD as before
Code:
ubuntu@ubuntu:~/Downloads$ sudo /opt/HGST/hdm/bin/hdm help format
usage: hdm format [-o, --output-format FORMAT] [-c, --config PATH] 
                  {-u, --uid UID | -p, --path PATH | -a, --alias ALIAS} [--sector-size SIZE] [--dif-level LEVEL] 
                  [--metadata-size SIZE] [--protection-interval NUM] [--force] 

Performs a low-level format on devices.

OPTIONS:
   -o, --output-format FORMAT  
                      Output format for the command.  Valid output formats are "text", "mini", "csv", and "json". The
                      default is "mini".
   -c, --config PATH   Path to the HDM configuration file.
   -u, --uid UID       Device unique identifier.
   -p, --path PATH     Platform path of the device as shown in the output of hdm scan.
   -a, --alias ALIAS   User supplied name for the device.
   --sector-size SIZE  Size of each sector in bytes.
   --dif-level LEVEL   Level of DIF protection.
   --metadata-size SIZE  
                      Size of metadata in bytes.
   --protection-interval NUM  
                      Number of protection information intervals for this operation.
  --force            Bypasses all confirmation messages.
 

c steff

Member
Apr 18, 2019
48
6
8
I see HDM format is identical to NVME FORMAT
I did: hdm format --sector-size 512 --metadata-size 0 --dif-level 0 /dev/nvme0
then created partition table and new ntfs partition with gparted
and now windows recovery environment seems to see the partition and write onto it (full windows doesn't work yet, gives me some blue screen every time and I'm working on it)
I will check what diskpart shows now
 
Last edited:

c steff

Member
Apr 18, 2019
48
6
8
WOW windows now works and shows the SSD in disk management and I can write to/delete data on the partition!!! finally!
I will try to do the same to the 2nd SSD
I am wondering if NTFS format is losing so much space as ssd of 7680GB shows after format 7153GB

*edit: I did the same for 2nd SSD and seems to be fine as well

Thanks everyone that posted here with ideas

I am hoping there won't be any issues with the megaraid card as I want to use the 2x ssds in raid 0 with this raid card in a small home server
 
Last edited:

c steff

Member
Apr 18, 2019
48
6
8
diskpart in windows

Code:
DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B        *
  Disk 1    Online         7153 GB      0 B        *
I found a website to calculate actual formatted size of a drive so yes 7680GB is actual usable size 7153GB
 

rivet

New Member
Jan 12, 2019
20
6
3
diskpart in windows

Code:
DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B        *
  Disk 1    Online         7153 GB      0 B        *
I found a website to calculate actual formatted size of a drive so yes 7680GB is actual usable size 7153GB
That looks a lot better. I'm glad you got it working.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,990
1,567
113
CA
Might I suggest updating post #1 so others aren't confused by the "SAS SSD" wording?
You could edit post #1 and put "EDIT: ...." in red and mention what you did\found out, etc, should anyone else stumble upon this thread it would be helpful :D
 

c steff

Member
Apr 18, 2019
48
6
8
Might I suggest updating post #1 so others aren't confused by the "SAS SSD" wording?
You could edit post #1 and put "EDIT: ...." in red and mention what you did\found out, etc, should anyone else stumble upon this thread it would be helpful :D
I will do shortly
 
  • Like
Reactions: T_Minus

c steff

Member
Apr 18, 2019
48
6
8
Might I suggest updating post #1 so others aren't confused by the "SAS SSD" wording?
You could edit post #1 and put "EDIT: ...." in red and mention what you did\found out, etc, should anyone else stumble upon this thread it would be helpful :D
done, please check if is clear enough :)
 

c steff

Member
Apr 18, 2019
48
6
8
if anyone is still reading this thread, I just researched pcie raid cards for nvme drives and this doesn't exist so that means the megaraid does raid only for sata/sas not nvme ?! software raid with windows server storage spaces seems the way to go unless the info I read is old and raid nvme cards exist? what is the point for such an expensive card to use with nvme ssds if no raid, I could use the m.2 nvme adapter + a pcie nvme adapter to use 2x nvme ssds in a small home server