Those HGST 3.84Tb SN260 drives and Cisco firmware...

[ElCoyote_]

I recently got a couple HUSMR7638BHP3Y1´s ( 3.84Tb ) from ebay.. Things went fine during transport but when I got the drives, I had the displeasure to notice that:
1) they had a Cisco UCS firmware (I had previously obtain HGST firmwares).
2) the v111 firmware was too old and did not work in my Dell PowerEdge T640 causing a PCIe error. I knew exactly what the error was and how to solve it: Update to the latest firmware (v122).

Here's how it looked (on the T630, which -did- boot without issues):

[root@palanthas ~]# nvme list
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          SDM0000----------        UCSC-NVME-H38401                         1           3.84  TB /   3.84  TB    512   B +  0 B   KNCCD111 <====
/dev/nvme1n1          SDM0000----------        UCSC-NVME-H38401                         1           3.84  TB /   3.84  TB    512   B +  0 B   KNCCD111 <====
/dev/nvme2n1          SDM0000----------        HUSMR7676BHP3Y1                          1           7.68  TB /   7.68  TB       4 KiB +  0 B   KNGND122
/dev/nvme3n1          SDM0000----------        HUSMR7676BHP3Y1                          1           7.68  TB /   7.68  TB       4 KiB +  0 B   KNGND122

Simple, I thought.. to fix it I just needed to obtain firmware from Cisco and flash my drives.
It turned out to be not so simple (including a few hackish move) but it worked. I am posting here in case it helps someone..

 

[ElCoyote_]

Issue #1 : How to do I get the firmware for the drives?

Solution #1: Register for an account on id.cisco.com and download the following ISO image: ucs-c125-huu-4.2.2b.iso
Apparently this drive was in a specific type of Cisco UCS chassis and the googled' document I found configured that v122 was the latest firmware for the drives: https://www.cisco.com/c/en/us/td/do...firmware-files_4_2/m_firmware_files_4_2_2.pdf

The ISO itself was: Software Download - Cisco Systems

 

[ElCoyote_]

Issue #2: What the H... is this encrypted firmware file?

Solution #2: No worries, the key to decryption is on the ISO too.

Once I started looking at the firmware files present on the ISO image, I found that everything seems encrypted:

# mount -o loop ucs-c125-huu-4.2.2a.iso /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.
# unsquashfs -f -d /tmp/fwm ucs-c125-huu-container-4.2.2a.squashfs
Parallel unsquashfs: Using 88 processors
6181 inodes (8606 blocks) to write

[=====================================================================================================================================================\] 8606/8606 100%

created 5826 files
created 627 directories
created 353 symlinks
created 0 devices
created 0 fifos
# find /tmp/fwm/|grep 122
/tmp/fwm/firmware/Common/HGST__Inc_/Ultrastar_SN200_Series_NVMe_SSD/KNCCD122
/tmp/fwm/firmware/Common/HGST__Inc_/Ultrastar_SN200_Series_NVMe_SSD/KNCCD122/KNCCD122.bin

# file /tmp/fwm/firmware/Common/HGST__Inc_/Ultrastar_SN200_Series_NVMe_SSD/KNCCD122/KNCCD122.bin
/tmp/fwm/firmware/Common/HGST__Inc_/Ultrastar_SN200_Series_NVMe_SSD/KNCCD122/KNCCD122.bin: openssl enc'd data with salted password

Luckily, the file summary provided by the 'file' Linux command gives it away and made me look in other places on the ISO to figure out if there were something to be learned. Answer: there was in the other squashfs archive the key to decryption:

unsquashfs -f -d /tmp/rootfs /mnt/rootfs.img
grep -r openssl /tmp/fwm/
strings -a /tmp/rootfs/usr/sbin/decrypt-file|grep openssl
openssl enc -aes-256-cbc -d -md sha256 -pbkdf2 -in KNCCD122.bin -out  KNCCD122.out  -k zfguijkophju@*%1]
mv KNCCD122.out KNCCD122.gz
gunzip *gz

So now I had a firmware file that looked similar to the HGST firmwares which I had used previously:
Cisco:

# strings -a KNCCD122.bin |grep SYS
SYS: Crash Occurred
SYS: Background Thread finished
SYS: Go into SBL mode
SYS: Inited
SYS: Inited (Post Crash)
SYS: ERROR - System initialization failed (Incompatible SBL)
SYS: ERROR - System initialization failed (Incompatible ASIC revision)
bSYSB

HGST:

# strings -a KNGND122.bin |grep SYS
SYS: Crash Occurred
SYS: Background Thread finished
SYS: Go into SBL mode
SYS: Inited
SYS: Inited (Post Crash)
SYS: ERROR - System initialization failed (Incompatible SBL)
SYS: ERROR - System initialization failed (Incompatible ASIC revision)
bSYSB

 

[ElCoyote_]

Issue #3 : The resulting firmware is not accepted by the 'nvme' utility.
Solution #3: Please forgive me... but it worked.

Here's what I had:

HGST firmware:

# ls -la KNGND122.bin
-rw-r--r--. 1 root root 1762048 Sep 17  2020 KNGND122.bin

Decrypted/gunzip'ed Cisco firmware:
# ls -la KNCCD122.bin
-rw-r--r--. 1 root root 1761310 Jul 22 18:50 KNCCD122.bin

Size suspiciously similar but the HGST firmware was about 700bytes larger..
Also, 'od' showed at the end of the Cisco firmware an sha256 checksum which I did not like too much. That thing wasn't present in the HGST firmware.

This was when I hit my latest roadblock with the Cisco firmware.. the procedure which I had used on the HGST firmware no longer worked:

# nvme fw-download /dev/nvme0 -f KNCCD122.bin
Invalid size:1761310 for f/w image

So I decided to do this:
1) strip the Cisco .BIN from its bizarre sha256 32 bytes marker at the end.
2) use the larger HGST firmware as a container for a new payload where I'd place the Cisco firmware.

So here's what I did:

# Notes:
1761278=1761310-32 (sha256 sig marker)

cp ../firmwares/KNGND122.bin NEW
dd if=KNCCD122.bin bs=1 of=NEW count=1761278 conv=notrunc
# nvme fw-download /dev/nvme0 -f NEW
Firmware download success
# nvme fw-commit /dev/nvme0
Success committing firmware action:0 slot:0
Multiple Update Detected (MUD) Value: 0

( I had just bought 4 of these drives from ebay and I wanted the v122 firmware on them ).
After a very stressful dilemma, I flashed one, verified everything worked (discard, TRIM, using it, etc..), and then proceeded to flash the others.

End result (all NVMe's on v122, and reformated them to 4k):

[root@palanthas ~]# nvme list
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          SDM0000----------        UCSC-NVME-H38401                         1           3.84  TB /   3.84  TB       4 KiB +  0 B   KNCCD122 <====
/dev/nvme1n1          SDM0000----------        UCSC-NVME-H38401                         1           3.84  TB /   3.84  TB       4 KiB +  0 B   KNCCD122 <====
/dev/nvme2n1          SDM0000----------        HUSMR7676BHP3Y1                          1           7.68  TB /   7.68  TB       4 KiB +  0 B   KNGND122
/dev/nvme3n1          SDM0000----------        HUSMR7676BHP3Y1                          1           7.68  TB /   7.68  TB       4 KiB +  0 B   KNGND122

 

[ElCoyote_]

Also, I'm not a firmware person. I am -very- relieved this small hack worked and that it did not kill the drives. They appear to work 100% correctly (tested in multiple machines and multiple power cycles/data cycles/etc...). Even the Dell T640 is happy with them now that they are on v122.

 

[Gio]

Kudos to you sir @ElCoyote_

I purchased some Cisco X710-DA2 off ebay that came with a very old firmware and while trying to figure out how to extract the firmware files from the cisco bootable .iso I found out they were partially encrypted. Your notes here came in clutch and finding the decryption key.

Now I am trying to sort out how and which files matter for flashing the intel cards with this update but that's something I'll do this weekend. cheers

 

[mirrormax]

nice work, maybe upload the modded fw if anyone needs it in the future

 

[rsumperl]

Hello everyone,
Does anyone have a procedure for flashing these either in Windows or the EFI command line?

Thanks
Ray

 

[dontwanna]

Does anyone have a procedure for flashing these either in Windows or the EFI command line?

I don't know about Windows or EFI command line, but I just flashed UCSC-NVMEHW-H3200 (3.2 TB SN200 U.2) straight from the Supermicro motherboard BIOS (X9DR3). The disk wasn't recognized by OS, and I initially thought it was dead, but the motherboard's BIOS sees it and after updating the firmware (it came with KNCCD101) it's now recognized by operating systems as well, and working fine. Didn't even need any firmware modding in my case, the stock Cisco firmware (KNCCD122.bin) was accepted.

Big thanks to ElCoyote_ for this thread and help with the firmware!

 

[ozkan]

Hi, I need help with firmware for HUSMR7638BHP3Y1.

I have 2 of them branded HGST, one is used for a long time and need to be replaced. I made a mistake while updating firmware. There was 5 firmware slot as below;

frs1 : 0x313630504e47544b (KTGNP061)
frs2 : 0x313730504e474e4b (KNGNP071)
frs3 : 0x313630504e47544b (KTGNP061)
frs4 : 0x313630504e47544b (KTGNP061) *
frs5 : 0x303031444e474e4b (KNGND100)


I accidentally activated the 4th slot and now NVMe is not recognised by the system. When I checked from lspci and other tools, NVMe push wrong PCI class and device id.

dmesg:

[ 1.395815] pci 0000:5e:00.0: [1c58:0022] type 00 class 0xff0802
[ 1.395864] pci 0000:5e:00.0: reg 0x10: [mem 0xb8810000-0xb8813fff 64bit]
[ 1.395929] pci 0000:5e:00.0: reg 0x20: [mem 0xb8800000-0xb880ffff 64bit]
[ 1.439873] pci 0000:5e:00.0: Adding to iommu group 1

lspci :

5e:00.0 Unassigned class [ff08]: HGST, Inc. Device 0022 (prog-if 02)
Subsystem: Device feed:beef
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
NUMA node: 0
IOMMU group: 1
Region 0: Memory at b8810000 (64-bit, non-prefetchable) [disabled] [size=16K]
Region 4: Memory at b8800000 (64-bit, non-prefetchable) [disabled] [size=64K]
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x2 (downgraded)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [40] Vendor Specific Information: Len=30 <?>


Class ID should be : 010802
Vendor ID : correct
Device ID should be : 0023

While thinking about fault, I tried the same firmware with the old one at another server and same result. Now both NVMe drivers shown as Unassigned class [ff08]: HGST, Inc. Device 0022.

Windows, Linux 6, linux 3.3, linux 2.6, FreeBSD 12,13 same result for all. It's not recognised as NVMe or any drive, so I can not change the firmware slot anymore.

Is there any other way to activate another firmware slot. I'm also curious about USB port on the card. I think It's a UART, but I don't have any debug tool for UART.