Can't upgrade firmware on Dell EMC 1.6 TB P4510 - Firmware VDV1VD05

[serverguy12]

Hi folks,

I'm a long-term STH reader, but first time poster!

Looking for advice from the community, I purchased 12x 1.6 TB P4510 (firmware VDV1VD05) and experiencing issues with the drives under sustained load on a dual EPYC 7003 platform, while failing simple fio benchmarks.

[ 107.044743] nvme nvme0: I/O tag 8 (e008) QID 86 timeout, completion polled
[ 107.808789] nvme nvme0: I/O tag 28 (101c) QID 74 timeout, completion polled
[ 110.888816] nvme nvme0: I/O tag 4 (7004) QID 72 timeout, completion polled
[ 143.661057] nvme nvme0: I/O tag 8 (4008) QID 80 timeout, completion polled

I have done an exhaustive kernel tuning, BIOS settings and different filesystems (ext4, ZFS, mdm) however can keep surfacing the nvme errors, which can pause a drive read/write for up to ~30 seconds which stalls IO. No smart issues on the drives, and I've tried 12 of them.

My conclusion is the drives need to be upgraded for the firmware, however the generic/Dell images are rejected in different stages (not supported drive), Intel MAS no longer supports the drives, and the `sst` command won't upgrade these either.

# sst show --ssd

- PHLJ037401DD2P0B NSID 1 -

Bootloader : 0145
Capacity : 1.60 TB (1,600,321,314,816 bytes)
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : VDV1VD05
FirmwareUpdateAvailable : No known update for SSD. If an update is expected, please contact your SSD Vendor representative about firmware update for this drive.
Index : 0
MaximumLBA : 3125627567 (1,600,321,314,816 bytes)
ModelNumber : INTEL SSDPE2KX016T8M 118033347
NamespaceId : 1
PercentOverProvisioned : 0.000000
ProductFamily : Intel SSD DC P4510 Series
SMARTEnabled : True
SectorDataSize : 512
SerialNumber : X

-

Any suggestions, or is there a direct (legacy) firmware update for these drives?

Note to self, I should have purchased one of these first, tested it, and ordered another 11!

 

[serverguy12]

Some extra context:

# lspci -vv -s a1:00.0
a1:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
Subsystem: Intel Corporation NVMe Datacenter SSD [3DNAND] SE 2.5" U.2 (P4510)
Physical Slot: 1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 152
NUMA node: 1
IOMMU group: 109

And trying to use intelmas:

# intelmas show -intelssd

- 0 Intel SSD DC P4510 Series PHLJ037401DD2P0B -

Capacity : 1.46 TB (1,600,321,314,816 bytes)
DevicePath : /dev/nvme0n1
DeviceStatus : Unknown
Firmware : VDV1VD05
Index : 0
MaximumLBA : 3125627567
ModelNumber : INTEL SSDPE2KX016T8M 118033347
ProductFamily : Intel SSD DC P4510 Series
SectorDataSize : 512
SerialNumber : X
Solidigm : This drive is no longer supported.

 

[nabsltd]

I suspect part of the issue is that the DC P4510 didn't have a 1.6TB size (at least according to Intel's product brief). I know there are tons of them available on eBay, but all seem to be Dell EMC OEM drives.

Your only real chance might be to get a Dell server of the same age as the drives, install them and use Dell's update utility.

 

[serverguy12]

Thanks for the reply @nabsltd - I've logged a support ticket with Solidigm, have some chatter back and forth, I will see if there is a firmware update available.

The more I have dug into this issue, seems like ZFS across multiple of these drives triggers the "nvme nvme0: I/O tag 8 (e008) QID 86 timeout, completion polled" errors very easily, and the same with using mdm across 2 of the drives.

However, while using a single drive with ext4, seems this issue does not crop up as easily. My sense is ZFS and mdm must use a larger queue depth for write/reads, which is triggering the VDV1VD05 to have a mild heart attack.

 

[UhClem]

Thanks for the reply @nabsltd - I've logged a support ticket with Solidigm, have some chatter back and forth, I will see if there is a firmware update available.

The more I have dug into this issue, seems like ZFS across multiple of these drives triggers the "nvme nvme0: I/O tag 8 (e008) QID 86 timeout, completion polled" errors very easily, and the same with using mdm across 2 of the drives.

However, while using a single drive with ext4, seems this issue does not crop up as easily. My sense is ZFS and mdm must use a larger queue depth for write/reads, which is triggering the VDV1VD05 to have a mild heart attack.

I suggest you experiment with fio to discover an exact, simple, and easilly reproducable command (using a single/raw /dev/nvmeXn1 drive) to submit with your support request.