# sg_inq -d /dev/sg0
standard INQUIRY:
PQual=0 Device_type=13 RMB=0 LU_CONG=0 version=0x05 [SPC-3]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=1]
EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=66 (0x42) Peripheral device type: enclosure services device
Vendor identification: DELL
Product identification: MD1200
Product revision level: 1.06
Version descriptors:
SAM-3 (no version claimed)
SAS-1.1 (no version claimed)
SPC-3 (no version claimed)
SES-2 (no version claimed)
# sg_inq -d -H /dev/sg0
00 0d 00 05 12 3d 00 90 00 44 45 4c 4c 20 20 20 20 ....=...DELL
10 4d 44 31 32 30 30 20 20 20 20 20 20 20 20 20 20 MD1200
20 31 2e 30 36 33 59 31 50 51 57 31 20 00 00 00 00 1.063Y1PQW1 ....
30 00 00 00 00 00 00 00 00 00 00 00 60 0c 00 03 00 ...........`....
40 03 e0 ..
# sg_inq -d /dev/sg0
standard INQUIRY:
PQual=0 Device_type=13 RMB=0 LU_CONG=0 version=0x05 [SPC-3]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=1]
EncServ=1 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=66 (0x42) Peripheral device type: enclosure services device
Vendor identification: DELL
Product identification: MD1200
Product revision level: 1.06
Version descriptors:
SAM-3 (no version claimed)
SAS-1.1 (no version claimed)
SPC-3 (no version claimed)
SES-2 (no version claimed)
# sg_inq -d -H /dev/sg0
00 0d 00 05 12 3d 00 d0 00 44 45 4c 4c 20 20 20 20 ....=...DELL
10 4d 44 31 32 30 30 20 20 20 20 20 20 20 20 20 20 MD1200
20 31 2e 30 36 33 59 31 50 51 57 31 20 00 00 00 00 1.063Y1PQW1 ....
30 00 00 00 00 00 00 00 00 00 00 00 60 0c 00 03 00 ...........`....
40 03 e0 ..
lsscsi -g | grep enclosu # Locate the /dev/sg? enclosure device
# Blink a led of a particular device (sdf in the case below)
sg_ses --sas-addr=`cat /sys/block/sdf/device/sas_address` --set=locate /dev/sg5
# Clear it
sg_ses --sas-addr=`cat /sys/block/sdf/device/sas_address` --clear=locate /dev/sg5
Great work! I'm surprised there was no checksum byte that needed updating after changing the firmware contents, or perhaps there is but the raw xmodem flashing routine doesn't check it. Did you try flashing with sg_utils as well?I have figured out how to get the SES EncServ bit to present as enabled (at least on an enclosure programmed as MD1200). So that /sys/class/enclosure gets populated by the Linux kernel.
Short version:
One bit in the MD12_106.bin firmware image that @fohdeesha shared in post #7 needs editing. Change hex value 90 to D0 at location 0002B7DF. Then flash the modified image and make it active.
Long version of how I figured it out:
Sharing the details in case others see any issues with the approach. Or want to try the same method on other firmware versions. Or just to try and share/document fully.
As @neonclash mentioned in post #111, the ENCSERV bit is 0 when doing an inquiry of the enclosure. This frustrated me too because /sys/class/enclosure was not populated, even though SES-2 is supposed to be supported per sg3-utils sg_inq command:
I dumped the hex values for the same inquiry:Code:# sg_inq -d /dev/sg0 standard INQUIRY: PQual=0 Device_type=13 RMB=0 LU_CONG=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=1] EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0 [SPI: Clocking=0x0 QAS=0 IUS=0] length=66 (0x42) Peripheral device type: enclosure services device Vendor identification: DELL Product identification: MD1200 Product revision level: 1.06 Version descriptors: SAM-3 (no version claimed) SAS-1.1 (no version claimed) SPC-3 (no version claimed) SES-2 (no version claimed)
I used motivation from this bug report:Code:# sg_inq -d -H /dev/sg0 00 0d 00 05 12 3d 00 90 00 44 45 4c 4c 20 20 20 20 ....=...DELL 10 4d 44 31 32 30 30 20 20 20 20 20 20 20 20 20 20 MD1200 20 31 2e 30 36 33 59 31 50 51 57 31 20 00 00 00 00 1.063Y1PQW1 .... 30 00 00 00 00 00 00 00 00 00 00 00 60 0c 00 03 00 ...........`.... 40 03 e0 ..
![]()
Dell 1220 and /sys/class/enclosure · Issue #31 · ewwhite/zfs-ha
First of all, thanks for this beautiful and elegant solution. Using it for my XenServer pool and works great! The only downside is that with Dell MD1220 there's no way to identify drives because /s...github.com
Where the user had recompiled the kernel with modifications to ses.c in order to "force" an over-ride for these enclosures based on string match for MD1200 or MD1220. And /sys/class/enclosure worked (at least partially) for them.
I also used this document:
Which has this figure on page 94:
View attachment 26945
Notice that the ENCSERV bit is part of byte 6 of the inquiry.
In the hex inquiry output I captured above, I could see that byte 6 was value 90 - which would make the ENCSERV bit 0 (since binary value would be 10010000). I decided to try and find the location of this data in the enclosure, and change it to D0 value (11010000) - so that only the ENCSERV bit would be different.
I had previously captured some "_flashdump 0 100000" output to try and map the different regions. I searched for part of the inquiry hex data "05 12 3d 00 90" and found it in the line at offset 1402B7D0 and that was part of the EMM image region 0:
View attachment 26946
So I opened the MD12_106.bin file @fohdeesha made available in post #7 with HxD editor. And searched for that hex data. I found it in the line at offset 0002B7D0. The specific byte I was looking for was at location 0002B7DF:
View attachment 26947
I made a copy of the firmware and changed that location's data from 90 to D0 using HxD editor:
View attachment 26948
Then flashed the modified firmware with _boot command so that all other EMM firmware regions would be wiped and there would be no conflict with multiple 1.06 versions present in different regions, but with different data.
After restarting the enclosure, it seems to work. Here is the new inquiry showing "EncServ=1" in the reply:
And D0 as the hex value in byte 6 now instead of 90 like before:Code:# sg_inq -d /dev/sg0 standard INQUIRY: PQual=0 Device_type=13 RMB=0 LU_CONG=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=1] EncServ=1 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0 [SPI: Clocking=0x0 QAS=0 IUS=0] length=66 (0x42) Peripheral device type: enclosure services device Vendor identification: DELL Product identification: MD1200 Product revision level: 1.06 Version descriptors: SAM-3 (no version claimed) SAS-1.1 (no version claimed) SPC-3 (no version claimed) SES-2 (no version claimed)
Code:# sg_inq -d -H /dev/sg0 00 0d 00 05 12 3d 00 d0 00 44 45 4c 4c 20 20 20 20 ....=...DELL 10 4d 44 31 32 30 30 20 20 20 20 20 20 20 20 20 20 MD1200 20 31 2e 30 36 33 59 31 50 51 57 31 20 00 00 00 00 1.063Y1PQW1 .... 30 00 00 00 00 00 00 00 00 00 00 00 60 0c 00 03 00 ...........`.... 40 03 e0 ..
And /sys/class/enclosure is populated and seems to be working for me so far. I can query the device in a slot, set and clear the slot's fault/locate LEDs (which was my main goal). The encled script (from GitHub - amarao/sdled: Control fault/locate indicators in disk slots in enclosures (SES devices)) now works as well.
One issue I'm still facing - I have a few disks which suffer from the issue that was mentioned in the zfs-ha bug. The device tree for the disk is present in /sys/class/enclosure/H:C:T:L/Slot X/device when first starting the enclosure. When I unplug, the device tree disappears as expected. Then when I hot plug them, they don't re-populate the "device" tree. But I have other drives which work fine. The drives with problems are also showing some kernel messages when hot plugged, so I think it is a drive problem, not an enclosure problem. They are Seagate IBM-branded drives and I am attempting a format of them with SeaTools as mentioned in this other STH thread:
And further disclosure - I only have a basic/simple setup with one enclosure in unified mode with a single EMM installed. So no testing of multiple enclosures, multipath, split mode, etc. I used a Proxmox based system with kernel version 5.15.83-1-pve to do the testing along with LSI 9207-8e HBA and mpt3sas driver.
I hope some others are able to test this and I hope that it works for them as well.
Thanks.Great work! I'm surprised there was no checksum byte that needed updating after changing the firmware contents, or perhaps there is but the raw xmodem flashing routine doesn't check it. Did you try flashing with sg_utils as well?
**** Devil Startup Complete (Based on vendor drop 00.00.63.00) ****
BlueDress.106.000 >_ver
********************* Bench Build **********************
Build Owner : GitHub
Build Number : 000
Build Version : 106
Build Date : Mon Aug 24 12:28:59 2015
FPGA Revision : A9
Board Revision : 8
CPLD Revision : 7
**************Firmware Image Information****************
Active--->Image Region 0 (Always Boot)
Revision : 0.0.63.0
Dell Version : 1.06
Total Image Size : 000414fc [267516]
Fast Boot : Yes
Image Address : 0x14000000
RegionOffset : 0x00000000
RegionSize : 0x00080000
RegionType : 0x00000000
Image Region 1
Revision : 255.255.255.255
Dell Version : ▒▒▒▒
Total Image Size : ffffffff [-1]
Fast Boot : No
Image Address : 0x14080000
RegionOffset : 0x00080000
RegionSize : 0x00080000
RegionType : 0x00000001
Image Region 2
Revision : 255.255.255.255
Dell Version : ▒▒▒▒
Total Image Size : ffffffff [-1]
Fast Boot : No
Image Address : 0x14100000
RegionOffset : 0x00100000
RegionSize : 0x00080000
RegionType : 0x00000002
********************************************************
# lsscsi -g |grep MD
[10:0:11:0] enclosu DELL MD1200 1.06 - /dev/sg8
# sg_write_buffer -b 4k -m dmc_offs_save -I MD12_106_JUmod.bin /dev/sg8
# echo $?
0
BlueDress.106.000 >_ver
********************* Bench Build **********************
Build Owner : GitHub
Build Number : 000
Build Version : 106
Build Date : Mon Aug 24 12:28:59 2015
FPGA Revision : A9
Board Revision : 8
CPLD Revision : 7
**************Firmware Image Information****************
Active--->Image Region 0 (Always Boot)
Revision : 0.0.63.0
Dell Version : 1.06
Total Image Size : 000414fc [267516]
Fast Boot : Yes
Image Address : 0x14000000
RegionOffset : 0x00000000
RegionSize : 0x00080000
RegionType : 0x00000000
Image Region 1
Revision : 255.255.255.255
Dell Version : ▒▒▒▒
Total Image Size : ffffffff [-1]
Fast Boot : No
Image Address : 0x14080000
RegionOffset : 0x00080000
RegionSize : 0x00080000
RegionType : 0x00000001
Image Region 2
Revision : 0.0.63.0
Dell Version : 1.06
Total Image Size : 000414fc [267516]
Fast Boot : Yes
Image Address : 0x14100000
RegionOffset : 0x00100000
RegionSize : 0x00080000
RegionType : 0x00000002
********************************************************
**** Devil Startup Complete (Based on vendor drop 00.00.63.00) ****
BlueDress.106.000 >_ver
********************* Bench Build **********************
Build Owner : GitHub
Build Number : 000
Build Version : 106
Build Date : Mon Aug 24 12:28:59 2015
FPGA Revision : A9
Board Revision : 8
CPLD Revision : 7
**************Firmware Image Information****************
Active--->Image Region 0 (Always Boot)
Revision : 0.0.63.0
Dell Version : 1.06
Total Image Size : 000414fc [267516]
Fast Boot : Yes
Image Address : 0x14000000
RegionOffset : 0x00000000
RegionSize : 0x00080000
RegionType : 0x00000000
Image Region 1
Revision : 255.255.255.255
Dell Version : ▒▒▒▒
Total Image Size : ffffffff [-1]
Fast Boot : No
Image Address : 0x14080000
RegionOffset : 0x00080000
RegionSize : 0x00080000
RegionType : 0x00000001
Image Region 2
Revision : 255.255.255.255
Dell Version : ▒▒▒▒
Total Image Size : ffffffff [-1]
Fast Boot : No
Image Address : 0x14100000
RegionOffset : 0x00100000
RegionSize : 0x00080000
RegionType : 0x00000002
********************************************************
As follow-up to the issue with the "device" symlink not being created and kernel messages I was facing. Two things I have figured out so far and wanted to share.One issue I'm still facing - I have a few disks which suffer from the issue that was mentioned in the zfs-ha bug. The device tree for the disk is present in /sys/class/enclosure/H:C:T:L/Slot X/device when first starting the enclosure. When I unplug, the device tree disappears as expected. Then when I hot plug them, they don't re-populate the "device" tree. But I have other drives which work fine. The drives with problems are also showing some kernel messages when hot plugged, so I think it is a drive problem, not an enclosure problem. They are Seagate IBM-branded drives and I am attempting a format of them with SeaTools as mentioned in this other STH thread:
=== START OF INFORMATION SECTION ===
Vendor: IBM-ESXS
Product: ST10000NM0226 E
Revision: ECG6
Compliance: SPC-4
User Capacity: 9,931,038,130,176 bytes [9.93 TB]
Logical block size: 4096 bytes
Formatted with type 2 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0xXXXXXXXXXXXXXXXX
Serial number: XXXXXXXXXXXXXXXXXXXX
Device type: disk
Transport protocol: SAS (SPL-3)
Feb 2 15:44:24 asrock kernel: [ 2.525106] sd 4:0:0:0: [sda] tag#2924 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
Feb 2 15:44:24 asrock kernel: [ 2.525110] sd 4:0:0:0: [sda] tag#2924 Sense Key : Illegal Request [current]
Feb 2 15:44:24 asrock kernel: [ 2.525112] sd 4:0:0:0: [sda] tag#2924 Add. Sense: Logical block reference tag check failed
Feb 2 15:44:24 asrock kernel: [ 2.525114] sd 4:0:0:0: [sda] tag#2924 CDB: Read(32)
Feb 2 15:44:24 asrock kernel: [ 2.525116] sd 4:0:0:0: [sda] tag#2924 CDB[00]: 7f 00 00 00 00 00 00 18 00 09 20 00 00 00 00 00
Feb 2 15:44:24 asrock kernel: [ 2.525118] sd 4:0:0:0: [sda] tag#2924 CDB[10]: 90 83 b7 f0 90 83 b7 f0 00 00 00 00 00 00 00 01
if (refresh)
ses_enclosure_data_process(edev, edev_sdev, 0);
mdelay(20000);
if (refresh)
ses_enclosure_data_process(edev, edev_sdev, 0);
#include <linux/delay.h>
**** Devil Startup Complete (Based on vendor drop 00.00.63.00) ****
BlueDress.106.000 >
**** Devil Startup Complete (Based on vendor drop 00.00.63.00) ****
BlueDress.106.000 >
**** Devil Startup Complete (Based on vendor drop 00.00.63.00) ****
BlueDress.106.000 >