I think I broke mkfs?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

Azelphur

New Member
Nov 23, 2023
4
3
3
Hi folks.

Satirical title aside, when I try to run mkfs.xfs (or ext4), my drive disconnects.

Code: 
$ sudo mkfs.xfs -f /dev/sdj1

meta-data=/dev/sdj1              isize=512    agcount=13, agsize=268435455 blks

         =                       sectsz=4096  attr=2, projid32bit=1

         =                       crc=1        finobt=1, sparse=1, rmapbt=1

         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1

         =                       exchange=1   metadir=0

data     =                       bsize=4096   blocks=3394240000, imaxpct=5

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=1

log      =internal log           bsize=4096   blocks=521728, version=2

         =                       sectsz=4096  sunit=1 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

         =                       rgcount=0    rgsize=0 extents

         =                       zoned=0      start=0 reserved=0

mkfs.xfs: libxfs_device_zero write failed: Remote I/O error

This comes up in dmesg:
Code: 
$ dmesg | tail -n 55

[   34.965824] /proc/cgroups lists only v1 controllers, use cgroup.controllers of root cgroup for v2 info
[   44.978743] systemd-journald[441]: Time jumped backwards, rotating.
[   71.348745] usb 7-1: reset high-speed USB device number 2 using xhci_hcd
[  128.221193] sd 0:0:5:0: device_block, handle(0x001f)
[  129.722893] sd 0:0:5:0: device_unblock and setting to running, handle(0x001f)
[  130.292228] sd 0:0:5:0: Power-on or device reset occurred
[  139.721048] sd 0:0:5:0: device_block, handle(0x001f)
[  139.722516] sd 0:0:5:0: [sdj] tag#2698 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722520] sd 0:0:5:0: [sdj] tag#2698 CDB: Write same(16) 93 00 00 00 00 00 60 07 00 fd 00 00 f6 07 00 00
[  139.722522] I/O error, dev sdj, sector 12888573928 op 0x9:(WRITE_ZEROES) flags 0x20000800 phys_seg 0 prio class 2
[  139.722548] sd 0:0:5:0: [sdj] tag#2697 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722549] sd 0:0:5:0: [sdj] tag#2697 CDB: Write same(16) 93 00 00 00 00 00 60 06 00 fe 00 00 ff ff 00 00
[  139.722550] I/O error, dev sdj, sector 12888049648 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722563] sd 0:0:5:0: [sdj] tag#2703 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722564] sd 0:0:5:0: [sdj] tag#2703 CDB: Write same(16) 93 00 00 00 00 00 60 00 01 04 00 00 ff ff 00 00
[  139.722565] I/O error, dev sdj, sector 12884903968 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722578] sd 0:0:5:0: [sdj] tag#2696 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722580] sd 0:0:5:0: [sdj] tag#2696 CDB: Write same(16) 93 00 00 00 00 00 60 03 01 01 00 00 ff ff 00 00
[  139.722581] I/O error, dev sdj, sector 12886476808 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722593] sd 0:0:5:0: [sdj] tag#2702 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722594] sd 0:0:5:0: [sdj] tag#2702 CDB: Write same(16) 93 00 00 00 00 00 60 04 01 00 00 00 ff ff 00 00
[  139.722595] I/O error, dev sdj, sector 12887001088 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722607] sd 0:0:5:0: [sdj] tag#2700 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722609] sd 0:0:5:0: [sdj] tag#2700 CDB: Write same(16) 93 00 00 00 00 00 60 05 00 ff 00 00 ff ff 00 00
[  139.722610] I/O error, dev sdj, sector 12887525368 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722621] sd 0:0:5:0: [sdj] tag#2704 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=9s
[  139.722623] sd 0:0:5:0: [sdj] tag#2704 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[  139.722624] I/O error, dev sdj, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[  139.722640] sd 0:0:5:0: [sdj] tag#2701 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722642] sd 0:0:5:0: [sdj] tag#2701 CDB: Write same(16) 93 00 00 00 00 00 60 02 01 02 00 00 ff ff 00 00
[  139.722643] I/O error, dev sdj, sector 12885952528 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  139.722655] sd 0:0:5:0: [sdj] tag#2699 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=12s
[  139.722657] sd 0:0:5:0: [sdj] tag#2699 CDB: Write same(16) 93 00 00 00 00 00 60 01 01 03 00 00 ff ff 00 00
[  139.722658] I/O error, dev sdj, sector 12885428248 op 0x9:(WRITE_ZEROES) flags 0x20000000 phys_seg 0 prio class 2
[  141.222778] sd 0:0:5:0: device_unblock and setting to running, handle(0x001f)
[  141.791033] sd 0:0:5:0: Power-on or device reset occurred
[  148.781802] sd 0:0:5:0: [sdj] tag#5577 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
[  148.781807] sd 0:0:5:0: [sdj] tag#5577 Sense Key : Illegal Request [current] [descriptor]
[  148.781809] sd 0:0:5:0: [sdj] tag#5577 Add. Sense: Invalid field in cdb
[  148.781811] sd 0:0:5:0: [sdj] tag#5577 CDB: Write(10) 2a 00 60 00 01 04 00 00 40 00
[  148.781812] critical target error, dev sdj, sector 12884903968 op 0x1:(WRITE) flags 0x8800 phys_seg 47 prio class 2
[  150.220876] sd 0:0:5:0: device_block, handle(0x001f)
[  150.222341] sd 0:0:5:0: [sdj] tag#5608 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=9s
[  150.222345] sd 0:0:5:0: [sdj] tag#5608 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
[  150.222346] I/O error, dev sdj, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  150.222366] Buffer I/O error on dev sdj, logical block 0, async page read
[  152.969848] sd 0:0:5:0: device_unblock and setting to running, handle(0x001f)
[  152.969873] sd 0:0:5:0: [sdj] tag#4226 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[  152.969876] sd 0:0:5:0: [sdj] tag#4226 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
[  152.969878] I/O error, dev sdj, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  152.969896] Buffer I/O error on dev sdj, logical block 0, async page read
[  152.992408] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000cca2904c43f1)
[  152.992414] mpt3sas_cm0: removing handle(0x001f), sas_addr(0x5000cca2904c43f1)
[  152.992416] mpt3sas_cm0: enclosure logical id(0x500062b20b81a740), slot(17)
[  152.992418] mpt3sas_cm0: enclosure level(0x0000), connector name(     )
The drive is then gone, the /dev/sdX node is removed and I have to unplug the drive and plug it back in again to get it to reappear.

Things I have tried:

- Replacing the drive. The drive is not faulty, maybe an issue with this specific model? Two different drives, two different serial numbers, same issue.
- Replacing the HBA. I swapped from a Perc H310 to a new 9305-24i, same issue.
- Replacing the cables. When I swapped the HBA, I alsohad to swap all the cables, same issue.
- Upgrading the firmware on the HBA, it's now 16.00.12.00
- `dd if=/dev/zero of=/dev/sdX` - it runs fine with no errors
- `dd if=/dev/sdX of=/dev/sdY` - dd a working drive onto a problematic one, it works, I'm able to mount the drive, read and write from it, etc. I could arguably give up and just dd images onto these drives and have them work, but that's quitter talk.

Only ideas I have:

- Maybe it's the firmware on the drive? But I'm not sure if it's possible to upgrade the firmware (The drives are WUH721414AL5204) - smartctl says that the "Revision" of the drive is DS05.
- Does my backplane have firmware? I'm using an X-Case XPC-424-HS - XPC-424-HS: 24 Bay Hotswap Storage Chassis

Hopefully somebody has some suggestions, thanks :)
 
Last edited:
A

Azelphur

New Member
Nov 23, 2023
4
3
3
I've self solved before the post passed moderation, but I'd like to leave this up in case anyone has the same problems in the future - I hope that's okay.

The answer was `sg_write_buffer -vvvvv -m 5 --in LDGNC400.bin /dev/sgX` to firmware update the drive. After I did that, it worked straight away.

I'm not sure if it's allowed for me to post LDGNC400.bin here, I found it online, but if you want a copy, feel free to contact me and I will provide it. Can message me here, or on Discord under the same username, my email is my user name at my user name dot com.
 
  • Like
Reactions: Stephan, tinfoil3d and Marsh
tinfoil3d

tinfoil3d

QSFP28
May 11, 2020
951
470
63
Japan
TY, interesting to note.
Seems very similar to how HGST 10TB Ultrastar HUH721010AL4200 behaves with firmware DD03.
Good to know WUH721414AL5204 DS05 is also acting up. Probably 2017 firmware as well. Mine was failing with ext4 while working with ext3. Unbelievable but...
I also performed upgrade to recent fw with similar command, https://forums.servethehome.com/ind...-ultrastar-he10-10tb-129-95.34395/post-505826
And that magically solved it. So I guess for future reference older HGST drives have this magic issue. What's manufacture date on yours?
 
  • Like
Reactions: Stephan
A

Azelphur

New Member
Nov 23, 2023
4
3
3
tinfoil3d said:
TY, interesting to note.
Seems very similar to how HGST 10TB Ultrastar HUH721010AL4200 behaves with firmware DD03.
Good to know WUH721414AL5204 DS05 is also acting up. Probably 2017 firmware as well. Mine was failing with ext4 while working with ext3. Unbelievable but...
I also performed upgrade to recent fw with similar command, https://forums.servethehome.com/ind...-ultrastar-he10-10tb-129-95.34395/post-505826
And that magically solved it. So I guess for future reference older HGST drives have this magic issue. What's manufacture date on yours?
Click to expand...
The manufacture date is 27 Jul 2021, and the label says FW: 2L0 :)
 
Last edited:
  • Wow
Reactions: tinfoil3d
You must log in or register to reply here.
Top Bottom