Same HDD dropping from system and ZFS on HPE DL380 Gen9

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I have a homelab rack setup with a few servers and my HDD NAS server is having an odd issue and I can't easily pinpoint if the issue is the controller, the slot, or the HDD itself.

Problem
The same HDD in drive bay 1 will be removed from the system. I don't see it drop in system logs but I see it re-attach in the system logs. ZFS doesn't usually re-add it back in to resilver it unless I physically remove the drive and pop it back into the system.

There doesn't seem to be any real correlation on what causes this, but it seems to be more likely to happen on heavy IO tasks like ZFS scrubs or a lot of file transfer movement.

Nothing seems to show that the drive is obviously defective as SMART shows normal and ZFS has no checksum errors. The array controller doesn't think anything is wrong either.


Environment
Server: HPE DL380 Gen9 15LFF
Controller: P440ar
OS: Proxmox 7.4
ZFS: RAID-Z 3 on 15 drives
HDDs: 15 x 10TB HGST Ultrastar He10 HUH721010ALE604 (all refurbished)

1688771577856.png
1688771979736.png
1688772074656.png
1688772100546.png

I will probably try getting more of these drives to test if the issue persists, as I need to have some spares on hand anyway, but I'm wondering if anyone has encountered this before.