Thanks! Here's what I see in dmesg. Not the same as yours, but I think you have pointed me in a good direction. I did update all the drive firmware, but not anything else. I think the problems actually started before the updates, but I will check the HBA out as well.
142.244535] blk_update_request: I/O error, dev sdfs, sector 268435416
[ 142.527517] sd 13:0:44:0: [sdfs] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 142.527532] sd 13:0:44:0: [sdfs] tag#0 Sense Key : Aborted Command [current]
[ 142.527539] sd 13:0:44:0: [sdfs] tag#0 Add. Sense: No additional sense information
[ 142.527560] sd 13:0:44:0: [sdfs] tag#0 CDB: Read(16) 88 00 00 00 00 00 0f ff ff d8 00 00 00 08 00 00
[ 142.527563] blk_update_request: I/O error, dev sdfs, sector 268435416
[ 142.527565] Buffer I/O error on dev sdfs, logical block 33554427, async page read
[ 142.810539] sd 13:0:44:0: [sdfs] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 142.810555] sd 13:0:44:0: [sdfs] tag#0 Sense Key : Aborted Command [current]
[ 142.810561] sd 13:0:44:0: [sdfs] tag#0 Add. Sense: No additional sense information
[ 142.810569] sd 13:0:44:0: [sdfs] tag#0 CDB: Read(16) 88 00 00 00 00 03 a3 81 2a a8 00 00 00 08 00 00
[ 142.810575] blk_update_request: I/O error, dev sdfs, sector 15628053160
[ 143.093557] sd 13:0:44:0: [sdfs] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 143.093573] sd 13:0:44:0: [sdfs] tag#0 Sense Key : Aborted Command [current]
[ 143.093580] sd 13:0:44:0: [sdfs] tag#0 Add. Sense: No additional sense information
[ 143.093587] sd 13:0:44:0: [sdfs] tag#0 CDB: Read(16) 88 00 00 00 00 03 a3 81 2a a8 00 00 00 08 00 00
[ 143.093593] blk_update_request: I/O error, dev sdfs, sector 15628053160
[ 143.093599] Buffer I/O error on dev sdfs, logical block 1953506645, async page read
[ 143.659545] Buffer I/O error on dev sdfs, logical block 1953506645, async page read
[ 186.534945] scsi 13:0:50:0: Direct-Access ATA ST8000NM000A-2KE SN02 PQ: 0 ANSI: 6
[ 186.534956] scsi 13:0:50:0: SATA: handle(0x003b), sas_addr(0x50050cc118059923), phy(35), device_name(0x0000000000000000)
[ 186.534959] scsi 13:0:50:0: enclosure logical id (0x50050cc102055160), slot(0)
[ 186.535073] scsi 13:0:50:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 186.535078] scsi 13:0:50:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[ 187.287292] sd 13:0:50:0: Power-on or device reset occurred
[ 187.287370] sd 13:0:50:0: Attached scsi generic sg188 type 0
[ 187.289071] sd 13:0:50:0: [sdfw] 15628053168 512-byte logical blocks: (8.00 TB/7.27 TiB)
[ 187.289074] sd 13:0:50:0: [sdfw] 4096-byte physical blocks
[ 217.884839] sd 13:0:50:0: attempting task abort! scmd(ffff9fbb23aaca00)
[ 217.884846] sd 13:0:50:0: tag#0 CDB: Mode Sense(6) 1a 00 3f 00 04 00
[ 217.884848] scsi target13:0:50: handle(0x003b), sas_address(0x50050cc118059923), phy(35)
[ 217.884850] scsi target13:0:50: enclosure logical id(0x50050cc102055160), slot(0)
[ 218.797459] sd 13:0:50:0: device_block, handle(0x003b)
[ 221.042118] sd 13:0:50:0: device_unblock and setting to running, handle(0x003b)
[ 221.782440] sd 13:0:50:0: task abort: SUCCESS scmd(ffff9fbb23aaca00)
[ 221.782459] sd 13:0:50:0: [sdfw] Write Protect is off
[ 221.782463] sd 13:0:50:0: [sdfw] Mode Sense: 00 00 00 00
[ 221.782482] sd 13:0:50:0: [sdfw] Asking for cache data failed
[ 221.782485] sd 13:0:50:0: [sdfw] Assuming drive cache: write through
[ 221.782934] sd 13:0:50:0: [sdfw] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[ 221.782937] sd 13:0:50:0: [sdfw] Sense not available.
[ 221.782975] sd 13:0:50:0: [sdfw] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[ 221.782978] sd 13:0:50:0: [sdfw] Sense not available.
[ 221.783040] sd 13:0:50:0: [sdfw] Attached SCSI disk
[root@dataserver0 log]# smartctl -a /dev/sdfs
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1127.19.1.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke,
www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST8000NM000A-2KE101
Serial Number: WKD0B211
LU WWN Device Id: 5 000c50 0c2a53721
Firmware Version: SN02
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Oct 27 14:05:40 2020 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Read SMART Data failed: scsi error aborted command
=== START OF READ SMART DATA SECTION ===
SMART Status command failed: scsi error aborted command
SMART overall-health self-assessment test result: UNKNOWN!
SMART Status, Attributes and Thresholds cannot be read.
Read SMART Log Directory failed: scsi error aborted command
Read SMART Error Log failed: scsi error aborted command
Read SMART Self-test Log failed: scsi error aborted command
Selective Self-tests/Logging not supported