9300-8i, Proxmox, only one disk showing

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

colinb

Member
Jul 19, 2022
43
3
8
Not 100% sure where this topic lies, so sorry if it's in the wrong subforum

After lots of "fun" attempting to flash a PERC H330 to IT mode, I gave up, and flashed an IBM LSI SAS9302-8i instead.

Fresh install of Proxmox, all updated, OS sees the HBA, but only one of the eight disks attached to it (nvme3n1) none of the disks are visible

Disks are all attached to HBA via a pair of 1-to-4 cables, each with a a single Mini SAS SFF-8643 at one end and four 29pin SFF-8482 connectors at the other (this one here in fact)

Anything obvious I'm doing or not doing?

lspci:
Code: 
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
lsblk:
Code: 
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme1n1            259:0    0 931.5G  0 disk
├─nvme1n1p1        259:1    0  1007K  0 part
├─nvme1n1p2        259:2    0     1G  0 part /boot/efi
└─nvme1n1p3        259:3    0 930.5G  0 part
  ├─pve-swap       253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root       253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta 253:2    0   8.1G  0 lvm
  │ └─pve-data     253:4    0 794.3G  0 lvm
  └─pve-data_tdata 253:3    0 794.3G  0 lvm
    └─pve-data     253:4    0 794.3G  0 lvm
nvme0n1            259:4    0 931.5G  0 disk
├─nvme0n1p1        259:9    0 931.5G  0 part
└─nvme0n1p9        259:10   0     8M  0 part
nvme2n1            259:5    0 931.5G  0 disk
├─nvme2n1p1        259:7    0 931.5G  0 part
└─nvme2n1p9        259:8    0     8M  0 part
nvme3n1            259:6    0   1.5T  0 disk
lsmod | grep "sas":
Code: 
xhci_pci_renesas       20480  1 xhci_pci
mpt3sas               376832  0
raid_class             16384  1 mpt3sas
scsi_transport_sas     53248  1 mpt3sas
# ./sas3ircu list:
Code: 
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
-----  ------------  ------  ------  -----------------    ------  ------
   0     SAS3008       1000h   97h    00h:03h:00h:00h      1000h   30e0h
SAS3IRCU: Utility Completed Successfully.
./sas3ircu 0 display
Code: 
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.37.00.00
  Firmware version                        : 16.00.10.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 1023
  Concurrent commands supported           : 9856
  Slot                                    : 24
  Segment                                 : 0
  Bus                                     : 3
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:0b2c9b80
  Numslots                                : 8
  StartSlot                               : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.
 
Last edited:
S

SnJ9MX

Active Member
Jul 18, 2019
148
96
28
I am impressed that your SAS3 controller sees any NVMe drives. As far as I know, SAS3 controllers don't support NVMe.
 
C

colinb

Member
Jul 19, 2022
43
3
8
SnJ9MX said:
I am impressed that your SAS3 controller sees any NVMe drives. As far as I know, SAS3 controllers don't support NVMe.
Click to expand...
Well then, I've no clue what that drive is! [E: ohh, yes I do - a drive I threw in at the last minute]

Just ran sas3ircu display, and seems no devices are detected at all. Top post edited with output
 
BLinux

BLinux

cat lover server enthusiast
Jul 7, 2016
2,682
1,089
113
artofserver.com
Since you're in a Linux environment, analyze the logs to see what's really going on behind the scenes.


Your problem is likely very different than what is covered in that above video, but the method for extracting information should be the same. And whatever messages you get might help identify what's going on.

Also, make sure your drives are actually spinning. touch them and see if you feel them vibrating. if not, your drives might have a "power disable" issue described here:

 
C

colinb

Member
Jul 19, 2022
43
3
8
Thanks, those look very helpful - now, and for working through things in the future
 
C

colinb

Member
Jul 19, 2022
43
3
8
So, I've made two steps forward, and then two steps backwards. See above, the NVMe drive was spurious (I forgot I'd added it to the system).

I've got the drives on and spinning (or powered, in the case of SSDs). Regardless of which drives I plug in, and into which of the HBA ports, I can only get proxmox to recognise a single drive:

# lsblk
Code: 
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                  8:0    0   1.5T  0 disk
├─sda1               8:1    0   1.5T  0 part
└─sda9               8:9    0     8M  0 part
sdb                  8:16   0     0B  0 disk
nvme3n1            259:0    0 931.5G  0 disk
├─nvme3n1p1        259:1    0  1007K  0 part
├─nvme3n1p2        259:2    0     1G  0 part /boot/efi
└─nvme3n1p3        259:3    0 930.5G  0 part
  ├─pve-swap       253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root       253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta 253:2    0   8.1G  0 lvm 
  │ └─pve-data     253:4    0 794.3G  0 lvm 
  └─pve-data_tdata 253:3    0 794.3G  0 lvm 
    └─pve-data     253:4    0 794.3G  0 lvm 
nvme0n1            259:4    0 931.5G  0 disk
├─nvme0n1p1        259:5    0 931.5G  0 part
└─nvme0n1p9        259:6    0     8M  0 part
nvme1n1            259:7    0 931.5G  0 disk
├─nvme1n1p1        259:8    0 931.5G  0 part
└─nvme1n1p9        259:9    0     8M  0 part
nvme2n1            259:10   0   1.5T  0 disk
I can't see /var/log/messages, so can't dig into that. Is it somewhere else in PVE?
 
BLinux

BLinux

cat lover server enthusiast
Jul 7, 2016
2,682
1,089
113
artofserver.com
colinb said:
I can't see /var/log/messages, so can't dig into that. Is it somewhere else in PVE?
Click to expand...
It might be that PVE/Debian is using journalctl. You might need to look up how to install rsyslogd on PVE to enable traditional text logging messages.

Also, try running:

dmesg > /tmp/dmesg.txt

Does the /tmp/dmesg.txt show the log messages you need? Alternatively, while you're plugging and unplugging drives, you can open a terminal a monitor it live:

dmesg -w
 
BLinux

BLinux

cat lover server enthusiast
Jul 7, 2016
2,682
1,089
113
artofserver.com
I think you can install syslog like this:

apt install rsyslog

(run as root of course) See if that enables traditional logging.
 
You must log in or register to reply here.
Top Bottom