Upgrading storage - broke it - mprutil shows more disks than the OS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jorsher

New Member
Jan 25, 2020
5
0
1
I've run out of space so decided to expand storage. I'm using TrueNAS Core 13. Wasn't sure where to post this since I'm not sure what the problem is yet.


Original config:

Pool 1:
  • Onboard SATA -> 2 x SSD in mirror

Pool 2:
  • Onboard LSI3008 -> 8 x HDD
  • Add-on LSI 92xx-8e -> 2 x HP expanders -> 16 x HDD


New config:

Pool 1:
  • Onboard LSI3008 -> 2 x SSD
Pool 2:
  • Add-on 9400-16e -> 5 x Adaptec 82885T -> 40 x HDD (four 1-to-2 SAS fanout cables, 8 drives per expander)

Detail:

When I look at the disks available to the system with camcontrol or in /dev, I don't see any drives on some expanders, only 1-2 drives on other expanders, and all but one drive on another expander. It's unclear to me why some get 'seen' by the OS but others don't
  • The 9400-16e detects all 40 drives while booting. The OS only sees a seemingly random number of them.
  • arcconf command doesn't show any controllers, I assume because I'm only using Adaptec expanders and not an Adaptec controller
  • storcli shows the 9400-16e, but does not show any attached expanders/drives/anything
  • mprutil shows the 9400-16e, along with every drive and adapter
  • /dev/* shows all 5 expanders (ses0, ses1, ses2, etc) and both controllers (mpr0, mpr1), but most disk drives are missing
  • sesutil shows all 5 expanders, but no disks attached
I'm confused how all 40 drives are seen by the HBA at boot, and can be seen within the OS using mprutil, however only a small number of them are available for use.

Any guidance?
 

jdnz

Member
Apr 29, 2021
80
19
8
boot linux off a pendrive and see what is visible - this sounds like a freeBSD driver issue

If it works on Linux you may have to switch from TrueNAS core (freeBSD based) to TrueNAS scale (linux based)
 

Jorsher

New Member
Jan 25, 2020
5
0
1
boot linux off a pendrive and see what is visible - this sounds like a freeBSD driver issue

If it works on Linux you may have to switch from TrueNAS core (freeBSD based) to TrueNAS scale (linux based)
Thanks. Will try that now. There is a newer version of the driver (24 vs 23) but as far as I know it'd be impossible to update the package persistently on TrueNAS Core.
 

Jorsher

New Member
Jan 25, 2020
5
0
1
boot linux off a pendrive and see what is visible - this sounds like a freeBSD driver issue

If it works on Linux you may have to switch from TrueNAS core (freeBSD based) to TrueNAS scale (linux based)
Ugh, well...

First of all, thanks for your suggestion. Put my mind at ease that there aren't any hardware problems. All 40 disks were visible when I live-booted Ubuntu and ran lsblk --scsi.

I really don't want to migrate to Scale. I'm happy with Core and have a number of jails configured that I'd prefer not to rebuild.

I will bring this to TrueNAS forums since that's probably the "right" place to ask, but is there any way to upgrade the drivers on Core? I understand it's not persistent, but if a hacky workaround of running a script after each boot is an option...