Hi guys, I have my hands on an ixsystems x20 Truenas dual controller enterprise box. This is a 12 bay high-availability SAS unit, no external expander boxes.
This unit is populated with both controllers, running fresh SLC flash m2 boot drives and running Truenas Scale 25.10 with success. Truenas recognizes the chassis it is in, recognizes enterprise functionality state. I installed fresh via console cable according to Truenas documentation, IP addresses properly configured of each controller for proper high availability/HA functionality.
Loading up some 14tb Seagate Exos SAS drives into the unit, I am able to see the drives, set up pools, enable HA, all good stuff! I have tested HA extensively and works flawlessly.
I then exported pools and removed the 14tb drives to switch over to 22tb Ultrastar HC570 drives and while they are beautifully detected by controller A, Truenas interface complains controller B cannot see the drives, thus HA is unsatisfied and cannot enable. If I log only into controller A, I can manipulate the drives and create pools, but once a pool is created, Truenas is even more upset because controller B cannot see the pools because it cannot see the drives.
Digging into the CLI, I find that these controllers utilize LSI 9300 controllers in IT mode.
In my deep reading of how to manage these LSI controllers, I decided to gunsling and try to update both controllers to 16.00.12.00 firmware version. I have to assume that these are utilizing SAS9300-4i4e_IT.bin firmware. I ran nvdata, flash, and bios (wasn't available) backups followed by sudo sas3flash -o -f SAS9300-4i4e_IT.bin and it reported success. I did this on both controllers. Powered down the system. Powered it back up.
Running sas3flash -list on either controller now, I see:
Truenas interface isn't unhappy, the 22tb Ultrastar drives are still perfectly detected by controller A but controller B cannot see them, same story.
Using the LSI command line utilities, I can show status from each controller:
Controller A
Controller B:
I am not reading any reports of these Ultrastar HC570 SAS drives being unable to be utilized other than some people reporting PWDIS pin 3 power control issues, but I have already tried blocking pin 3 for fun, zero changes. Even with the firmware flashing I've done so far, when I throw the Exos 14tb SAS disks back in, they are populating beautifully fine on both controllers and HA works perfectly.
Ixsystems is rightfully not going to help me with this EOL machine and had kicked me to the communities for help.
Any insight?
This unit is populated with both controllers, running fresh SLC flash m2 boot drives and running Truenas Scale 25.10 with success. Truenas recognizes the chassis it is in, recognizes enterprise functionality state. I installed fresh via console cable according to Truenas documentation, IP addresses properly configured of each controller for proper high availability/HA functionality.
Loading up some 14tb Seagate Exos SAS drives into the unit, I am able to see the drives, set up pools, enable HA, all good stuff! I have tested HA extensively and works flawlessly.
I then exported pools and removed the 14tb drives to switch over to 22tb Ultrastar HC570 drives and while they are beautifully detected by controller A, Truenas interface complains controller B cannot see the drives, thus HA is unsatisfied and cannot enable. If I log only into controller A, I can manipulate the drives and create pools, but once a pool is created, Truenas is even more upset because controller B cannot see the pools because it cannot see the drives.
Digging into the CLI, I find that these controllers utilize LSI 9300 controllers in IT mode.
Code:
ix20[~]$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.
Adapter Selected is a Avago SAS: SAS3008(C0)
Controller Number : 0
Controller : SAS3008(C0)
PCI Address : 00:04:00:00
SAS Address : 500e0ec-a-094f-fb40
NVDATA Version (Default) : 0e.00.00.00
NVDATA Version (Persistent) : 0e.00.00.00
Firmware Product ID : 0x2221 (IT)
Firmware Version : 14.00.00.00
NVDATA Vendor : LSI
NVDATA Product ID : Undefined
BIOS Version : N/A
UEFI BSD Version : N/A
FCODE Version : N/A
Board Name : Eval Board
Board Assembly : N/A
Board Tracer Number : N/A
Finished Processing Commands Successfully.
Exiting SAS3Flash.Running sas3flash -list on either controller now, I see:
Code:
$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.
Adapter Selected is a Avago SAS: SAS3008(C0)
Controller Number : 0
Controller : SAS3008(C0)
PCI Address : 00:04:00:00
SAS Address : 500e0ec-a-094f-fb40
NVDATA Version (Default) : 0e.01.00.08
NVDATA Version (Persistent) : 0e.01.00.09
Firmware Product ID : 0x2221 (IT)
Firmware Version : 16.00.12.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9300-4i4e
BIOS Version : N/A
UEFI BSD Version : N/A
FCODE Version : N/A
Board Name : SAS9300-4i4e
Board Assembly : N/A
Board Tracer Number : N/A
Finished Processing Commands Successfully.
Exiting SAS3Flash.Using the LSI command line utilities, I can show status from each controller:
Controller A
Code:
$ sudo sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS3008
BIOS version : 0.00.00.00
Firmware version : 16.00.12.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 1023
Concurrent commands supported : 10048
Slot : Unknown
Segment : 0
Bus : 4
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 2
Slot # : 0
SAS Address : 5000cca-2-c801-81dd
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0UPME
Unit Serial No(VPD) : 2TG0UPME
GUID : 5000cca2c80181dc
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 1
SAS Address : 5000cca-2-c801-71c1
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0TMBE
Unit Serial No(VPD) : 2TG0TMBE
GUID : 5000cca2c80171c0
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 2
SAS Address : 5000cca-2-c801-6b99
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0T6NE
Unit Serial No(VPD) : 2TG0T6NE
GUID : 5000cca2c8016b98
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 3
SAS Address : 5000cca-2-c801-56f1
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0RV0E
Unit Serial No(VPD) : 2TG0RV0E
GUID : 5000cca2c80156f0
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 4
SAS Address : 5000cca-2-c800-e9e5
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0HKWE
Unit Serial No(VPD) : 2TG0HKWE
GUID : 5000cca2c800e9e4
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 5
SAS Address : 5000cca-2-c801-5891
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0RYBE
Unit Serial No(VPD) : 2TG0RYBE
GUID : 5000cca2c8015890
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 6
SAS Address : 5000cca-2-c800-e9a5
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0HKBE
Unit Serial No(VPD) : 2TG0HKBE
GUID : 5000cca2c800e9a4
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 7
SAS Address : 5000cca-2-c800-ecb9
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0HSRE
Unit Serial No(VPD) : 2TG0HSRE
GUID : 5000cca2c800ecb8
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 8
SAS Address : 5000cca-2-c801-5bbd
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0S4XE
Unit Serial No(VPD) : 2TG0S4XE
GUID : 5000cca2c8015bbc
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 9
SAS Address : 5000cca-2-c801-7829
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0U1LE
Unit Serial No(VPD) : 2TG0U1LE
GUID : 5000cca2c8017828
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 10
SAS Address : 5000cca-2-c800-ea41
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0HLME
Unit Serial No(VPD) : 2TG0HLME
GUID : 5000cca2c800ea40
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 11
SAS Address : 5000cca-2-c801-5fd5
State : Ready (RDY)
Size (in MB)/(in sectors) : 20981759/5371330559
Manufacturer : WDC
Model Number : WUH722222AL4207
Firmware Revision : G730
Serial No : 2TG0SEBE
Unit Serial No(VPD) : 2TG0SEBE
GUID : 5000cca2c8015fd4
Protocol : SAS
Drive Type : SAS_HDD
Device is a Enclosure services device
Enclosure # : 2
Slot # : 24
SAS Address : 500e0ec-a-0995-82bd
State : Standby (SBY)
Manufacturer : CELESTIC
Model Number : P3217-B
Firmware Revision : 0403
Serial No : x484033
Unit Serial No(VPD) : CTH115PM0BF01Z
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500e0eca:094ffb40
Numslots : 8
StartSlot : 0
Enclosure# : 2
Logical ID : 500e0eca:099582ff
Numslots : 25
StartSlot : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.Controller B:
Code:
$ sudo sas3ircu 0 display
[sudo] password for truenas_admin:
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS3008
BIOS version : 0.00.00.00
Firmware version : 16.00.12.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 1023
Concurrent commands supported : 10048
Slot : Unknown
Segment : 0
Bus : 4
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Enclosure services device
Enclosure # : 2
Slot # : 24
SAS Address : 500e0ec-a-0995-82fd
State : Standby (SBY)
Manufacturer : CELESTIC
Model Number : P3217-B
Firmware Revision : 0403
Serial No : x484033
Unit Serial No(VPD) : CTH115PM0BF01Z
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500e0eca:094ff938
Numslots : 8
StartSlot : 0
Enclosure# : 2
Logical ID : 500e0eca:099582ff
Numslots : 25
StartSlot : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.
truenas_admin@ix20-b[~]$Ixsystems is rightfully not going to help me with this EOL machine and had kicked me to the communities for help.
Any insight?
