Fujitsu TX1320 M3 - Cheap low power server (barebone)

[luckylinux]

I was actually trying to see if I could just get in via SSH on the iRMC Controller Interface and see if I could do something there:

user@UBUNTU:~$ ssh admin@192.168.7.45
admin@192.168.7.45's password:

**************************************************
*    Welcome to PRIMERGY Remote Manager          *
*    Firmware Revision 9.08F (1.00)              *
*    SDR 3.16  ID 0458 TX1320M3                  *
*    Firmware built  Mar 5 2018 14:02:35 CEST    *
**************************************************

System Type  : PRIMERGY TX1320 M3
System ID    : YMBK014917
System Name  : CORE05
System OS    : Proxmox VE
System Status: CRITICAL (Identify LED is OFF)
Power Status : On
Asset Tag    : System Asset Tag

    Enclosure Information Menu
    
(e) System Eventlog
(i) Internal Eventlog
(t) Temperature
(v) Voltages/Current
(f) Fans
(p) Power Supplies
(d) Door Lock
(m) Memory Sensors
(c) CPU Sensors
(s) Component Status
(l) List All Sensors

Enter selection or (0) to quit:
                   |Sensor|Sensor| Event|Entity|Entity|                         
Sensor Name        |  Num | Type | RdCd |  ID  | Inst | Sensor TypeString       
-------------------+------+------+------+------+------+-------------------------
Ambient            | 0x01 | 0x01 | 0x01 | 0x37 | 0x00 | Temperature
Systemboard        | 0x02 | 0x01 | 0x01 | 0x07 | 0x00 | Temperature
CPU                | 0x03 | 0x01 | 0x01 | 0x03 | 0x00 | Temperature
MEM A              | 0x04 | 0x01 | 0x01 | 0x20 | 0x00 | Temperature
MEM B              | 0x05 | 0x01 | 0x01 | 0x20 | 0x01 | Temperature
PSU1 Inlet         | 0x06 | 0x01 | 0x01 | 0x0A | 0x04 | Temperature
PSU2 Inlet         | 0x07 | 0x01 | 0x01 | 0x0A | 0x08 | Temperature
PSU1               | 0x08 | 0x01 | 0x01 | 0x0A | 0x04 | Temperature
PSU2               | 0x09 | 0x01 | 0x01 | 0x0A | 0x08 | Temperature
BBU                | 0x0A | 0x01 | 0x01 | 0x28 | 0x01 | Temperature
RAID Controller    | 0x0B | 0x01 | 0x01 | 0x0B | 0x01 | Temperature
BATT 3.0V          | 0x0D | 0x02 | 0x01 | 0x28 | 0x00 | Voltage
MAIN 12V           | 0x0E | 0x02 | 0x01 | 0x07 | 0x00 | Voltage
MAIN 5V            | 0x0F | 0x02 | 0x01 | 0x07 | 0x00 | Voltage
MAIN 3.3V          | 0x10 | 0x02 | 0x01 | 0x07 | 0x00 | Voltage
FAN1 SYS           | 0x11 | 0x04 | 0x01 | 0x1D | 0x00 | Fan
FAN2 SYS           | 0x12 | 0x04 | 0x01 | 0x1D | 0x01 | Fan
FAN3 SYS           | 0x13 | 0x04 | 0x01 | 0x1D | 0x02 | Fan
FAN PSU            | 0x14 | 0x04 | 0x01 | 0x0A | 0x00 | Fan
FAN PSU1           | 0x15 | 0x04 | 0x01 | 0x0A | 0x04 | Fan
FAN PSU2           | 0x16 | 0x04 | 0x01 | 0x0A | 0x08 | Fan
PSU1 Power         | 0x17 | 0x0B | 0x01 | 0x0A | 0x04 | Other
PSU2 Power         | 0x18 | 0x0B | 0x01 | 0x0A | 0x08 | Other
Total Power        | 0x19 | 0x0B | 0x01 | 0xE0 | 0x00 | Other
Total Power Out    | 0x1A | 0x0B | 0x01 | 0xE0 | 0x01 | Other
I2C1 error ratio   | 0x1B | 0xC0 | 0x01 | 0x06 | 0x01 | I2C Bus
I2C2 error ratio   | 0x1C | 0xC0 | 0x01 | 0x06 | 0x02 | I2C Bus
I2C3 error ratio   | 0x1D | 0xC0 | 0x01 | 0x06 | 0x03 | I2C Bus
I2C4 error ratio   | 0x1E | 0xC0 | 0x01 | 0x06 | 0x04 | I2C Bus
I2C5 error ratio   | 0x1F | 0xC0 | 0x01 | 0x06 | 0x05 | I2C Bus
I2C6 error ratio   | 0x20 | 0xC0 | 0x01 | 0x06 | 0x06 | I2C Bus
I2C7 error ratio   | 0x21 | 0xC0 | 0x01 | 0x06 | 0x07 | I2C Bus
I2C8 error ratio   | 0x22 | 0xC0 | 0x01 | 0x06 | 0x08 | I2C Bus
SEL Level          | 0x23 | 0x26 | 0x01 | 0x06 | 0x00 | Monitor ASIC
Ambient            | 0x24 | 0x01 | 0x08 | 0x37 | 0x00 | Temperature
Ambient            | 0x25 | 0x01 | 0x05 | 0x37 | 0x00 | Temperature
CPU                | 0x26 | 0x07 | 0x6F | 0x03 | 0x00 | Processor
Power Limit        | 0x27 | 0x09 | 0x05 | 0x13 | 0x00 | Power Unit
Power Unit         | 0x28 | 0x09 | 0x0B | 0x13 | 0x00 | Power Unit
PSU Config         | 0x29 | 0x2B | 0x6F | 0x13 | 0x00 | Version Change
PSU                | 0x2A | 0x09 | 0x6F | 0x13 | 0x00 | Power Unit
PSU                | 0x2B | 0xE8 | 0x6F | 0x0A | 0x00 | Power Supply Status
PSU1               | 0x2C | 0xE8 | 0x6F | 0x0A | 0x04 | Power Supply Status
PSU2               | 0x2D | 0xE8 | 0x6F | 0x0A | 0x08 | Power Supply Status
Power Level        | 0x2E | 0xDD | 0x6F | 0xE0 | 0x02 | OEM reserved #dd
P-STATE Throttle   | 0x2F | 0xEB | 0x6F | 0x2E | 0x00 | OEM reserved #eb
System State       | 0x30 | 0x12 | 0x0A | 0x07 | 0x00 | System Event
FAN1 SYS           | 0x31 | 0xE6 | 0x6F | 0x1D | 0x00 | Fan Status
FAN2 SYS           | 0x32 | 0xE6 | 0x6F | 0x1D | 0x01 | Fan Status
FAN3 SYS           | 0x33 | 0xE6 | 0x6F | 0x1D | 0x02 | Fan Status
FAN PSU            | 0x34 | 0xE6 | 0x6F | 0x0A | 0x00 | Fan Status
FAN PSU1           | 0x35 | 0xE6 | 0x6F | 0x0A | 0x04 | Fan Status
FAN PSU2           | 0x36 | 0xE6 | 0x6F | 0x0A | 0x08 | Fan Status
Watchdog           | 0x37 | 0x23 | 0x6F | 0x00 | 0x00 | Watchdog
Housing open       | 0x38 | 0x05 | 0x6F | 0x17 | 0x00 | Physical Security
CPU detection      | 0x39 | 0x12 | 0x6F | 0x07 | 0x00 | System Event
ME                 | 0x3A | 0x28 | 0x6F | 0x21 | 0x00 | Management Subsystem Health
iRMC request       | 0x3B | 0xEA | 0x6F | 0x06 | 0x00 | BMC Communication
I2C1               | 0x3C | 0xC0 | 0x6F | 0x06 | 0x01 | I2C Bus
I2C2               | 0x3D | 0xC0 | 0x6F | 0x06 | 0x02 | I2C Bus
I2C3               | 0x3E | 0xC0 | 0x6F | 0x06 | 0x03 | I2C Bus
I2C4               | 0x3F | 0xC0 | 0x6F | 0x06 | 0x04 | I2C Bus
I2C5               | 0x40 | 0xC0 | 0x6F | 0x06 | 0x05 | I2C Bus
I2C6               | 0x41 | 0xC0 | 0x6F | 0x06 | 0x06 | I2C Bus
I2C7               | 0x42 | 0xC0 | 0x6F | 0x06 | 0x07 | I2C Bus
I2C8               | 0x43 | 0xC0 | 0x6F | 0x06 | 0x08 | I2C Bus
Config backup      | 0x44 | 0xEF | 0x6F | 0x21 | 0x00 | OEM reserved #ef
Power Unit         | 0x45 | 0x25 | 0x6F | 0x13 | 0x00 | Entity Presence
PSU                | 0x46 | 0x25 | 0x6F | 0x0A | 0x00 | Entity Presence
PSU1               | 0x47 | 0x25 | 0x6F | 0x0A | 0x04 | Entity Presence
PSU2               | 0x48 | 0x25 | 0x6F | 0x0A | 0x08 | Entity Presence
Total Power        | 0x49 | 0x25 | 0x6F | 0xE0 | 0x00 | Entity Presence
Total Power Out    | 0x4A | 0x25 | 0x6F | 0xE0 | 0x01 | Entity Presence
Power Level        | 0x4B | 0x25 | 0x6F | 0xE0 | 0x02 | Entity Presence
FAN3 SYS           | 0x4C | 0x25 | 0x6F | 0x1D | 0x02 | Entity Presence
Ambient            | 0x4D | 0x01 | 0x07 | 0x37 | 0x00 | Temperature
CPU                | 0x4E | 0x07 | 0x07 | 0x03 | 0x00 | Processor
DIMM-1A            | 0x4F | 0x0C | 0x07 | 0x20 | 0x01 | Memory
DIMM-2A            | 0x50 | 0x0C | 0x07 | 0x20 | 0x00 | Memory
DIMM-1B            | 0x51 | 0x0C | 0x07 | 0x20 | 0x03 | Memory
DIMM-2B            | 0x52 | 0x0C | 0x07 | 0x20 | 0x02 | Memory
FAN1 SYS           | 0x53 | 0x04 | 0x07 | 0x1D | 0x00 | Fan
FAN2 SYS           | 0x54 | 0x04 | 0x07 | 0x1D | 0x01 | Fan
FAN3 SYS           | 0x55 | 0x04 | 0x07 | 0x1D | 0x02 | Fan
FAN PSU            | 0x57 | 0x04 | 0x07 | 0x0A | 0x00 | Fan
FAN PSU1           | 0x58 | 0x04 | 0x07 | 0x0A | 0x04 | Fan
FAN PSU2           | 0x59 | 0x04 | 0x07 | 0x0A | 0x08 | Fan
BATT 3.0V          | 0x5A | 0x02 | 0x07 | 0x28 | 0x00 | Voltage
Voltages           | 0x5B | 0x02 | 0x07 | 0xE1 | 0x00 | Voltage
Temp               | 0x5C | 0x01 | 0x07 | 0xE1 | 0x00 | Temperature
BBU                | 0x5D | 0x29 | 0x07 | 0x28 | 0x01 | Battery
Power Unit         | 0x5E | 0x09 | 0x07 | 0x13 | 0x01 | Power Unit
PSU Config         | 0x5F | 0x2B | 0x07 | 0x13 | 0x00 | Version Change
PSU                | 0x60 | 0x08 | 0x07 | 0x0A | 0x00 | Power Supply
PSU1               | 0x61 | 0x08 | 0x07 | 0x0A | 0x04 | Power Supply
PSU2               | 0x62 | 0x08 | 0x07 | 0x0A | 0x08 | Power Supply
Power Level        | 0x63 | 0xDD | 0x07 | 0xE0 | 0x02 | OEM reserved #dd
Slot1              | 0x64 | 0x13 | 0x07 | 0x31 | 0x00 | Critical Interrupt
Slot2              | 0x65 | 0x13 | 0x07 | 0x31 | 0x01 | Critical Interrupt
Slot3              | 0x66 | 0x13 | 0x07 | 0x31 | 0x02 | Critical Interrupt
Slot4              | 0x67 | 0x13 | 0x07 | 0x31 | 0x03 | Critical Interrupt
RAID Controller    | 0x68 | 0x17 | 0x07 | 0x0B | 0x00 | Add-in Card
HDD                | 0x69 | 0x0D | 0x07 | 0x04 | 0x00 | Drive Slot / Bay
HDD0               | 0x6A | 0x0D | 0x07 | 0x04 | 0x01 | Drive Slot / Bay
HDD1               | 0x6B | 0x0D | 0x07 | 0x04 | 0x02 | Drive Slot / Bay
HDD2               | 0x6C | 0x0D | 0x07 | 0x04 | 0x03 | Drive Slot / Bay
HDD3               | 0x6D | 0x0D | 0x07 | 0x04 | 0x04 | Drive Slot / Bay
HDD4               | 0x6E | 0x0D | 0x07 | 0x04 | 0x05 | Drive Slot / Bay
HDD5               | 0x6F | 0x0D | 0x07 | 0x04 | 0x06 | Drive Slot / Bay
HDD6               | 0x70 | 0x0D | 0x07 | 0x04 | 0x07 | Drive Slot / Bay
HDD7               | 0x71 | 0x0D | 0x07 | 0x04 | 0x08 | Drive Slot / Bay
BIOS               | 0x72 | 0x00 | 0x07 | 0x22 | 0x00 | reserved
Agent              | 0x73 | 0x00 | 0x07 | 0x21 | 0x00 | reserved
ME                 | 0x75 | 0x28 | 0x07 | 0x21 | 0x00 | Management Subsystem Health
iRMC               | 0x76 | 0x00 | 0x07 | 0x06 | 0x00 | reserved
System Mgmt SW     | 0x77 | 0xEE | 0x70 | 0x21 | 0x00 | Comm. Sensor
Local Monitor      | 0x78 | 0x1A | 0x09 | 0x21 | 0x00 | Other FRU
Pwr Btn override   | 0x79 | 0x22 | 0x6F | 0x0C | 0x00 | System ACPI Power State
NMI                | 0x7A | 0x13 | 0x6F | 0x21 | 0x00 | Critical Interrupt
System BIOS        | 0xC0 | 0xEE | 0x70 | 0x22 | 0x00 | Comm. Sensor
iRMC               | 0xFE | 0xEE | 0x70 | 0x06 | 0x00 | Comm. Sensor

Press any key to continue
**************************************************
*    Welcome to PRIMERGY Remote Manager          *
*    Firmware Revision 9.08F (1.00)              *
*    SDR 3.16  ID 0458 TX1320M3                  *
*    Firmware built  Mar 5 2018 14:02:35 CEST    *
**************************************************

System Type  : PRIMERGY TX1320 M3
System ID    : YMBK014917
System Name  : CORE05
System OS    : Proxmox VE
System Status: CRITICAL (Identify LED is OFF)
Power Status : On
Asset Tag    : System Asset Tag

    Enclosure Information Menu
    
(e) System Eventlog
(i) Internal Eventlog
(t) Temperature
(v) Voltages/Current
(f) Fans
(p) Power Supplies
(d) Door Lock
(m) Memory Sensors
(c) CPU Sensors
(s) Component Status
(l) List All Sensors

Enter selection or (0) to quit:
Fan                |  6 Sensor(s)
                   |  Speed | Rev |   Fail  |  Fail |                           
Sensor Name        |  [rpm] | [%] | Reaction| Delay | Status                   
-------------------+--------+-----+---------+-------+---------------------------
FAN1 SYS           |    1560|  103| continue|     90| Fan ON, running
FAN2 SYS           |    1080|  104| continue|     90| Fan ON, running
FAN3 SYS           |       0|    0| continue|     90| N/A
FAN PSU            |     880|  100| continue|     90| Fan ON, running
FAN PSU1           |       0|    0| continue|     90| N/A
FAN PSU2           |       0|    0| continue|     90| N/A

Press any key to continue

Not sure if the Sensor Num / Sensor Type really help much.


One can connect to their SMASH CLP Shell using a bit of a "Trick" (need to specify some Command in the ssh Command, this causes each further Command you issue / Key you Press to need to be confirmed with [ENTER]):

ssh admin@192.168.7.45 'ls -l /sys/'

**************************************************
*    Welcome to PRIMERGY Remote Manager          *
*    Firmware Revision 9.08F (1.00)              *
*    SDR 3.16  ID 0458 TX1320M3                  *
*    Firmware built  Mar 5 2018 14:02:35 CEST    *
**************************************************

System Type  : PRIMERGY TX1320 M3
System ID    : YMBK014917
System Name  : CORE05
System OS    : Proxmox VE
System Status: CRITICAL (Identify LED is OFF)
Power Status : On
Asset Tag    : System Asset Tag

    Shell Menu
    
(1) Start SMASH CLP shell...

Enter selection or (0) to quit:


-> help
help
   Command Status: COMMAND COMPLETED

SMASH CLP is a command line interface for the remote server management. The commands are of the form of :
     <verb> [<options>] [<target>] [<properties>]

Commands List:
     cd
     create
     delete
     dump
     exit
     help
     load
     reset
     set
     show
     start
     stop
     version
     OEM_COMMANDS


For a comprehensive help please refer to the CLP help manual!
-> show
show

   Command Status: COMMAND COMPLETED

   ufip=/
   Targets:
    admin1
   Verbs:
    cd
    exit
    help
    show
    version

-> version
version
   Command Status: COMMAND COMPLETED
   CLP Version:1.0.2
   SM ME Addressing Version:1.0.0

-> dump
dump
   COMMAND SYNTAX ERROR

-> help dump
help dump
   Command Status: COMMAND COMPLETED
Command Name: Dump
Moves a binary image from the MAP to a URI.

Usage: dump -destination <URI> [-options] [target]
destination <URI> - specifies the location to store the image.

Options:
     destination - destination place to transfer the image
     examine - instructs the command processor to validate the command syntax without executing the command.
     help - shows help on how to use Dump
     keep - retain the Job status information for the specified time
     output - specifies the format for the output.
         syntax: -output format=<text | keyword | clpxml>
     version - shows the smash version
     wait - wait for command to complete execution before returning.

-> OEM_COMMANDS
OEM_COMMANDS
   COMMAND SYNTAX ERROR

-> help OEM_COMMANDS
help OEM_COMMANDS
   COMMAND SYNTAX ERROR

-> OEM_COMMANDS help
OEM_COMMANDS help
   COMMAND SYNTAX ERROR

-> elp
     load
     reset
     set
     show
     start
     stop
     version
     OEM_COMMANDS
elp
   COMMAND SYNTAX ERROR

-> elp
     load
     reset
     set
     show
     start
     stop
     version
     OEM_COMMANDS
     load
   COMMAND SYNTAX ERROR

->      reset
   Command Status: COMMAND EXECUTION FAILED
   reset verb is not supported for /

->      set
   COMMAND SYNTAX ERROR

->      show

   Command Status: COMMAND COMPLETED

   ufip=/
   Targets:
    admin1
   Verbs:
    cd
    exit
    help
    show
    version

->      start
   Command Status: COMMAND EXECUTION FAILED
   start verb is not supported for /

->      stop
   Command Status: COMMAND EXECUTION FAILED
   stop verb is not supported for /

->      version
   Command Status: COMMAND COMPLETED
   CLP Version:1.0.2
   SM ME Addressing Version:1.0.0

->      OEM_COMMANDS
   COMMAND SYNTAX ERROR

-> elp
   COMMAND SYNTAX ERROR

Not that you can do much anyways .

I would have hoped I could find pwm / hwmon Interfaces on the iRMC OS, but it might be a more complicated than that .

 

[hmartin]

The GPL archive I have is for the TX140 S2, which is also iRMC S4. Probably this is a common release for all iRMC S4 devices, but I don't know.

IIRC, the TX140 S2 iRMC releases from Fujitsu are older than what the TX1320 M3 has available, but probably not much has changed.

 

[slowlaris]

OK I finally got M4 and can report some findings here.

The server arrived with Xeon E-2236, 16+16 GB of RAM, 1TB Samsung 970 PRO and NVIDIA GT 710. It also has two redundant 450W PSUs. I've installed latest Ubuntu 24.04.3 just to test the hardware and power consumption.

dmesg: dpaste: ACXCBPVLH

Spoiler: lspci

root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.1 Communication controller: Intel Corporation Device a361 (rev 10)
00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1c.3 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
03:00.0 PCI bridge: Emulex Corporation x1 PCIe Gen2 Bridge[Pilot4]
04:01.0 Co-processor: Fujitsu Technology Solutions Device 1228
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

First observations: right from the start, it's impossible to turn it on until iRMC finished booting and ID led blinked (around 20 secs after power is applied). Maybe it's a standard server thing which I already forgot...
Also, sometimes (every other time) I'm unable to enter the BIOS or boot menu after hard power off, keyboard just doesn't work (I've tried 3 different ones). After I boot the OS and then reboot from it, I can enter BIOS just fine. Weird.

PSU
Since I've got dual redundant PSUs, BMC gives a warning when one of them is not connected, which's okay but IMO overkill for a home lab use. I disassembled one of them to check if it's viable to do Noctua swap as @Gruenschnabel wanted to do, but I'm not really sure it's worth it:
https://www.fansco.com/uploads/2024/04/MGT4012UB-W28-Protechnic.pdf this is the fan inside (but 4 pin), and per suitable Noctua NF A4-20PWM specs, Noctua's flow rate is ~6CFM (vs 23 CFM) and RPM is also 5K vs 13K. I think what will happen is that BMC will try to force full 13K RPM on startup/check and won't be able to do that. All of this of course if PWM signal is the same. Last nail to the coffin for me is the pricing: new Noctua fan is 16 EUR, while non-redundant PSU is 17 EUR on eBay... the only downside of standard PSU for me (apart from max power, but I doubt I'll go over that) is that it's not rated as Premium, so efficiency will be lower. You win some, you lose some...
Lastly, noise: redundant PSUs are not loud per se, but this small fan is running all the time and produces quite audible low-frequency hum, which only goes away when everything's off and the plug is pulled. This hum is very well transmitted over the floor and I can tell the difference when the server is switched on in the other room.

Drives backplane and controller
Per @hmartin 's blog suggestion I've replaced the backplane with a unified one (where single board can handle 8 drives). I wish I had read further about SATA card
So long story short, I've decided to buy Broadcom HBA 9500-8i SAS/SATA/PCIe(NVME) 12Gb/s SAS3808 controller for 100 EUR from China. It works, but the problem is that it has Slimline SAS x8 connector. The only matching cable I was able to find to plug into backplane has straight connectors, which prevents backplane fan shroud from closing. That's very disappointing. I've looked into Fujitsu spare parts list, and their controllers based on the same chip (like EP520i) have "standard" SFF-8643 connectors, which allows plugging cheap and vastly abundant Mini SAS HD (SFF-8643) cables. Unless I find an angled cable or adapter I won't be able to use the this controller. Also, backplane fan is quite noisy and power hungry: Delta DS07025B12U, 12V 0.7A. Fan from M3 is 0.3 or 0.4A, but is also thinner by approx 5mm, so it won't fit in the same frame... and of course 70x70mm is pretty non-standard size and apart from Delta and random Chinese vendors you won't find anything. Ah yes, fan shroud + fan for M4 costs 46 EUR on eBay...

I've also had a random LSI 3008 controller laying around which I wanted to compare. As you can see, 9500 has ASPM support enabled, while 3008 does not. Also 3008 gets really got even when nothing is plugged in.

Spoiler: lspci of Broadcom HBA 9500-8i

root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci -vv -s 01:00.0
01:00.0 Serial Attached SCSI controller: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx
    Subsystem: Broadcom / LSI 9500-8i Tri-Mode HBA
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    IOMMU group: 1
    Region 0: Memory at 9a200000 (64-bit, prefetchable) [size=1M]
    Region 2: Memory at 9a100000 (64-bit, prefetchable) [size=1M]
    Region 4: Memory at 9c300000 (32-bit, non-prefetchable) [size=1M]
    Region 5: I/O ports at 6000
    Expansion ROM at 9c400000 [disabled] [size=256K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
        DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s (downgraded), Width x8
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
        LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: Upstream Port
    Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
        Vector table: BAR=0 offset=00002000
        PBA: BAR=0 offset=00003000
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [148 v1] Power Budgeting <?>
    Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap: MFVC- ACS-, Next Function: 0
        ARICtl: MFVC- ACS-, Function Group: 0
    Capabilities: [168 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [188 v1] Physical Layer 16.0 GT/s <?>
    Capabilities: [1b0 v1] Lane Margining at the Receiver <?>
    Capabilities: [218 v1] Dynamic Power Allocation <?>
    Capabilities: [248 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
    Capabilities: [348 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
    Capabilities: [380 v1] Data Link Feature <?>
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

Spoiler: lspci of SAS3008

root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci -vv -s 01:00.0
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02)
    Subsystem: Lenovo ServeRAID M1210 SAS/SATA Controller
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    IOMMU group: 1
    Region 0: I/O ports at 6000 
    Region 1: Memory at 9b300000 (64-bit, non-prefetchable) [size=64K][/SPOILER][SPOILER=lspci of SAS3008]
    Region 3: Memory at 9b200000 (64-bit, non-prefetchable) [size=1M]
    Expansion ROM at 9b100000 [disabled] [size=1M]
    Capabilities: [50] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
        DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <2us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x4
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [d0] Vital Product Data
        Not readable
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [c0] MSI-X: Enable+ Count=97 Masked-
        Vector table: BAR=1 offset=0000e000
        PBA: BAR=1 offset=0000f000
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 04000001 0000000f 20010000 66fe5e71
    Capabilities: [1e0 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [1c0 v1] Power Budgeting <?>
    Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap: MFVC- ACS-, Next Function: 0
        ARICtl: MFVC- ACS-, Function Group: 0
    Kernel driver in use: megaraid_sas
    Kernel modules: megaraid_sas

Also I able to confirm that 2nd M.2 slot doesn't recognize NVMe SSDs. I don't have any MSATA M.2 SSDs around, and since I wanted to have NVMe mirror, I plan to utilize that Viking U.2 adapter. The only question is where to put it, and there are not many options: either PCIe slot (but then it will take 1.5 slots, and won't be able to cool it properly), or try to construct some kind of holder to put into RDX drive slot, which has very weird 1.75x4 inch dimensions. Maybe I'll buy some old RDX drive just for the parts...if this works out, two small Noctuas should cool it reliably, but then again total solution cost is quite high. (Viking 20 + RDX frame 30 + U.2 to M.2 cable + 20 == 70 EUR).

Power consumption
Finally, some numbers (taken from the wall).

Switched off (iRMC booted up): 7W (I blame redundant PSU fan)
Booted into desktop (NVMe SSD and GT710): 24W
9500 controller installed: 28.4W (no drives attached)
3008 controller installed: 33W (no drives attached)

These are without any powertop tuning. All C-states are enabled in BIOS, but I feel like at least 10W can be saved if those Delta fans were replaced.

 

[hmartin]

First observations: right from the start, it's impossible to turn it on until iRMC finished booting and ID led blinked (around 20 secs after power is applied). Maybe it's a standard server thing which I already forgot...

Yes, this is sadly the case. Fujitsu also did not optimise iRMC for fast boot times at all. I guess enterprise customers don't complain about this enough.

I wish I had read further about SATA card

I don't know why you don't consider an ASM1166 based card? If you plan to use the 8 drive backplane, you can only connect one half of the BP to the PCIe controller and leave the other one with the onboard ports, and you won't suffer any performance penalty. As a bonus, they are really cheap compared to a "big boy HBA" and consume basically no power.

Switched off (iRMC booted up): 7W (I blame redundant PSU fan)

This is typical, replacing the redundant PSU with the fixed model will not improve it more than 1 or 2 watts. The PSU needs to maintain the +11VSB and the BMC is not power optimised in the slightest. Looking at the smart plug on mine, it's drawing 6W powered off (so only iRMC powered).

All C-states are enabled in BIOS, but I feel like at least 10W can be saved if those Delta fans were replaced.

Not a chance, but you'll save probably 5W if you remove the GT710.

Edit: also, you might consider making a new thread to discuss the M4, I am sure more people will buy some once the prices come down

 

[slowlaris]

I don't know why you don't consider an ASM1166 based card? If you plan to use the 8 drive backplane, you can only connect one half of the BP to the PCIe controller and leave the other one with the onboard ports, and you won't suffer any performance penalty. As a bonus, they are really cheap compared to a "big boy HBA" and consume basically no power.

Yeah I understand it now. I'll probably sell that HBA and cable then.

EDIT: @hmartin could you share a link to the cable you've used with this card? There are plenty of MiniSAS HD to 4 SATA cables, but the only angled SAS connector (so that backplane fan shroud can close) I am able to find has some additional connector which I don't need: Supermicro CBL-00129-01-A-R

 

[hmartin]

EDIT: @hmartin could you share a link to the cable you've used with this card? There are plenty of MiniSAS HD to 4 SATA cables, but the only angled SAS connector (so that backplane fan shroud can close) I am able to find has some additional connector which I don't need: Supermicro CBL-00129-01-A-R

They exist on AliExpress, around 10 Euros. Search something like "SFF-8643 90 degree to SFF-8087"

 

[slowlaris]

They exist on AliExpress, around 10 Euros. Search something like "SFF-8643 90 degree to SFF-8087"

Thanks! But then the card is having SFF-8087 connectors as well, not plain SATA.

 

[hmartin]

Thanks! But then the card is having SFF-8087 connectors as well, not plain SATA.

Yes, see my blog post about ASM1166 controllers. There are models that come with SFF-8087.

 

[luckylinux]

Thanks! But then the card is having SFF-8087 connectors as well, not plain SATA.

Make sure you get the right Variant too.

Forward Cables vs Reverse Cables are particularly relevant for SFF-8087. I think SFF-8643 as well.

Forward: SFF-8087 / SFF-8643 HBA (e.g. LSI 9211-8i / 9300-8i) via Motherboard PCIe Slot ----> 4 x SATA Drives on Backplane

Reverse: 4 x SATA Connectors on Motherboard -----> 1 x SFF-8087 / SFF-8643 Connector on Backplane

 

[slowlaris]

This is typical, replacing the redundant PSU with the fixed model will not improve it more than 1 or 2 watts. The PSU needs to maintain the +11VSB and the BMC is not power optimised in the slightest. Looking at the smart plug on mine, it's drawing 6W powered off (so only iRMC powered).

I'm happy to report that the system with standard (non-redundant) PSU is pulling 3.5W in a switched off state (I guess mainly because PSU fan is not running).

Unfortunately my PSU unit came without the mounting frame, so I cannot install it properly: PRIMERGY Illustrated Spares Catalog.

Here are relevant parts of the manual:




I doubt it has a dedicated part number, but it might explain why "Fujitsu Primergy 1320 M4 PSU" was 16 EUR and exactly the same one but looked up by the part number costs 130 EUR...

 

[luckylinux]

I'm happy to report that the system with standard (non-redundant) PSU is pulling 3.5W in a switched off state (I guess mainly because PSU fan is not running).

Unfortunately my PSU unit came without the mounting frame, so I cannot install it properly: PRIMERGY Illustrated Spares Catalog.

Here are relevant parts of the manual:
View attachment 46666

View attachment 46667

I doubt it has a dedicated part number, but it might explain why "Fujitsu Primergy 1320 M4 PSU" was 16 EUR and exactly the same one but looked up by the part number costs 130 EUR...

The Mounting Frame should be Part of the Chassis, NOT the PSU, as it shouldn't be required to be replaced during a PSU Replacement (it's just a Metal Part after all).

I only see 2 Options:
- Design and 3D Print some Bracket (make sure you use some relatively high Temperature Material such as PETG)
- The TX1320 M3 Operating Manual seems to indicate that the Bracket is the same (is it really though ???), so you might be able to get a Barebones Chassis without even Motherboard

You can try to ask e.g. the Seller at FUJITSU TX1320 M3 SERVER MAINBOARD D3373-B12-GS3 POWER SUPP KÜHLER BACKPLANE BR1 | eBay.de since he's selling Motherboards/PSU/etc separately (but has no individual Listing for the empty Chassis), if he's willing to sell an empty TX1320 M3 Chassis (possible) or just the empty Bracket (unlikely).

Other than that you can try digging a bit deeper than I did (I couldn't find anything) with 26113-E564-V71 Bracket (that's the PSU Model Number).

 

[slowlaris]

The Mounting Frame should be Part of the Chassis, NOT the PSU, as it shouldn't be required to be replaced during a PSU Replacement (it's just a Metal Part after all).

I only see 2 Options:
- Design and 3D Print some Bracket (make sure you use some relatively high Temperature Material such as PETG)
- The TX1320 M3 Operating Manual seems to indicate that the Bracket is the same (is it really though ???), so you might be able to get a Barebones Chassis without even Motherboard

You can try to ask e.g. the Seller at FUJITSU TX1320 M3 SERVER MAINBOARD D3373-B12-GS3 POWER SUPP KÜHLER BACKPLANE BR1 | eBay.de since he's selling Motherboards/PSU/etc separately (but has no individual Listing for the empty Chassis), if he's willing to sell an empty TX1320 M3 Chassis (possible) or just the empty Bracket (unlikely).

Other than that you can try digging a bit deeper than I did (I couldn't find anything) with 26113-E564-V71 Bracket (that's the PSU Model Number).

I think you're right and it's a part of the frame; if the unit is ordered with redundant PSUs from the factory, they probably don't include the bracket anyway because this kind of reverse PSU swap doesn't make sense for a customer. I spent few hours yesterday trying to find the frame but I think it's a lost cause.

I don't have a 3D printer and I think I will just reinstall redundant frame back and just use one of the PSUs, so iRMC doesn't give me an error when only one PSU is plugged in (but both are installed). Maybe try the Noctua mod, since I have two PSUs anyway

I think I will also revert to the old backplane (which is comprised of two halves) because ultimately I need CD power and Molex for NVMe connectors, and included backplane power cable conveniently has all of them. I don't mind crimping my own cable, but I trust Fujitsu more

 

[slowlaris]

I decided to check drive prices again and small capacity (< 4 TB) SAS are really cheap, cheaper than HDDs and I think it's guaranteed they're not SMR. I've never used SAS drives before and I really don't know what to expect, but if I get 7200 RPM drive I think it shouldn't be much noisier? I've also checked power ratings and they're consuming 2-3 W more than regular 2.5" SATA HDDs, but if spin down command works I don't think it's going to be a problem. Also as far as I understand, they should still work with that ASM1166 card.

In my old NAS I'm going to migrate I currently have 6x1TB 2.5" WD Reds, but since I'll have 8 slots in the backplane now I think I can put 2 SAS drives and see how it goes.

Could you share your SAS experience?

 

[luckylinux]

I decided to check drive prices again and small capacity (< 4 TB) SAS are really cheap, cheaper than HDDs and I think it's guaranteed they're not SMR. I've never used SAS drives before and I really don't know what to expect, but if I get 7200 RPM drive I think it shouldn't be much noisier? I've also checked power ratings and they're consuming 2-3 W more than regular 2.5" SATA HDDs, but if spin down command works I don't think it's going to be a problem. Also as far as I understand, they should still work with that ASM1166 card.

In my old NAS I'm going to migrate I currently have 6x1TB 2.5" WD Reds, but since I'll have 8 slots in the backplane now I think I can put 2 SAS drives and see how it goes.

Could you share your SAS experience?

ASM1166 is only a SATA Adapter. How should SAS even work there ?

The Physical Interface might be the same, but the Protocol is different.

It's the same Reason why SAS Drives won't work on your Motherboard SATA Ports ...

 

[slowlaris]

ASM1166 is only a SATA Adapter. How should SAS even work there ?

The Physical Interface might be the same, but the Protocol is different.

It's the same Reason why SAS Drives won't work on your Motherboard SATA Ports ...

Yeah you're right. I do have a SAS controller though, I'm just wondering if going with SAS disks does make sense...

 

[hmartin]

Since you're targeting sub-4TB: 1.92/3.84TB SATA enterprise SSDs would be a better fit, IMHO. They're much lower power and will have better longevity than a mechanical drive.

They were a great buy 6-12 months ago. Not sure about pricing now though.

As for SAS experience, I have none to share. I only use SATA SSDs in this system.

 

[celemine1gig]

If the presented options are

• SAS HDDs and SAS controller
• OR
• SATA Enterprise SSDs and SATA controller

then keep in mind, that most Enterprise SSDs intentionally do NOT support powermanagement states. As a result, this will keep the SATA controller from reaching low power states, which in turn will keep the CPU from reaching low (probably anything lower than C2) package C-States.

So, in other words:
If power is a concern, then low power is really hard to reach, regardless if we are talking SAS HDDs, or Enterprise SATA SSDs.
From my experience Enterprise grade SSDs with power management abilities are very hard to come by.

 

[slowlaris]

If the presented options are

• SAS HDDs and SAS controller
• OR
• SATA Enterprise SSDs and SATA controller

then keep in mind, that most Enterprise SSDs intentionally do NOT support powermanagement states. As a result, this will keep the SATA controller from reaching low power states, which in turn will keep the CPU from reaching low (probably anything lower than C2) package C-States.

So, in other words:
If power is a concern, then low power is really hard to reach, regardless if we are talking SAS HDDs, or Enterprise SATA SSDs.
From my experience Enterprise grade SSDs with power management abilities are very hard to come by.

Thanks for the insight! I got tired of my worries and made a proper spreadsheet which helps me to decide between drive types. Sadly Enterprise SAS SSDs are very expensive (~ 300 EUR for 3.84 TB drive and consequently ~170 EUR for half capacity), so I decided to skip them. To put SAS HDDs pricing into a perspective, there's a seller in UK which sells 2TB Dell 7.2K RPM SAS HDDs for 28 EUR a piece that's pretty hard to beat.

So I took current prices, avg electricity price of 0.32 EUR/KWh and here are the results:



Sum indicates total upfront drive cost + running them for a year. I've pulled drive power ratings from datasheets and drive labels. Btw CMR drive is also a used SAS drive, I was just wondering does it compare to others; it's also a 3.5" drive so I can't use it "natively".

Apart from it you can see that nothing beats Jet2Holiday that 28 EUR price - only SMR drives are approaching it in EUR / TB efficiency, but:
- They are SMR, so resilver would take days and going beyond mirror doesn't make sense;
- Even two SMRs have almost double upfront cost comparing to SAS, whereas with SAS same redundant 4 TB of capacity would cost me ~140 EUR (All prices are without S&H).

Here are my power calculations. I'm assuming that the drive is idle 80% of the time and is active 20% of the time. That 1.52W idle of SAS are due some special Seagate feature, and I'm hoping I would be able to active that or spin down a drive completely. Otherwise "usual" idle is around 3W.

KWh per year for RAIDz1 has double of a mirror, but even if we take maximum of 92KWh per year at 0.32 EUR we get 29.44 EUR per year for 4 SAS drives (assuming no spindown takes place). I mean... that's not that bad? Prices are not getting lower, but spending double or triple amount of money today so I can break even in 5 years?..

 

[slowlaris]

So I've tried to mitigate fan problem with just bluntly swapping the fans (I've tried a different CPU cooler) and SYS_FAN1 (near the drive cage), but BMC thinks the fans are broken since they have a different RPM setting. I've also tried setting fan thresholds with `ipmitool`, but it didn't work, the same as sending raw commands. I guess it should work, but Fujitsu probably locked it down, or a different raw command ID is needed.

These are the resources I've used:

How To: Change IPMI Sensor Thresholds using ipmitool

This is the discussion thread for the Resource available here. It can also be found by using tabs at the top of the thread. The discussion was copied over from the original thread, which is now locked. The original post follows below, in the spoiler tag:

www.truenas.com

Fan Scripts for Supermicro Boards Using PID Logic

Glorious1 submitted a new resource: Fan Scripts for Supermicro Boards Using PID Logic - Scripts for fan fans and fanboys Since motherboards have no way to access drive temperatures, they can’t really regulate them. The fan control scripts presented here read and respond to both drive and CPU...

www.truenas.com

Quieting an R720xd with ipmitool

I mentioned a bit in this post about my R720XD being louder in comparison to my other servers. This has proven to continue, with the fans spinning up easily as high as 60%. With the fans having a peak capacity, even 40% fan speed is LOUD. Luckily, there's an easy

tcude.net

For now I think I'll put everything back as it was, but what I think should be possible is to plug all fan headers with PWM fan spoofers for Bitcoin rigs (I have to test it though) and drive all fans from a separate fan controller. Granted, you'll need to code in a custom fan curve and it feels a bit wrong to outsource even CPU fan to a user-running service, but approximate savings could be around 20W! Also, there won't be a situation where BMC would crank all fans to the max because the room is too hot or because there's a "Fan test" scheduled to run every midnight (I've had it on mine).

Well, maybe I'd leave CPU fan alone, although 6W at full speed feels like a waste of power, but SYS FAN 1 and 2 and probably PSU fan can be replaced for good, I believe.

 

[celemine1gig]

I would suggest leaving the original fan control alone, but instead spoofing it intentionally via microcontrollers.
Reading in the original PWM signal and modifying it, like intended.
Also make sure to also spoof the RPM signal, in the same proportionality.
So, you would modify:

• The PWM-signal coming from the board, going to your intended fan.
• The RPM-signal, coming from your intended fan, and going to the board.

That should work just fine, if done right. And you could use nearly every fan type, that you like.
If it would be enough for the cooling needs of the system, though, would logically be a different question.