Hi everyone, this thread is a continuation of similar thread for M3 model. Now the time has passed M4 is getting cheaper and it's possible to get a quite good deal on eBay. I've seen offers as low as 130 EUR for dual 3.5 HDD model (which I sadly missed), but I was still able to get a good deal on dual redundant PSU 8x2.5 model.
I will quote my post from a previous thread to share my experience.
I will quote my post from a previous thread to share my experience.
OK I finally got M4 and can report some findings here.
The server arrived with Xeon E-2236, 16+16 GB of RAM, 1TB Samsung 970 PRO and NVIDIA GT 710. It also has two redundant 450W PSUs. I've installed latest Ubuntu 24.04.3 just to test the hardware and power consumption.
dmesg: dpaste: ACXCBPVLH
Code:root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci 00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07) 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07) 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) 00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10) 00:15.1 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10) 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) 00:16.1 Communication controller: Intel Corporation Device a361 (rev 10) 00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10) 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) 00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0) 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0) 00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0) 00:1c.3 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 (rev f0) 00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0) 00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10) 00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10) 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) 00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) 01:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) 03:00.0 PCI bridge: Emulex Corporation x1 PCIe Gen2 Bridge[Pilot4] 04:01.0 Co-processor: Fujitsu Technology Solutions Device 1228 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
First observations: right from the start, it's impossible to turn it on until iRMC finished booting and ID led blinked (around 20 secs after power is applied). Maybe it's a standard server thing which I already forgot...
Also, sometimes (every other time) I'm unable to enter the BIOS or boot menu after hard power off, keyboard just doesn't work (I've tried 3 different ones). After I boot the OS and then reboot from it, I can enter BIOS just fine. Weird.
PSU
Since I've got dual redundant PSUs, BMC gives a warning when one of them is not connected, which's okay but IMO overkill for a home lab use. I disassembled one of them to check if it's viable to do Noctua swap as @Gruenschnabel wanted to do, but I'm not really sure it's worth it:
https://www.fansco.com/uploads/2024/04/MGT4012UB-W28-Protechnic.pdf this is the fan inside (but 4 pin), and per suitable Noctua NF A4-20PWM specs, Noctua's flow rate is ~6CFM (vs 23 CFM) and RPM is also 5K vs 13K. I think what will happen is that BMC will try to force full 13K RPM on startup/check and won't be able to do that. All of this of course if PWM signal is the same. Last nail to the coffin for me is the pricing: new Noctua fan is 16 EUR, while non-redundant PSU is 17 EUR on eBay... the only downside of standard PSU for me (apart from max power, but I doubt I'll go over that) is that it's not rated as Premium, so efficiency will be lower. You win some, you lose some...
Lastly, noise: redundant PSUs are not loud per se, but this small fan is running all the time and produces quite audible low-frequency hum, which only goes away when everything's off and the plug is pulled. This hum is very well transmitted over the floor and I can tell the difference when the server is switched on in the other room.
Drives backplane and controller
Per @hmartin 's blog suggestion I've replaced the backplane with a unified one (where single board can handle 8 drives). I wish I had read further about SATA card
So long story short, I've decided to buy Broadcom HBA 9500-8i SAS/SATA/PCIe(NVME) 12Gb/s SAS3808 controller for 100 EUR from China. It works, but the problem is that it has Slimline SAS x8 connector. The only matching cable I was able to find to plug into backplane has straight connectors, which prevents backplane fan shroud from closing. That's very disappointing. I've looked into Fujitsu spare parts list, and their controllers based on the same chip (like EP520i) have "standard" SFF-8643 connectors, which allows plugging cheap and vastly abundant Mini SAS HD (SFF-8643) cables. Unless I find an angled cable or adapter I won't be able to use the this controller. Also, backplane fan is quite noisy and power hungry: Delta DS07025B12U, 12V 0.7A. Fan from M3 is 0.3 or 0.4A, but is also thinner by approx 5mm, so it won't fit in the same frame... and of course 70x70mm is pretty non-standard size and apart from Delta and random Chinese vendors you won't find anything. Ah yes, fan shroud + fan for M4 costs 46 EUR on eBay...
I've also had a random LSI 3008 controller laying around which I wanted to compare. As you can see, 9500 has ASPM support enabled, while 3008 does not. Also 3008 gets really got even when nothing is plugged in.
Code:root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci -vv -s 01:00.0 01:00.0 Serial Attached SCSI controller: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx Subsystem: Broadcom / LSI 9500-8i Tri-Mode HBA Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 IOMMU group: 1 Region 0: Memory at 9a200000 (64-bit, prefetchable) [size=1M] Region 2: Memory at 9a100000 (64-bit, prefetchable) [size=1M] Region 4: Memory at 9c300000 (32-bit, non-prefetchable) [size=1M] Region 5: I/O ports at 6000 Expansion ROM at 9c400000 [disabled] [size=256K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s (downgraded), Width x8 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS- LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: Upstream Port Capabilities: [b0] MSI-X: Enable+ Count=128 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=0 offset=00003000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [148 v1] Power Budgeting <?> Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [168 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [188 v1] Physical Layer 16.0 GT/s <?> Capabilities: [1b0 v1] Lane Margining at the Receiver <?> Capabilities: [218 v1] Dynamic Power Allocation <?> Capabilities: [248 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [348 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Capabilities: [380 v1] Data Link Feature <?> Kernel driver in use: mpt3sas Kernel modules: mpt3sas
Code:root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci -vv -s 01:00.0 01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) Subsystem: Lenovo ServeRAID M1210 SAS/SATA Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 IOMMU group: 1 Region 0: I/O ports at 6000 Region 1: Memory at 9b300000 (64-bit, non-prefetchable) [size=64K][/SPOILER][SPOILER=lspci of SAS3008] Region 3: Memory at 9b200000 (64-bit, non-prefetchable) [size=1M] Expansion ROM at 9b100000 [disabled] [size=1M] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <2us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [c0] MSI-X: Enable+ Count=97 Masked- Vector table: BAR=1 offset=0000e000 PBA: BAR=1 offset=0000f000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 04000001 0000000f 20010000 66fe5e71 Capabilities: [1e0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [1c0 v1] Power Budgeting <?> Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: megaraid_sas Kernel modules: megaraid_sas
Also I able to confirm that 2nd M.2 slot doesn't recognize NVMe SSDs. I don't have any MSATA M.2 SSDs around, and since I wanted to have NVMe mirror, I plan to utilize that Viking U.2 adapter. The only question is where to put it, and there are not many options: either PCIe slot (but then it will take 1.5 slots, and won't be able to cool it properly), or try to construct some kind of holder to put into RDX drive slot, which has very weird 1.75x4 inch dimensions. Maybe I'll buy some old RDX drive just for the parts...if this works out, two small Noctuas should cool it reliably, but then again total solution cost is quite high. (Viking 20 + RDX frame 30 + U.2 to M.2 cable + 20 == 70 EUR).
Power consumption
Finally, some numbers (taken from the wall).
Switched off (iRMC booted up): 7W (I blame redundant PSU fan)
Booted into desktop (NVMe SSD and GT710): 24W
9500 controller installed: 28.4W (no drives attached)
3008 controller installed: 33W (no drives attached)
These are without any powertop tuning. All C-states are enabled in BIOS, but I feel like at least 10W can be saved if those Delta fans were replaced.
