Fujitsu TX1320 M3 - Cheap low power server (barebone)

[luckylinux]

My 1240 v6 is hovering around 80 under high usage so no worries. There is setting in the BIOS for faster CPU fan speed.
Keep in mind that this will just set minimum fan speed to 50% (iirc).
You have to acquire a valid iRMC license to enable remote console.

Oh yeah, extremely annoying on Full which is probably more like 70-80% or so .

Sad that there is no /sys Interface like on e.g. ASUS Consumer Motherboards where you could just do it like

# Set Fans to Fixed Speed
speedreference="225"
mapfile -t fans < <( find /sys -iwholename *hwmon/*pwm* | grep -E "[0-9]$" )
for fan in "${fans[@]}"
do
    # Set to Manual Control Mode
    echo "1" > ${fan}_enable

    # Set Duty Cycle / Speed Reference (0 = Stopped, 255 = Max Speed)
    echo "${speedreference}" > ${fan}
done

I guess it's similar to Supermicro but I would need to Reverse Engineer the Registers to enable Variable Speed Mode.

On the Supermicro I wrote a Python Tool based on B3N Script. I later discovered there was at least a few others around .

At first glance it looks like there aren't many References for ipmitool raw Registers for Fujitsu, so that we could set the Duty Cycle for Variable Fan Speed. And I don't really know how to reverse engineer that ...

 

[luckylinux]

To lower the CPU Temperature a bit, I looked into:

Lower the Minimum CPU Frequency (otherwise I'm stuck at 4.2GHz all the Time):

cpupower frequency-set -d 0.8GHz

Lower the Maximum CPU Frequency:

cpupower frequency-set -u 3.6GHz

Install undervolt Tool (Official Repository):

python3 -m venv /opt/undervolt
source /opt/undervolt/bin/activate
python3 -m pip install --upgrade pip
pip install undervolt

Then undervolt a bit (I tried 80mV on both Core and Cache, it seems to work OK so far):

undervolt --core -80 --cache -80

Like this I lowered the Maximum from 80-82°C to approx. 60°C .

 

[luckylinux]

I was playing a bit around Yesterday and Today.

So far I must say I'm quite DISAPPOINTED.

I just upgraded the BIOS and iRMC to the latest Version:
- BIOS: BIOS_V5.0.0.11_R1.39.0_2024-08-19
- iRMC: iRMC_D3373-B1_09.69F_sdr03.18

The BIOS Options are EXTREMELY limited.

ASPM is set to Auto (only L0s/Auto/Disabled are available as Option), yet the is extremely limited in Terms of what can be done.

- With no PCIe Devices -> CPU Package PC7s without Issues (Idle Power Consumption around 10W)
- With 1 x Mellanox ConnectX-4 LX + 1 x Intel i350-T4 -> CPU Package PC3 (Idle Power Consumption around 25-30W)

Of course this seems to be due to buggy BIOS:

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.11.5+bpo-amd64 root=UUID=afadc67e-4b3e-40e8-bf24-9b1c9a25f39a ro pcie_aspm=force quiet pci=realloc=off mpt3sas.msix_disable=1 pcie_aspm=force
[    0.014910] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.11.5+bpo-amd64 root=UUID=afadc67e-4b3e-40e8-bf24-9b1c9a25f39a ro pcie_aspm=force quiet pci=realloc=off mpt3sas.msix_disable=1 pcie_aspm=force
[    0.014944] PCIe ASPM is forcibly enabled
[    0.014985] PCIe ASPM is forcibly enabled
[    0.180233] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.241658] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.243589] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration

Which I can bypass, but even then, something seems off for the Intel i350-T4, which really does NOT want to work with ASPM.
On the Supermicro X11SSL-F I had no problems to do that (it required pcie_aspm=force as Boot Parameters).
On this Fujitsu Motherboard no luck.

I guess I'll try another PCIe Slot.

EDIT 1: No, no Luck with the Mellanox ConnectX-4 LX. And compared to the Supermicro X11SSL-F, this board doesn't have a PCH x4 in x8 Slot (it's NOT open-ended unlike the Blue ones). The Intel X710-DA2 works out of the Box as usual ...

EDIT 2: with Intel X710-DA2, Standby (OFF) Power Consumption 7W (with 1 x SFP+ SMF LR Fiber Transceiver)

EDIT 3: with Intel X710-DA2, Idle Power Consumption 20W-25W (with 1 x SFP+ SMF LR Fiber Transceiver) ... So even though ASPM works here (compared to the Mellanox ConnectX-4 LX), the Power Consumption is not *that* much Lower. Nowhere near the 10W or so Idle that some Users reported here .

EDIT 4: to be clear, with the Intel X710-DA2, I can get as low as PC6/PC7, so ASPM behaves better compared as with the Mellanox ConnectX-4 (Stuck in PC3). Problem is that this doesn't translate into significant Power Savings at all

 

[luckylinux]

@richard.dzavoronok : do you know how to reset the Error LED Light ?



It's not really an issue and I don't know what's causing it. I just changed the CMOS Battery because the Voltage was extremely low.

Other than that it seems the BIOS Component has an Error Message, not sure why though ...

 

[richard.dzavoronok]

@richard.dzavoronok : do you know how to reset the Error LED Light ?

It's not really an issue and I don't know what's causing it. I just changed the CMOS Battery because the Voltage was extremely low.

Other than that it seems the BIOS Component has an Error Message, not sure why though ...

I had a lot of errors/reboots when I changed the CPU, maybe also yours need a couple of reboots to reset LEDs ?
I'm just shooting in the dark here. I didn't have such an error sorry :/

 

[luckylinux]

I had a lot of errors/reboots when I changed the CPU, maybe also yours need a couple of reboots to reset LEDs ?
I'm just shooting in the dark here. I didn't have such an error sorry :/

Yeah or upgrade the BIOS I guess. I upgraded 1 out of 5 Systems so far, not sure if the old or new BIOS is better to be honest.

 

[Rand__]

@richard.dzavoronok : do you know how to reset the Error LED Light ?

View attachment 41409

It's not really an issue and I don't know what's causing it. I just changed the CMOS Battery because the Voltage was extremely low.

Other than that it seems the BIOS Component has an Error Message, not sure why though ...
View attachment 41408

I have the same, latest Bios, reboots didnt help.

I ignore it

 

[luckylinux]

I just finished the Assembly of the last Piece. I only had a Intel Xeon E3 1245 v6 left. This is giving me trouble. It's OK until GRUB menu, but after that, KVM Video Freezes, VGA Video blacks out, nothing gets on the Network (my LiveUSB running Debian automatically should connect to the DHCP Server and request an IP Address, but nothing shows up).

I just updated iRMC and BIOS to latest, but there is no difference. Something is seriously broken here ...

EDIT 1: worse still, after BIOS Upgrade, I'm now stuck at the "Fujitsu" Initial Screen (NOT the AMI BIOS + Fujitsu Screen which lists Memory etc, I mean the INITIAL Fujitsu Screen which basically only lists the Serial Number and iRMC IP Address if any). iRMC doesn't even list an IP Address

EDIT 2: I tried removing the CMOS Battery and AC Cord, so I'll let it like this for a while and see if clearing CMOS might help. Otherwise maybe BIOS Recovery ? Unsure. Or possibly flashing a slightly older BIOS than the latest one via iRMC ?

 

[richard.dzavoronok]

EDIT 1: worse still, after BIOS Upgrade, I'm now stuck at the "Fujitsu" Initial Screen (NOT the AMI BIOS + Fujitsu Screen which lists Memory etc, I mean the INITIAL Fujitsu Screen which basically only lists the Serial Number and iRMC IP Address if any). iRMC doesn't even list an IP Address

Do you have dedicated GPU installed? If so, the VGA port on the mb will be stuck at the fujitsu logo. The gpu's one should be working properly.
Manual states this: " Remote Video direction via iRMC must be disabled. " in the section with Graphics card. Maybe try that?

 

[luckylinux]

Do you have dedicated GPU installed? If so, the VGA port on the mb will be stuck at the fujitsu logo. The gpu's one should be working properly.
Manual states this: " Remote Video direction via iRMC must be disabled. " in the section with Graphics card. Maybe try that?

Is that Setting really needed ???

I thought that the Intel iGPU wasn't powered at all (thus the TX1320 M3 cannot be used for e.g. Jellifin Server for 3D Acceleration) . So I didn't think there would be any Issues.

I do NOT have a DEDICATED GPU (PCIe Card).

I was about to do BIOS Recovery Procedure now. But then again, that doesn't explain why I'm not getting an IP Address which I can see on the DHCP Server. It's weird. Everything seems to freeze after GRUB Menu (and now stuck on the Initial Screen, after I finished the BIOS Upgrade).

To be clear: I'm just using the Intel Xeon E3-1245 v6 since I ran out of Intel Xeon E3-1270 v6 and I didn't want to buy another CPU just for this, since I already had a v6 around. I didn't specifically want to use this CPU on this System (originally) .

 

[luckylinux]

The Manual is very funny ...

10.1 Flash Memory Recovery Mode
- Prepare a bootable USB stick as described in section Flash BIOS Update with a USB stick.
- Switch off the system and disconnect the power plug.
- Open the chassis and switch on "Recovery" (BIOS-RCVR) using the jumper / DIP switch on the system board.
- Reconnect the power plug and boot the system with the inserted bootable
USB stick.
- Boot the system from the inserted bootable USB stick.
- Change the directory with cd DOS and start the Flash BIOS Update with the command DosFlash. Then follow the instructions on the screen.
- Observe the update process on the screen, until it is finished. The recovery
update may take several minutes.
- Switch-off the system and disconnect the power plug.
- Remove the USB stick.
- Return the "Recovery" (BIOS-RCVR) jumper / DIP switch which have been
changed to the initial position.
- Reconnect the power plug and switch on the system. The system will be booted with the new BIOS revision.
- Check the settings in the BIOS setup utility. If necessary, reconfigure the settings again

Like ... "Boot the system from the inserted bootable USB stick." if the BIOS is bricked

EDIT 1: I attempted now for a 2nd Time and I definitely can hear a lot of power on/power off cycles (you can hear the "clicks" of a Relay, I assume inside the PSU) like after a BIOS Upgrade (probably because it needs to "recalibrate" the Memory Speed JEDEC Timings etc).

But nothing really happens. Then it gets stuck again.

EDIT 2: iRMC Log Shows

POST - A recovery capsule couldn't be located during BIOS recovery flash

A ... "Recovery Capsule" ? Like what, is this NASA Software or what now ?

EDIT 3: I'll try with a known working extremely old FreeDOS USB Stick that at least previously worked. No clue why this one doesn't ....

EDIT 4: with the Old FreeDos USB Stick it also didn't work

EDIT 5: with the Old FreeDos Stick AND with the Files of the Fujitsu downloaded BIOS in the "ROOT" Folder of the USB Drive, it started flashing automatically and beeping. At least something seems to have happened

EDIT 6: I flashed the Recovery successfully (1.38.0 to try a different one than the failing 1.39.0). But nope, still Stuck on the Initial Screen. I took less than to setup the other 5 Systems and lost an entire Day trying to setup this one



EDIT 7: What gives ?

I tried to switch back to the Low Firmware Version that was working before and it's still stuck.

Furthermore I don't know ... it seems it won't detect any Memory or what ? I previously also tried with a single Stick of RAM, same Result.

Or is this simply because the Boot Process is not going forward for some Reason ?

The VERY first Time it kept freezing just after selecting "Debian (Default)" in GRUB Menu. Now it's not even going to the point that I can enter BIOS Setup.

It seems well and truly broken.





EDIT 8: Now apparently the Boot Watchdog kicks in as well ...

BIOS POST Watchdog - Action: Power Cycle (Post Code: 0x78)

EDIT 9: Damn ... it looks like you cannot downgrade the firmware even in recovery mode
Googling fujitsu tx1320 m3 downgrade bios states [I]Downgrading is disabled by default from the tool provided by fujitsu. You need to disable the BIOS build date time checking in order it to work.[/I]that on a Fujitsu Forum Thread, but unfortunately all of them have been taken down and Wayback Archive Machine doesn't contain those pages apparently

I can only read part (maybe the full ?) of the Command, but still that doesn't help, because Recovery Firmware is an automated Process, despite what the Manual Says (change Directory and Run Script)

dosflash /bbl

EDIT 10: Now after powering off for a bit and another Recovery Flash using 1.30.0, it seems to be on that Version.
Still stuck at boot though but at least the CPU and 1 x DIMM (I temporarily removed the other 3x) are getting detected ...



Not that it matters, at the end of the Day, it's still NOT working at all ...

 

[luckylinux]

On another Note, has anybody had any success with SR-IOV ?

There is no Option in the BIOS, heck not even hidden inside the disassembled BIOS .

According to this SUSE Note (PRIMERGY® TX1320 M3), SR-IOV appears to NOT be supported .

Which is a Shame considering that on Supermicro Motherboards there is an Option to enable SR-IOV as well as MANY other Options ...

 

[luckylinux]

For Information, I installed an Intel Xeon E3-1230 v6 and the System booted up just fine (as it should).

Whatever the Reason, it seems that the Fujitsu TX1320 M3 does NOT like Xeon E3 CPUs with Integrated Graphics (E3-12x5 v6). Or I got a faulty CPU, that could also be a Possibility.

 

[Rand__]

I ran 1245 v6's on multiple boxes... Had issues with PCIe (bifurcation, nvme compatibility, see early posts here) but never attributed that to the 5...
All in all the 5 have been running fine for years, so wouldn't say they're problematic per se

 

[luckylinux]

I ran 1245 v6's on multiple boxes... Had issues with PCIe (bifurcation, nvme compatibility, see early posts here) but never attributed that to the 5...
All in all the 5 have been running fine for years, so wouldn't say they're problematic per se

Yeah I think I saw your post in Page 1 or 2. Don't know what to think. With the E3-1245 v6 it seemed to work once, boot into LiveUSB, then absolute Nightmare, sometimes being able to go into BIOS. But lately absolutely not working.

Put the E3-1230 v6 in and it worked out of the Box.

Could be a faulty CPU. Or something else entirely. I didn't want to lose too much sleep over 30 EUR though. Already lost a couple Days.

 

[Stueckchen]

Hey there,
did anyone found out how to Control the FAN?
I would like to cool an Tesla P4 with an FAN. On Fan Header SYS3 its to low on SYS4 its to load?

 

[luckylinux]

Hey there,
did anyone found out how to Control the FAN?
I would like to cool an Tesla P4 with an FAN. On Fan Header SYS3 its to low on SYS4 its to load?

Nope . As I said there is no HWMON Exposed to the End System. And there doesn't appear to be a Fan Control Mechanism (besides Auto/Other). That's the same that the official Documents report for iRMC. Without knowing how they coded their IPMI Registers & Protocol there is no Way to find out how/if we can set the Duty Cycle of the Fans. Otherwise you could send a RAW Command via IPMITool to set the Fan Duty Cycle, like we can for instance do on some Supermicro Motherboards.

The only viable "Solution" might be an ESP32-xx Controller and some small components (fan Headers etc) hoping that iRMC won't complain that the measured Fan Speed doesn't match the Reference Value. Or you need to "fake" the Tacho Signal if that happens of course.

 

[luckylinux]

Also might be interesting for some:
Slot 3 is a PCH port and can be further bifurcated. If you have full access to the BIOS flash descriptor (i.e. Programmer ready and direct flash access). This way I had an Intel Optane H20 drive in PCIe 2x2 config running in their.

Are you actually sure about that ? And where do you see it can Bifurcated ?

Usually the Color is an Indicator and the first 3 PCIe Slots (Slot 4, Slot 3 and Slot 2) are Blue, while the latter one (Slot 1) is Black.




I'm NOT sure if I ever tested to put the Mellanox ConnectX-4 LX in Slot3, but USUALLY ASPM works if it's a PCH/DMI/Chipset connected Slot, whereas USUALLY ASPM does NOT work if it's a direct CPU-connected Slot.

I'm afraid that PCIe Slot3 is also directly CPU Connected and NOT PCH/DMI/Chipset Connected .

 

[cowmix]

Has anyone tried a dGPU? Or tried methods of forcing L1 aspm? I was thinking of getting one of these and using like an intel arc for transcoding since you can't use a CPU with an iGPU but the notes about not having an L1 ASPM concerns me that the gpu would not be able to fully idle with this motherboard when not in use?

 

[celemine1gig]

Are you actually sure about that ? And where do you see it can Bifurcated ?

Don't misunderstand me, or interpret the answer as arrogant, but yes, I am sure. Edit: Scratch that. I just realized, that I wrote port 3 (as I counted from the top), while in the Fujitsu documentation this is actually port "2". So, I am talking about the 1x4 port.
(About said 1x4 port):
It is a PCH port, which can be bifurcated through low-level settings in the Intel Flash Descriptor (first 4K of SPI storage space in the Boot SPI Flash). Also the Intel corresponding Intel "fitc" tool lets you modify it via a GUI.
You can then reconfigure the port to no longer be a 1x4 port, but rather a 2x2 (or maybe even 4x1; don't remember) port. And for a dual function Optane, like the H10 or H20, you will need a split port with 2x2.
As I said, as proof I had the mentioned Intel Optane running in there, with both of its functions active and working.
Edit: The two 1x8 ports are actually the already bifurcated PEG-slot. So, the 1x 16 PEG lanes, split into 2x8. Theoretically, one of those ports (which one, depends on the question of enabling lane reversal or not) could be further split into 2x4, as the PEG supports 1x8 and 2x4. This would need to be done via CPU Straps. See here: S/W strap for x16 or x8/x8 on PEG

Usually the Color is an Indicator and the first 3 PCIe Slots (Slot 4, Slot 3 and Slot 2) are Blue, while the latter one (Slot 1) is Black.
View attachment 43077

View attachment 43078

Sorry, but do you really think, that differentiation by color coding is a safe way of telling the connection type, when you can actually analyze the function in an OS or the EFI-shell? A simple "lspci -t" in Linux will tell you where a device connection actually goes, port-wise. Very helpful stuff.

I'm NOT sure if I ever tested to put the Mellanox ConnectX-4 LX in Slot3, but USUALLY ASPM works if it's a PCH/DMI/Chipset connected Slot, whereas USUALLY ASPM does NOT work if it's a direct CPU-connected Slot.

I'm afraid that PCIe Slot3 is also directly CPU Connected and NOT PCH/DMI/Chipset Connected .

"Usually" seems to be one of your favorite words. I would say that "usually" no recent Connect-X card with a recent firmware version has enabled ASPM, unless you somehow force it.
If you are looking for ASPM and 10G at the same time, then Intel X710 is a nice choice.

Has anyone tried a dGPU? Or tried methods of forcing L1 aspm? I was thinking of getting one of these and using like an intel arc for transcoding since you can't use a CPU with an iGPU but the notes about not having an L1 ASPM concerns me that the gpu would not be able to fully idle with this motherboard when not in use?

There is no real reason, why a dGPU should not work. It's just that, due to the onboard BMC, integrated CPU graphics was fully disabled.