Fujitsu TX1320 M3 - Cheap low power server (barebone)

[3333]

Has anyone ever transplanted one of those to a SM chassis?

I run one of these as TNC box (very nice low power) but need some more drives so was considering transplanting to a 216 or 836/846.
2x8 and one x4 slot are perfect for a HBA, 10 (40G) card and single nvme

In the end I assume Id only use the board and the psu (and maybe a fan so it won't complain), right? I dont mind If i cant use screws, its not as if I move those boxes all day long...

I was thinking about the same thing some time ago and I was doing the research. I haven't done it, but I've seen several posts stating that the PSU can be replaced with anything else, but you'll need and adapter, as Fujitsu has different pinout and longer or shorter plug. But what you also need is this little front panel thing with power button as the server won't boot up if it's not detected.

 

[yoggi]

@yoggi, I'm pretty sure my machine is acting similarly, when it's started after the power break. However, when the server was powered off, but still plugged in, then it's not rebooting during startup and doesn't have the 2 minutes lag.

@3333, It is like the machine is waiting at the boot screen, I have looked through the bios settings many times but I can't find anything that affects the waiting. The machine is set to boot from disk, silent booting etc.

Do anybody have any idea why my machine is waiting at boot when there is no error?

yoggi

 

[yoggi]

I was thinking about the same thing some time ago and I was doing the research. I haven't done it, but I've seen several posts stating that the PSU can be replaced with anything else, but you'll need and adapter, as Fujitsu has different pinout and longer or shorter plug. But what you also need is this little front panel thing with power button as the server won't boot up if it's not detected.

@3333 and @Rand__
This page might be helpful inorder to determine what needs to be transfered with the motherboard when moving to a new chassi

Fujitsu TX140 S2 | «WatchMySys» Blog

From same website, good information about the TX1320 M3
Fujitsu TX1320 M3 | «WatchMySys» Blog

 

[luckylinux]

I still find it hard to believe you can get such low Power Consumption Numbers around 10-15 W at Idle .

I think my Supermicro X10SLL-F E3-1230 v3 is around 40-50 W and the Supermicro X11SSL-F E3-1230 v5 is probably around the same. This with 4 sticks of RAM (maxed out at 32GB DDR3 ECC UDIMM and 64GB DDR4 ECC UDIMM respectively) and 2-6 x Crucial MX500 SSDs, 4 x Arctic F8 PWM Fans, CPU Cooler NH-L12s with Fan PWN controlled and a Seasonic SSR-360GP 360W PSU (80+ Gold).

Granted, some of them have a LSI 9211-8i HBA in IT mode passed through to a VM and/or some Mellanox ConnectX-2/3 10gbps NIC.

So indeed there might be something going totally wrong with ASPM. And a 80+ Gold PSU isn't very efficient at all at around 10-15 % load ... And yes, Supermicro BMC is probably adding like 10W of Power Consumption just by itself .

I plan on replacing the Mellanox ConnectX-2/3 with either an Intel X710 or at least a Mellanox ConnectX-4, both of which (the Intel much better than the Mellanox) support ASPM low power states.

But considering how some Members achieved around 20W-25W with an Intel X540 (which does NOT support ASPM) I'm a bit puzzled.

 

[chrisgassen]

To contribute to this amazing thread: I designed some fan shrouds for my TX1320 drive bays which are available here:

Fujitsu TX1320 M3 fan bracket - standard length by chrisgassen

fan bracket for TX1320 M3 (maybe also for other models, feel free to contact me, so that I can update here!)The model has roughly the same dimensions as the original part, it needs M3 screws to attach the fan though. Either M3x20 if you cut a thread or M3x25 if you use nuts (then the screwhole...

www.thingiverse.com

https://www.thingiverse.com/thing:6572448
(additional room for cables, with the downside of having to assemble the cpu heatsink afterwards...)

 

[Black_Pyramid]

Hi,

how can I reset the iRMC S4 to factory defaults? I could not find anything in the web interface.

 

[IvanBayan]

If someone need, there is a holder for 2.5" disks in place of BBU holder.
Designed for tx1320 m3, though not sure if it fit to server with double hotplug PSU.
Looks like C26361-K644-C970 used not only in TX1320 M3, so may fit other FSC servers as well.

Fujitsu BBU 2.5" HDD/SSD holder by IvanBayan

It's a holder for 2.5" HDD or SSD drive in place of Fujitsu BBU Holder C26361-K644-C970Tested with TX1320 M3

www.thingiverse.com

 

[luckylinux]

I might chime in here, as I happen to have played a litte with one of these really nice little machines, too.

Happy to share some takeaways, too:

• power consumption: 9W idle is the lowest I managed to get (proxmox/debian + powertop --auto-tune, just one SATA ssd). I doubt you can get any lower given that the iRMC draws about 4.5 W alone (if someone would find a way to switch it off, though...)
• CPUs: tested Pentium G4560, Xeon 1220 v6, and Pentium G4600 (which is not on the official compatibility list, marginally faster than the G4560, works fine)
• caddys: the previously mentioned STL file on thingiverse works sort of (3d-printed a few myself), at least for blind caddies. BUT: the screw holes are "on the wrong side" and not within the right distance from the backplane to let you connect hdds/ssds (at least for my TX1320 M3). Also there is a slightly thicker part on one of the side pieces, that does not really seem to go too well with the the TX1320 once you have screwed in a ssd or hdd. So I have been fiddling for a while with tinkercad (I'm no engineer!) to "fix" this in the STL file and ended up with something that's actually working well enough for my needs. Did not bother to upload the new file on thingiverse yet as the result is certainly far from perfect, but happy to share it here (check it out for yourself and only use at your own risk, of course!). Printing in PLA and PETG both works fine, could not test ABS (would be my preferred material due to highest ruggedness). Stiffness etc. is good enough in either material. Temperature should be no issue as you probably do not plan to let temps go up higher than 50 Celsius among the drives, I presume.
• orange blinking led: whenever you change CPU or RAM that's detected (hence the blinking led); to turn that off just go into BIOS (it will report the hardware change very briefly on screen, if I remember correctly), maybe saving (unchanged) BIOS settings is required too, to get rid of the blinking led (can't remember). After a new start the blinking led should be gone.
• 3.5" hdd: it's quite possible to fit in one in the big bay near the power supply. I had to very slightly bend the pretty stiff sheet metal guiding rail, though. And there are even two nicely fitting holes on the outer side of the cage to fix the drive with standard screws, which I found sufficient. No vibrational decoupling from the rest of the case, however.

It would be absolutely great to find some way to mount a second 3.5" hdd for a zfs mirror. But that seems almost impossible given the cramped space (without removing the 2.5 bays, of course).

Also, should someone find a simple way to elegantly improvise a 2x 3.5" cage for the space the 2.5" bays occupy (without having to purchase the OEM parts), I might consider getting a second machine. ;-)

Very Late Reply and not sure if you found a Solution in the meantime.

I don't currently own this PC but I'm tempted to get a few of them to use as OPNSense Router .

You mentioned about disabling the iRMC to lower Power Consumption.

There might be a Way.

Taking the latest BIOS I could see that there is a ServerMgmtSetup Section, besides the usual Setup Section.

Veryy crude Patching might be done like this (untested of course):

setup_var.efi ServerSetup(0x2):0x0=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x1=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x452=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46F=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x459=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45E=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45A=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45D=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x463=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45C=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x458=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45F=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x460=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x462=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x464=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x465=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x466=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x468=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB3=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB5=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB4=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB6=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB7=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB9=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xBA=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB2=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xBB=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x454=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x455=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x456=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x457=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x469=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46A=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46B=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46C=<To_Be_Manually_Filled_In>
setup_var.efi SystemAccess(0x1):0x0=<To_Be_Manually_Filled_In>
setup_var.efi fjOnboardVideoGrayout(0x3):0x0=<To_Be_Manually_Filled_In>

I just quickly did a survey scan on V5.0.0.11_R1.39.0_2024-08-19 on the D3373-B1.UPD File. Not sure which one is the "Best" or "Correct" one.

If I understand right, when you have

SuppressIf
                        True

Or

DisableIf
                        True

Then that Option is hidden from the BIOS.

The Options could be searched e.g. via "Enable" or "Power" etc.

EDIT 1: Option Power Saving Mode", Help: "[Enabled] iRMC will be powered down, when system is shut off. [Disabled] iRMC will go into stand-by mode, when system is shut off. This allows communication with iRMC even when the system is powered off. could be useful if you turn OFF the system and don't need iRMC to be able to turn it back ON.

Whereas OneOf Prompt: "BMC Support", Help: "Enable/Disable interfaces to communicate with BMC", QuestionFlags: 0x10, QuestionId: 0x1, VarStoreId: 0x2, VarOffset: 0x0, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x1, Step: 0x0 looks like the main "Switch", but I guess it only disables the Network Part of it, not the whole BMC ....

 

[luckylinux]

Has anybody found good Deals ? The original Price of 29 EUR that @Rand__ originally listed 2 Years ago is now at the very least 61 EUR (plus Shipping) and no Option to place an Offer. I can always try to contact the Seller (alles4pc! on Ebay) and see if they agree to a lower Price if I buy 2-4 of these Barebones but there is no guarantee that they will . No Competition unfortunately .

Plus the price of the Caddies of course ...

I assume that @Bjorn Smith you don't need any more Caddies, do you ? I also live in Denmark by the Way .

I thought to go with the readily available HP PN "371589-001" otherwise, although those are around 4-6 EUR / Piece plus Shipping ...

Thanks for your Help

 

[Black_Pyramid]

29 € or 61 € IMHO does not really matter. The ECC RAM for this barebone is much more expensive than this price difference. Also you have to print the caddies or the price will multiply.

Do those HP PN "371589-001" fit? Was it announced here in the thread? EDIT: Yes, I've found it.

 

[luckylinux]

29 € or 61 € IMHO does not really matter. The ECC RAM for this barebone is much more expensive than this price difference. Also you have to print the caddies or the price will multiply.

Do those HP PN "371589-001" fit? Was it announced here in the thread? EDIT: Yes, I've found it.

Well if you order 4-5 Servers I'd say it matters . Less is More .

Yeah, RAM is more expensive for sure. Best I could find for ECC Unbuffered was 34 EUR / piece for PC4-2400T. So 4 x 16GB = 136 EUR.

Not too bad I'd say considering I was paying 55 EUR for new a few Months ago ...

 

[Black_Pyramid]

Well, yes, you are right. In May I've paid 245 € for 8 pieces of 16 GB Samsung ECC unbuffered 2400T. That's less than I've thought. I've thought the price was for 4 pieces only

But I'm pretty sure you won't get the barebone for 29 € again. The 61 € is the cheapest I have seen and I've bought three of them from alles4pc! on Ebay, too.

 

[Black_Pyramid]

Does anybody run a SAS card to use all 8 disk slots?

I wonder if the tiny airflow of the fans running at low speed is enough to cool it down. Obvious choices would be:

• SAS3008 based card (9300-8i): Has the SFF-8643 connector, so original cables can be used. But it draws huge 13 W and only has a small heat sink. Old and cheap (40 €).
• SAS3408 based card (9400-8i): SFF-8643 connector, draws 10 W and has a bigger heat sink, 120 €.
• SAS3808 based card (9500-8i): 7 W, 150 €. But new cables needed (50 €). New cables also means new fan brackets, that have been postet above.

Any other options?
What's your experience?

 

[luckylinux]

Well, yes, you are right. In May I've paid 245 € for 8 pieces of 16 GB Samsung ECC unbuffered 2400T. That's less than I've thought. I've thought the price was for 4 pieces only

But I'm pretty sure you won't get the barebone for 29 € again. The 61 € is the cheapest I have seen and I've bought three of them from alles4pc! on Ebay, too.

I got a very slight Discount and bought 6 Servers . I might have exaggerated

Might see if I got enough RAM otherwise I'll "just" install 32GB per Server and upgrade later on if I find a better deal.

Although to be fair SSDs are also crazy expensive (and I wanted to buy some Intel S3610 but eBay denied Purchase from the US)

 

[luckylinux]

Does anybody run a SAS card to use all 8 disk slots?

I wonder if the tiny airflow of the fans running at low speed is enough to cool it down. Obvious choices would be:

• SAS3008 based card (9300-8i): Has the SFF-8643 connector, so original cables can be used. But it draws huge 13 W and only has a small heat sink. Old and cheap (40 €).
• SAS3408 based card (9400-8i): SFF-8643 connector, draws 10 W and has a bigger heat sink, 120 €.
• SAS3808 based card (9500-8i): 7 W, 150 €. But new cables needed (50 €). New cables also means new fan brackets, that have been postet above.

Any other options?
What's your experience?

I don't have the Server yet ... Just some Observations based on similar (?) Experiences.

Concerning Airflow/Temperatures: I have some Servers in a Silverstone ML03B. Some Servers had 3 PCIe Cards (1 x Mellanox ConnectX-2/3 10gbps NIC, 1 x LSI 9211-8 HBA, 1 x Intel HP NC364T) "Sandwiched" with no Room to breathe. The Mellanox was likely to hit 110-125 °C (!). In one Server, when 2 out of 4 Fans stopped working (and Monitoring apparently did NOT work), the HP NC364T NIC started throwing out Errors on a regular basis. It's been used for a long time and it's an old NIC, will be replaced by an i350-T4. It's not an Issue for this Part. But think about it if you have something more "precious" nowadays in that "Sandwitch".

Concerning Power Losses: the Power Loss of the HBA itself is kind of Peanuts a) Compared to 8 x SSDs / HDDs that are active b) Secondary to the additional Power Losses due to the System NOT being able to "sleep" with deep PACKAGE C-States (notably PC6 and better/higher such as e.g. PC8 on newer Generations). A big part of it is due to ASPM. See Z8 Posts here and here about ASPM, as well as my Repository for some Ideas (nothing "Final", I'm also learning along the Way).

 

[luckylinux]

By the way, does somebody have Information about how these PCIe Slots are connected/routed ?

I tried going through the User Manual, the Operation Manual, the Upgrade Manual as well as the BIOS Settings Manual but I couldn't find anywhere this Information (not even in the disassembled BIOS ). There is no clear Block Diagram like it is on the Supermicro Motherboard Manuals for Instance. Fujitsu has a LOT of Documentation, but not that (unless I missed it). I guess it shold be possible to determine via lspci -tv and then check if the PCIe Slot is connected to a Root Port (in that case it's in a CPU-connected PCIe Slot) or if it's "Parent" is the Chipset C236.

I ask because NICs should as the Mellanox ConnectX-4 LX work with ASPM only (thus far for me at least) if they are connected in a PCH-connected (Chipset/PCH/DMI) PCIe Slot. **NOT** in a CPU-connected PCIe Slot (cannot remember if like that the best I achieved was PC3 or even PC2, anyways it was not super good) !

 

[Black_Pyramid]

Concerning Power Losses: the Power Loss of the HBA itself is kind of Peanuts a) Compared to 8 x SSDs / HDDs that are active

Well, most of the time everything is just idling in my home server. The power consumption of my TX1320-M3 with 64 GB RAM, 2 NVMe operating system disks and 6 SATA SSDs currently is about 20 W. The 6 SATA SSDs may have increased power usage by 6 W, sadly I don't have exact numbers by hand.

b) Secondary to the additional Power Losses due to the System NOT being able to "sleep" with deep PACKAGE C-States (notably PC6 and better/higher such as e.g. PC8 on newer Generations). A big part of it is due to ASPM. See Z8 Posts here and here about ASPM, as well as my Repository for some Ideas (nothing "Final", I'm also learning along the Way).

I know these problems.ASPM is activated in BIOS but there is already a device, that does not support it: the Matrox graphics card. As I'm running FreeBSD I don't have exact numbers because the lack of the nice powertop, but at least C3 states are reached by the CPU:

# sysctl dev.cpu.0 | grep cx
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 20376 4363 58133
dev.cpu.0.cx_usage: 24.58% 5.26% 70.14% last 21913us
dev.cpu.0.cx_lowest: C8
dev.cpu.0.cx_supported: C1/1/1 C2/2/151 C3/3/256

Also note this from your link:

Here’s where things get really interesting. The official LSI users guide for the 92xx series HBAs explicitly lists ASPM as a feature, but ASPM is set to disabled. Now why would that be? Turns out that back in 2013 someone filed a bug report stating that their system would constantly lock up during high read workloads. The solution for this turned out to be to simply disable ASPM. For any of the MPI v2.0 chipsets listed in mpi2_cnfg.h ASPM will explicitly be disabled.

As I'm running FreeBSD this does not apply. I guess this really would be try and measure.

Still my initial question was more about cooling of these cards. I doubt that a 13 W device with small heat sink is cooled down enough.

 

[luckylinux]

Well, most of the time everything is just idling in my home server. The power consumption of my TX1320-M3 with 64 GB RAM, 2 NVMe operating system disks and 6 SATA SSDs currently is about 20 W. The 6 SATA SSDs may have increased power usage by 6 W, sadly I don't have exact numbers by hand.


I know these problems.ASPM is activated in BIOS but there is already a device, that does not support it: the Matrox graphics card. As I'm running FreeBSD I don't have exact numbers because the lack of the nice powertop, but at least C3 states are reached by the CPU

20W at "only" PC3 and with 6 SSDs + 2xNVME and only 1W/SSD at Idle ? That's VERY good actually. Do you trust your "Kill-A-Watt" Measurement Device though ? Or is it oscillating a lot / "unstable" ?

Are you sure that is the "Package" C-State though ?

I don't have much experience with FreeBSD, just a bit a long Time ago and some headaches trying to build some Ports/Packages on OPNSense .

Lucky you that it works though. It MIGHT also work on Linux but it was disabled for a Reason, not sure it's worth the Risk of compiling a Kernel from Source with that Patch/Quirk removed.

Still my initial question was more about cooling of these cards. I doubt that a 13 W device with small heat sink is cooled down enough.

Well given that there is almost no Airflow in the Chassis with significant Static Pressure I'd tend to agree. My Issue was on the Silverstone ML03 with 4 Fans blowing Air into the Chassis. With 2 Fans failed, the HBA/NIC were cooking essentially.

Put some 40mm or possibly 60mm Noctua on it or something .

On the cheap there are plenty of 40mm Fans also on Digikey that you can buy also from relatively Reputable Brands (let alone Aliexpress, Amazon, ... although I'd say they can be more Expensive). I'll have to investigate whether it's possible to replace those proprietary 70mm Fans with some standard 80mm ones. I was thinking of using some 3D Printed Adapter for 70mm -> 80mm Fans that can be found on Thingiverse (e.g. this) or similar Sites. Since I don't have a 3D Printer, I'll have to order online I guess. Not sure if there are other Locations where it's possible to add a Chassis Fan though.

Not sure why Sunon requires some additional Protection though (fuse), that seems stupid. Unless this is because it's meant for System Integrators and NOT targeted to Motherboards as an End-Product. Indeed you'll need to crimp these and make a Connector (not a huge deal given it's 2x Cheaper than the others).

 

[luckylinux]

I just received my first Fujitsu TX1320 M3 Server.

Installed 64GB of DDR4 Unbuffered ECC Memory, an Intel Xeon E3-1270 v6, booted a Debian LiveUSB as I usually do.

Change Password for User sp_adm:

ipmitool user set password 2

Set IP Address Source to DHCP:

ipmitool lan set 2 ipsrc dhcp

Then reboot the IRMC:

ipmitool bmc reset cold

So I can then access the WebUI and get the following Information:

[TABLE]
[TR]
[TH]BIOS Version:[/TH]
[TD]V5.0.0.11 R1.18.0 for D3373-B1x[/TD]
[/TR]
[TR]
[TH][/TH]

[TD][/TD]
[/TR]
[/TABLE]
[TABLE]
[TR]
[TH]Firmware Version:[/TH]
[TD]9.08F (Base: 9.08.F)[/TD]
[/TR]
[TR]
[TH]Firmware Date:[/TH]
[TD]Mar 5 2018 - 14:02:35 CEST[/TD]
[/TR]
[TR]
[TH]Firmware Running:[/TH]
[TD]Low Firmware Image[/TD]
[/TR]
[TR]
[TH]SDRR Version:[/TH]
[TD]3.16   ID 0458  TX1320M3[/TD]
[/TR]
[/TABLE]

BIOS seems to be latest, iRMC definitively NOT (from Fujitsu Technical Support pages from Fujitsu United Kingdom latest seems to be either D3373-B1_09.69F_sdr03.18 (26/08/2022))

EDIT 1: actually the BIOS seems a bit old. I was mislead/confused since it's already V5.0.0.11. But the sub-version is only R1.18.0, while R1.39.0 is latest

However:
- Fans appear to be stuck at minimum Speed (CPU cooking at 81-82°C during a Linux Kernel Build which I use as a stress test with make -j 16) and there does NOT seem to be a Fan Setting (not in the iRMC at least)
- I cannot seem to find the way to get remote Console (KVM) working - Where is that ?

EDIT 2: To list the Users on the System, just issue (apparently it's on Channel Number 2):

ipmitool user list 2

 

[richard.dzavoronok]

However:
- Fans appear to be stuck at minimum Speed (CPU cooking at 81-82°C during a Linux Kernel Build which I use as a stress test with make -j 16) and there does NOT seem to be a Fan Setting (not in the iRMC at least)
- I cannot seem to find the way to get remote Console (KVM) working - Where is that ?

My 1240 v6 is hovering around 80 under high usage so no worries. There is setting in the BIOS for faster CPU fan speed.
Keep in mind that this will just set minimum fan speed to 50% (iirc).
You have to acquire a valid iRMC license to enable remote console.