Fujitsu TX1320 M3 - Cheap low power server (barebone)

[3333]

Has anyone ever transplanted one of those to a SM chassis?

I run one of these as TNC box (very nice low power) but need some more drives so was considering transplanting to a 216 or 836/846.
2x8 and one x4 slot are perfect for a HBA, 10 (40G) card and single nvme

In the end I assume Id only use the board and the psu (and maybe a fan so it won't complain), right? I dont mind If i cant use screws, its not as if I move those boxes all day long...

I was thinking about the same thing some time ago and I was doing the research. I haven't done it, but I've seen several posts stating that the PSU can be replaced with anything else, but you'll need and adapter, as Fujitsu has different pinout and longer or shorter plug. But what you also need is this little front panel thing with power button as the server won't boot up if it's not detected.

 

[yoggi]

@yoggi, I'm pretty sure my machine is acting similarly, when it's started after the power break. However, when the server was powered off, but still plugged in, then it's not rebooting during startup and doesn't have the 2 minutes lag.

@3333, It is like the machine is waiting at the boot screen, I have looked through the bios settings many times but I can't find anything that affects the waiting. The machine is set to boot from disk, silent booting etc.

Do anybody have any idea why my machine is waiting at boot when there is no error?

yoggi

 

[yoggi]

I was thinking about the same thing some time ago and I was doing the research. I haven't done it, but I've seen several posts stating that the PSU can be replaced with anything else, but you'll need and adapter, as Fujitsu has different pinout and longer or shorter plug. But what you also need is this little front panel thing with power button as the server won't boot up if it's not detected.

@3333 and @Rand__
This page might be helpful inorder to determine what needs to be transfered with the motherboard when moving to a new chassi

Fujitsu TX140 S2 | «WatchMySys» Blog

From same website, good information about the TX1320 M3
Fujitsu TX1320 M3 | «WatchMySys» Blog

 

[luckylinux]

I still find it hard to believe you can get such low Power Consumption Numbers around 10-15 W at Idle .

I think my Supermicro X10SLL-F E3-1230 v3 is around 40-50 W and the Supermicro X11SSL-F E3-1230 v5 is probably around the same. This with 4 sticks of RAM (maxed out at 32GB DDR3 ECC UDIMM and 64GB DDR4 ECC UDIMM respectively) and 2-6 x Crucial MX500 SSDs, 4 x Arctic F8 PWM Fans, CPU Cooler NH-L12s with Fan PWN controlled and a Seasonic SSR-360GP 360W PSU (80+ Gold).

Granted, some of them have a LSI 9211-8i HBA in IT mode passed through to a VM and/or some Mellanox ConnectX-2/3 10gbps NIC.

So indeed there might be something going totally wrong with ASPM. And a 80+ Gold PSU isn't very efficient at all at around 10-15 % load ... And yes, Supermicro BMC is probably adding like 10W of Power Consumption just by itself .

I plan on replacing the Mellanox ConnectX-2/3 with either an Intel X710 or at least a Mellanox ConnectX-4, both of which (the Intel much better than the Mellanox) support ASPM low power states.

But considering how some Members achieved around 20W-25W with an Intel X540 (which does NOT support ASPM) I'm a bit puzzled.

 

[chrisgassen]

To contribute to this amazing thread: I designed some fan shrouds for my TX1320 drive bays which are available here:

Thingiverse - The community for Open Hardware

Thingiverse - The community for Open Hardware

www.thingiverse.com

https://www.thingiverse.com/thing:6572448
(additional room for cables, with the downside of having to assemble the cpu heatsink afterwards...)

 

[Black_Pyramid]

Hi,

how can I reset the iRMC S4 to factory defaults? I could not find anything in the web interface.

 

[IvanBayan]

If someone need, there is a holder for 2.5" disks in place of BBU holder.
Designed for tx1320 m3, though not sure if it fit to server with double hotplug PSU.
Looks like C26361-K644-C970 used not only in TX1320 M3, so may fit other FSC servers as well.

Fujitsu BBU 2.5" HDD/SSD holder by IvanBayan

It's a holder for 2.5" HDD or SSD drive in place of Fujitsu BBU Holder C26361-K644-C970Tested with TX1320 M3

www.thingiverse.com

 

[Black_Pyramid]

29 € or 61 € IMHO does not really matter. The ECC RAM for this barebone is much more expensive than this price difference. Also you have to print the caddies or the price will multiply.

Do those HP PN "371589-001" fit? Was it announced here in the thread? EDIT: Yes, I've found it.

 

[luckylinux]

I might chime in here, as I happen to have played a litte with one of these really nice little machines, too.

Happy to share some takeaways, too:

• power consumption: 9W idle is the lowest I managed to get (proxmox/debian + powertop --auto-tune, just one SATA ssd). I doubt you can get any lower given that the iRMC draws about 4.5 W alone (if someone would find a way to switch it off, though...)
• CPUs: tested Pentium G4560, Xeon 1220 v6, and Pentium G4600 (which is not on the official compatibility list, marginally faster than the G4560, works fine)
• caddys: the previously mentioned STL file on thingiverse works sort of (3d-printed a few myself), at least for blind caddies. BUT: the screw holes are "on the wrong side" and not within the right distance from the backplane to let you connect hdds/ssds (at least for my TX1320 M3). Also there is a slightly thicker part on one of the side pieces, that does not really seem to go too well with the the TX1320 once you have screwed in a ssd or hdd. So I have been fiddling for a while with tinkercad (I'm no engineer!) to "fix" this in the STL file and ended up with something that's actually working well enough for my needs. Did not bother to upload the new file on thingiverse yet as the result is certainly far from perfect, but happy to share it here (check it out for yourself and only use at your own risk, of course!). Printing in PLA and PETG both works fine, could not test ABS (would be my preferred material due to highest ruggedness). Stiffness etc. is good enough in either material. Temperature should be no issue as you probably do not plan to let temps go up higher than 50 Celsius among the drives, I presume.
• orange blinking led: whenever you change CPU or RAM that's detected (hence the blinking led); to turn that off just go into BIOS (it will report the hardware change very briefly on screen, if I remember correctly), maybe saving (unchanged) BIOS settings is required too, to get rid of the blinking led (can't remember). After a new start the blinking led should be gone.
• 3.5" hdd: it's quite possible to fit in one in the big bay near the power supply. I had to very slightly bend the pretty stiff sheet metal guiding rail, though. And there are even two nicely fitting holes on the outer side of the cage to fix the drive with standard screws, which I found sufficient. No vibrational decoupling from the rest of the case, however.

It would be absolutely great to find some way to mount a second 3.5" hdd for a zfs mirror. But that seems almost impossible given the cramped space (without removing the 2.5 bays, of course).

Also, should someone find a simple way to elegantly improvise a 2x 3.5" cage for the space the 2.5" bays occupy (without having to purchase the OEM parts), I might consider getting a second machine. ;-)

Very Late Reply and not sure if you found a Solution in the meantime.

I don't currently own this PC but I'm tempted to get a few of them to use as OPNSense Router .

You mentioned about disabling the iRMC to lower Power Consumption.

There might be a Way.

Taking the latest BIOS I could see that there is a ServerMgmtSetup Section, besides the usual Setup Section.

Veryy crude Patching might be done like this (untested of course):

setup_var.efi ServerSetup(0x2):0x0=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x1=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x452=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46F=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x459=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45E=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45A=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45D=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x463=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45C=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x458=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x45F=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x460=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x462=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x464=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x465=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x466=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x468=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB3=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB5=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB4=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB6=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB7=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB9=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xBA=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xB2=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0xBB=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x454=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x455=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x456=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x457=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x469=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46A=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46B=<To_Be_Manually_Filled_In>
setup_var.efi ServerSetup(0x2):0x46C=<To_Be_Manually_Filled_In>
setup_var.efi SystemAccess(0x1):0x0=<To_Be_Manually_Filled_In>
setup_var.efi fjOnboardVideoGrayout(0x3):0x0=<To_Be_Manually_Filled_In>

I just quickly did a survey scan on V5.0.0.11_R1.39.0_2024-08-19 on the D3373-B1.UPD File. Not sure which one is the "Best" or "Correct" one.

If I understand right, when you have

SuppressIf
                        True

Or

DisableIf
                        True

Then that Option is hidden from the BIOS.

The Options could be searched e.g. via "Enable" or "Power" etc.

EDIT 1: Option Power Saving Mode", Help: "[Enabled] iRMC will be powered down, when system is shut off. [Disabled] iRMC will go into stand-by mode, when system is shut off. This allows communication with iRMC even when the system is powered off. could be useful if you turn OFF the system and don't need iRMC to be able to turn it back ON.

Whereas OneOf Prompt: "BMC Support", Help: "Enable/Disable interfaces to communicate with BMC", QuestionFlags: 0x10, QuestionId: 0x1, VarStoreId: 0x2, VarOffset: 0x0, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x1, Step: 0x0 looks like the main "Switch", but I guess it only disables the Network Part of it, not the whole BMC ....

 

[Black_Pyramid]

Well, yes, you are right. In May I've paid 245 € for 8 pieces of 16 GB Samsung ECC unbuffered 2400T. That's less than I've thought. I've thought the price was for 4 pieces only

But I'm pretty sure you won't get the barebone for 29 € again. The 61 € is the cheapest I have seen and I've bought three of them from alles4pc! on Ebay, too.

 

[luckylinux]

Has anybody found good Deals ? The original Price of 29 EUR that @Rand__ originally listed 2 Years ago is now at the very least 61 EUR (plus Shipping) and no Option to place an Offer. I can always try to contact the Seller (alles4pc! on Ebay) and see if they agree to a lower Price if I buy 2-4 of these Barebones but there is no guarantee that they will . No Competition unfortunately .

Plus the price of the Caddies of course ...

I assume that @Bjorn Smith you don't need any more Caddies, do you ? I also live in Denmark by the Way .

I thought to go with the readily available HP PN "371589-001" otherwise, although those are around 4-6 EUR / Piece plus Shipping ...

Thanks for your Help

 

[Black_Pyramid]

Does anybody run a SAS card to use all 8 disk slots?

I wonder if the tiny airflow of the fans running at low speed is enough to cool it down. Obvious choices would be:

• SAS3008 based card (9300-8i): Has the SFF-8643 connector, so original cables can be used. But it draws huge 13 W and only has a small heat sink. Old and cheap (40 €).
• SAS3408 based card (9400-8i): SFF-8643 connector, draws 10 W and has a bigger heat sink, 120 €.
• SAS3808 based card (9500-8i): 7 W, 150 €. But new cables needed (50 €). New cables also means new fan brackets, that have been postet above.

Any other options?
What's your experience?

 

[Black_Pyramid]

Concerning Power Losses: the Power Loss of the HBA itself is kind of Peanuts a) Compared to 8 x SSDs / HDDs that are active

Well, most of the time everything is just idling in my home server. The power consumption of my TX1320-M3 with 64 GB RAM, 2 NVMe operating system disks and 6 SATA SSDs currently is about 20 W. The 6 SATA SSDs may have increased power usage by 6 W, sadly I don't have exact numbers by hand.

b) Secondary to the additional Power Losses due to the System NOT being able to "sleep" with deep PACKAGE C-States (notably PC6 and better/higher such as e.g. PC8 on newer Generations). A big part of it is due to ASPM. See Z8 Posts here and here about ASPM, as well as my Repository for some Ideas (nothing "Final", I'm also learning along the Way).

I know these problems.ASPM is activated in BIOS but there is already a device, that does not support it: the Matrox graphics card. As I'm running FreeBSD I don't have exact numbers because the lack of the nice powertop, but at least C3 states are reached by the CPU:

# sysctl dev.cpu.0 | grep cx
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 20376 4363 58133
dev.cpu.0.cx_usage: 24.58% 5.26% 70.14% last 21913us
dev.cpu.0.cx_lowest: C8
dev.cpu.0.cx_supported: C1/1/1 C2/2/151 C3/3/256

Also note this from your link:

Here’s where things get really interesting. The official LSI users guide for the 92xx series HBAs explicitly lists ASPM as a feature, but ASPM is set to disabled. Now why would that be? Turns out that back in 2013 someone filed a bug report stating that their system would constantly lock up during high read workloads. The solution for this turned out to be to simply disable ASPM. For any of the MPI v2.0 chipsets listed in mpi2_cnfg.h ASPM will explicitly be disabled.

As I'm running FreeBSD this does not apply. I guess this really would be try and measure.

Still my initial question was more about cooling of these cards. I doubt that a 13 W device with small heat sink is cooled down enough.

 

[luckylinux]

29 € or 61 € IMHO does not really matter. The ECC RAM for this barebone is much more expensive than this price difference. Also you have to print the caddies or the price will multiply.

Do those HP PN "371589-001" fit? Was it announced here in the thread? EDIT: Yes, I've found it.

Well if you order 4-5 Servers I'd say it matters . Less is More .

Yeah, RAM is more expensive for sure. Best I could find for ECC Unbuffered was 34 EUR / piece for PC4-2400T. So 4 x 16GB = 136 EUR.

Not too bad I'd say considering I was paying 55 EUR for new a few Months ago ...

 

[richard.dzavoronok]

However:
- Fans appear to be stuck at minimum Speed (CPU cooking at 81-82°C during a Linux Kernel Build which I use as a stress test with make -j 16) and there does NOT seem to be a Fan Setting (not in the iRMC at least)
- I cannot seem to find the way to get remote Console (KVM) working - Where is that ?

My 1240 v6 is hovering around 80 under high usage so no worries. There is setting in the BIOS for faster CPU fan speed.
Keep in mind that this will just set minimum fan speed to 50% (iirc).
You have to acquire a valid iRMC license to enable remote console.

 

[richard.dzavoronok]

@richard.dzavoronok : do you know how to reset the Error LED Light ?

It's not really an issue and I don't know what's causing it. I just changed the CMOS Battery because the Voltage was extremely low.

Other than that it seems the BIOS Component has an Error Message, not sure why though ...

I had a lot of errors/reboots when I changed the CPU, maybe also yours need a couple of reboots to reset LEDs ?
I'm just shooting in the dark here. I didn't have such an error sorry :/

 

[Rand__]

@richard.dzavoronok : do you know how to reset the Error LED Light ?

View attachment 41409

It's not really an issue and I don't know what's causing it. I just changed the CMOS Battery because the Voltage was extremely low.

Other than that it seems the BIOS Component has an Error Message, not sure why though ...
View attachment 41408

I have the same, latest Bios, reboots didnt help.

I ignore it

 

[luckylinux]

Well, yes, you are right. In May I've paid 245 € for 8 pieces of 16 GB Samsung ECC unbuffered 2400T. That's less than I've thought. I've thought the price was for 4 pieces only

But I'm pretty sure you won't get the barebone for 29 € again. The 61 € is the cheapest I have seen and I've bought three of them from alles4pc! on Ebay, too.

I got a very slight Discount and bought 6 Servers . I might have exaggerated

Might see if I got enough RAM otherwise I'll "just" install 32GB per Server and upgrade later on if I find a better deal.

Although to be fair SSDs are also crazy expensive (and I wanted to buy some Intel S3610 but eBay denied Purchase from the US)

 

[richard.dzavoronok]

EDIT 1: worse still, after BIOS Upgrade, I'm now stuck at the "Fujitsu" Initial Screen (NOT the AMI BIOS + Fujitsu Screen which lists Memory etc, I mean the INITIAL Fujitsu Screen which basically only lists the Serial Number and iRMC IP Address if any). iRMC doesn't even list an IP Address

Do you have dedicated GPU installed? If so, the VGA port on the mb will be stuck at the fujitsu logo. The gpu's one should be working properly.
Manual states this: " Remote Video direction via iRMC must be disabled. " in the section with Graphics card. Maybe try that?

 

[luckylinux]

Does anybody run a SAS card to use all 8 disk slots?

I wonder if the tiny airflow of the fans running at low speed is enough to cool it down. Obvious choices would be:

• SAS3008 based card (9300-8i): Has the SFF-8643 connector, so original cables can be used. But it draws huge 13 W and only has a small heat sink. Old and cheap (40 €).
• SAS3408 based card (9400-8i): SFF-8643 connector, draws 10 W and has a bigger heat sink, 120 €.
• SAS3808 based card (9500-8i): 7 W, 150 €. But new cables needed (50 €). New cables also means new fan brackets, that have been postet above.

Any other options?
What's your experience?

I don't have the Server yet ... Just some Observations based on similar (?) Experiences.

Concerning Airflow/Temperatures: I have some Servers in a Silverstone ML03B. Some Servers had 3 PCIe Cards (1 x Mellanox ConnectX-2/3 10gbps NIC, 1 x LSI 9211-8 HBA, 1 x Intel HP NC364T) "Sandwiched" with no Room to breathe. The Mellanox was likely to hit 110-125 °C (!). In one Server, when 2 out of 4 Fans stopped working (and Monitoring apparently did NOT work), the HP NC364T NIC started throwing out Errors on a regular basis. It's been used for a long time and it's an old NIC, will be replaced by an i350-T4. It's not an Issue for this Part. But think about it if you have something more "precious" nowadays in that "Sandwitch".

Concerning Power Losses: the Power Loss of the HBA itself is kind of Peanuts a) Compared to 8 x SSDs / HDDs that are active b) Secondary to the additional Power Losses due to the System NOT being able to "sleep" with deep PACKAGE C-States (notably PC6 and better/higher such as e.g. PC8 on newer Generations). A big part of it is due to ASPM. See Z8 Posts here and here about ASPM, as well as my Repository for some Ideas (nothing "Final", I'm also learning along the Way).

 

[Rand__]

I ran 1245 v6's on multiple boxes... Had issues with PCIe (bifurcation, nvme compatibility, see early posts here) but never attributed that to the 5...
All in all the 5 have been running fine for years, so wouldn't say they're problematic per se

 

[luckylinux]

By the way, does somebody have Information about how these PCIe Slots are connected/routed ?

I tried going through the User Manual, the Operation Manual, the Upgrade Manual as well as the BIOS Settings Manual but I couldn't find anywhere this Information (not even in the disassembled BIOS ). There is no clear Block Diagram like it is on the Supermicro Motherboard Manuals for Instance. Fujitsu has a LOT of Documentation, but not that (unless I missed it). I guess it shold be possible to determine via lspci -tv and then check if the PCIe Slot is connected to a Root Port (in that case it's in a CPU-connected PCIe Slot) or if it's "Parent" is the Chipset C236.

I ask because NICs should as the Mellanox ConnectX-4 LX work with ASPM only (thus far for me at least) if they are connected in a PCH-connected (Chipset/PCH/DMI) PCIe Slot. **NOT** in a CPU-connected PCIe Slot (cannot remember if like that the best I achieved was PC3 or even PC2, anyways it was not super good) !

 

[Stueckchen]

Hey there,
did anyone found out how to Control the FAN?
I would like to cool an Tesla P4 with an FAN. On Fan Header SYS3 its to low on SYS4 its to load?

 

[cowmix]

Has anyone tried a dGPU? Or tried methods of forcing L1 aspm? I was thinking of getting one of these and using like an intel arc for transcoding since you can't use a CPU with an iGPU but the notes about not having an L1 ASPM concerns me that the gpu would not be able to fully idle with this motherboard when not in use?

 

[luckylinux]

Well, most of the time everything is just idling in my home server. The power consumption of my TX1320-M3 with 64 GB RAM, 2 NVMe operating system disks and 6 SATA SSDs currently is about 20 W. The 6 SATA SSDs may have increased power usage by 6 W, sadly I don't have exact numbers by hand.


I know these problems.ASPM is activated in BIOS but there is already a device, that does not support it: the Matrox graphics card. As I'm running FreeBSD I don't have exact numbers because the lack of the nice powertop, but at least C3 states are reached by the CPU

20W at "only" PC3 and with 6 SSDs + 2xNVME and only 1W/SSD at Idle ? That's VERY good actually. Do you trust your "Kill-A-Watt" Measurement Device though ? Or is it oscillating a lot / "unstable" ?

Are you sure that is the "Package" C-State though ?

I don't have much experience with FreeBSD, just a bit a long Time ago and some headaches trying to build some Ports/Packages on OPNSense .

Lucky you that it works though. It MIGHT also work on Linux but it was disabled for a Reason, not sure it's worth the Risk of compiling a Kernel from Source with that Patch/Quirk removed.

Still my initial question was more about cooling of these cards. I doubt that a 13 W device with small heat sink is cooled down enough.

Well given that there is almost no Airflow in the Chassis with significant Static Pressure I'd tend to agree. My Issue was on the Silverstone ML03 with 4 Fans blowing Air into the Chassis. With 2 Fans failed, the HBA/NIC were cooking essentially.

Put some 40mm or possibly 60mm Noctua on it or something .

On the cheap there are plenty of 40mm Fans also on Digikey that you can buy also from relatively Reputable Brands (let alone Aliexpress, Amazon, ... although I'd say they can be more Expensive). I'll have to investigate whether it's possible to replace those proprietary 70mm Fans with some standard 80mm ones. I was thinking of using some 3D Printed Adapter for 70mm -> 80mm Fans that can be found on Thingiverse (e.g. this) or similar Sites. Since I don't have a 3D Printer, I'll have to order online I guess. Not sure if there are other Locations where it's possible to add a Chassis Fan though.

Not sure why Sunon requires some additional Protection though (fuse), that seems stupid. Unless this is because it's meant for System Integrators and NOT targeted to Motherboards as an End-Product. Indeed you'll need to crimp these and make a Connector (not a huge deal given it's 2x Cheaper than the others).

 

[celemine1gig]

Are you actually sure about that ? And where do you see it can Bifurcated ?

Don't misunderstand me, or interpret the answer as arrogant, but yes, I am sure. Edit: Scratch that. I just realized, that I wrote port 3 (as I counted from the top), while in the Fujitsu documentation this is actually port "2". So, I am talking about the 1x4 port.
(About said 1x4 port):
It is a PCH port, which can be bifurcated through low-level settings in the Intel Flash Descriptor (first 4K of SPI storage space in the Boot SPI Flash). Also the Intel corresponding Intel "fitc" tool lets you modify it via a GUI.
You can then reconfigure the port to no longer be a 1x4 port, but rather a 2x2 (or maybe even 4x1; don't remember) port. And for a dual function Optane, like the H10 or H20, you will need a split port with 2x2.
As I said, as proof I had the mentioned Intel Optane running in there, with both of its functions active and working.
Edit: The two 1x8 ports are actually the already bifurcated PEG-slot. So, the 1x 16 PEG lanes, split into 2x8. Theoretically, one of those ports (which one, depends on the question of enabling lane reversal or not) could be further split into 2x4, as the PEG supports 1x8 and 2x4. This would need to be done via CPU Straps. See here: S/W strap for x16 or x8/x8 on PEG

Usually the Color is an Indicator and the first 3 PCIe Slots (Slot 4, Slot 3 and Slot 2) are Blue, while the latter one (Slot 1) is Black.
View attachment 43077

View attachment 43078

Sorry, but do you really think, that differentiation by color coding is a safe way of telling the connection type, when you can actually analyze the function in an OS or the EFI-shell? A simple "lspci -t" in Linux will tell you where a device connection actually goes, port-wise. Very helpful stuff.

I'm NOT sure if I ever tested to put the Mellanox ConnectX-4 LX in Slot3, but USUALLY ASPM works if it's a PCH/DMI/Chipset connected Slot, whereas USUALLY ASPM does NOT work if it's a direct CPU-connected Slot.

I'm afraid that PCIe Slot3 is also directly CPU Connected and NOT PCH/DMI/Chipset Connected .

"Usually" seems to be one of your favorite words. I would say that "usually" no recent Connect-X card with a recent firmware version has enabled ASPM, unless you somehow force it.
If you are looking for ASPM and 10G at the same time, then Intel X710 is a nice choice.

Has anyone tried a dGPU? Or tried methods of forcing L1 aspm? I was thinking of getting one of these and using like an intel arc for transcoding since you can't use a CPU with an iGPU but the notes about not having an L1 ASPM concerns me that the gpu would not be able to fully idle with this motherboard when not in use?

There is no real reason, why a dGPU should not work. It's just that, due to the onboard BMC, integrated CPU graphics was fully disabled.

 

[luckylinux]

I just received my first Fujitsu TX1320 M3 Server.

Installed 64GB of DDR4 Unbuffered ECC Memory, an Intel Xeon E3-1270 v6, booted a Debian LiveUSB as I usually do.

Change Password for User sp_adm:

ipmitool user set password 2

Set IP Address Source to DHCP:

ipmitool lan set 2 ipsrc dhcp

Then reboot the IRMC:

ipmitool bmc reset cold

So I can then access the WebUI and get the following Information:

[TABLE]
[TR]
[TH]BIOS Version:[/TH]
[TD]V5.0.0.11 R1.18.0 for D3373-B1x[/TD]
[/TR]
[TR]
[TH][/TH]

[TD][/TD]
[/TR]
[/TABLE]
[TABLE]
[TR]
[TH]Firmware Version:[/TH]
[TD]9.08F (Base: 9.08.F)[/TD]
[/TR]
[TR]
[TH]Firmware Date:[/TH]
[TD]Mar 5 2018 - 14:02:35 CEST[/TD]
[/TR]
[TR]
[TH]Firmware Running:[/TH]
[TD]Low Firmware Image[/TD]
[/TR]
[TR]
[TH]SDRR Version:[/TH]
[TD]3.16   ID 0458  TX1320M3[/TD]
[/TR]
[/TABLE]

BIOS seems to be latest, iRMC definitively NOT (from Fujitsu Technical Support pages from Fujitsu United Kingdom latest seems to be either D3373-B1_09.69F_sdr03.18 (26/08/2022))

EDIT 1: actually the BIOS seems a bit old. I was mislead/confused since it's already V5.0.0.11. But the sub-version is only R1.18.0, while R1.39.0 is latest

However:
- Fans appear to be stuck at minimum Speed (CPU cooking at 81-82°C during a Linux Kernel Build which I use as a stress test with make -j 16) and there does NOT seem to be a Fan Setting (not in the iRMC at least)
- I cannot seem to find the way to get remote Console (KVM) working - Where is that ?

EDIT 2: To list the Users on the System, just issue (apparently it's on Channel Number 2):

ipmitool user list 2

 

[celemine1gig]

Thank you for your Reply . I doubted it could be Bifurcated since there was no BIOS Option for it. And not quite sure what the link you pointed to shows besides being required to

It can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following website:

How exactly did you enable Bifurcation then ?

Simple and short answer: I am an engineer with the privilige to access all those tools. Fun fact: Even worked for Fujitsu for a short time.
Hint: You can get the tools online, although it is not legally OK. Buzz-words: "CSME System Tools".
Edit: That only applies to the 1x4 slot. For the 2 x8 PEG slots, you need to connect CPU LGA lands, in order to change the bifurcation.

Well, I expected them to have some "Logic" behind it. But I guess it's because the black PCIe Slot is actually only x1 Electrical or ?

It would have been better if they just did a Block Diagram in their Documentation like every other Motherboard Manufacturers on the Planet (e.g. Supermicro or even ASUS/ASRock Rack).

Of course you can say that Fujitsu didn't sell this as a Motherboard but rather a configurable complete System, but still ...

Same here. I also did not know excatly what they did, partly due to the seemingly color coded slots, until I tested in detail.

Uhm, lspci -tv (so you also get the Description in Text as well) works, but it does NOT tell you where the PCIe Slot goes (e.g. the Logical Mapping to e.g. PCIe Root Port XX will probably be different than the Silkscreen on the PCB or the "Slot XX" in the Manual), wouldn't it ?

So how exactly does that Help you ? Trial & Error and keep moving the Card you have plugged in the Slot until you achieve the desired Output in lspci -tv, so you can reverse engineer their Layout ?

I don't need to reverse engineer. If you look closely at the lspci decoded output, you will see that the PEG and the PCH can be differentiated by the naming, as well as the PEG being listed with a lower bus/device/function number.
So, if you have the "tree" output and see where your devices are connected, then you can deduct, which one is a PEG connection and which one is a PCH-connection.

Well, I haven't been proven wrong on that and also other People experienced the same. It's maybe NOT conclusive (hence the "Usual" Word) and I didn't test 300'000 different Motherboards Makes & Model, but on the Supermicro X10SLL* and X11SS* that I tested, turbostat / powertop would show PC6 Package C-State when plugged into a PCH/DMI/Chipset connected Slot.

That was Mellanox ConnectX-4 LX with latest Firmware from Mellanox .

Can it be a "Fake" / Buggy Reporting ? Possibly. Not sure if I checked the Kill-a-Watt Meter for those Test to see if the Power Consumption was close to Baseline.

But for sure with the Mellanox ConnectX-4 in a CPU connected PCIe Slot, it would NOT go below PC3.


It is a nice Choice, although more expensive and less available good Deals IMHO (especially in Europe) .

To be honest I never bothered with anything newer than ConnectX-3.
However, if you want cheap X710 cards, look for the HP 562SFP+. This can be found at VERY reasonable prices, throughout Europe.

 

[celemine1gig]

I don't have the system anymore, so I cannot give you a direct example for lspci.
However, if you post an example/examples for an "lspci" and also an additional "lspci -t -s xx:xx.x" for your desired device, I can explain what I meant.

And concerning the X710: If 80,-EUR is too expensive in your opinion, then of course, the 562SFP+ does not cut it. I obviously already considered this a good deal. Of course, this cannot compete with your 45,-EUR bargain example. Nice buy!