Fujitsu TX1320 M3 - Cheap low power server (barebone)

[luckylinux]

Don't misunderstand me, or interpret the answer as arrogant, but yes, I am sure. Edit: Scratch that. I just realized, that I wrote port 3 (as I counted from the top), while in the Fujitsu documentation this is actually port "2". So, I am talking about the 1x4 port.
(About said 1x4 port):
It is a PCH port, which can be bifurcated through low-level settings in the Intel Flash Descriptor (first 4K of SPI storage space in the Boot SPI Flash). Also the Intel corresponding Intel "fitc" tool lets you modify it via a GUI.
You can then reconfigure the port to no longer be a 1x4 port, but rather a 2x2 (or maybe even 4x1; don't remember) port. And for a dual function Optane, like the H10 or H20, you will need a split port with 2x2.
As I said, as proof I had the mentioned Intel Optane running in there, with both of its functions active and working.
Edit: The two 1x8 ports are actually the already bifurcated PEG-slot. So, the 1x 16 PEG lanes, split into 2x8. Theoretically, one of those ports (which one, depends on the question of enabling lane reversal or not) could be further split into 2x4, as the PEG supports 1x8 and 2x4. This would need to be done via CPU Straps. See here: S/W strap for x16 or x8/x8 on PEG

Thank you for your Reply . I doubted it could be Bifurcated since there was no BIOS Option for it. And not quite sure what the link you pointed to shows besides being required to

It can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following website:

How exactly did you enable Bifurcation then ?

Sorry, but do you really think, that differentiation by color coding is a safe way of telling the connection type, when you can actually analyze the function in an OS or the EFI-shell?

Well, I expected them to have some "Logic" behind it. But I guess it's because the black PCIe Slot is actually only x1 Electrical or ?

It would have been better if they just did a Block Diagram in their Documentation like every other Motherboard Manufacturers on the Planet (e.g. Supermicro or even ASUS/ASRock Rack).

Of course you can say that Fujitsu didn't sell this as a Motherboard but rather a configurable complete System, but still ...

A simple "lspci -t" in Linux will tell you where a device connection actually goes, port-wise. Very helpful stuff.

Uhm, lspci -tv (so you also get the Description in Text as well) works, but it does NOT tell you where the PCIe Slot goes (e.g. the Logical Mapping to e.g. PCIe Root Port XX will probably be different than the Silkscreen on the PCB or the "Slot XX" in the Manual), wouldn't it ?

So how exactly does that Help you ? Trial & Error and keep moving the Card you have plugged in the Slot until you achieve the desired Output in lspci -tv, so you can reverse engineer their Layout ?

"Usually" seems to be one of your favorite words. I would say that "usually" no recent Connect-X card with a recent firmware version has enabled ASPM, unless you somehow force it.

Well, I haven't been proven wrong on that and also other People experienced the same. It's maybe NOT conclusive (hence the "Usual" Word) and I didn't test 300'000 different Motherboards Makes & Model, but on the Supermicro X10SLL* and X11SS* that I tested, turbostat / powertop would show PC6 Package C-State when plugged into a PCH/DMI/Chipset connected Slot.

That was Mellanox ConnectX-4 LX with latest Firmware from Mellanox .

Can it be a "Fake" / Buggy Reporting ? Possibly. Not sure if I checked the Kill-a-Watt Meter for those Test to see if the Power Consumption was close to Baseline.

But for sure with the Mellanox ConnectX-4 in a CPU connected PCIe Slot, it would NOT go below PC3.

If you are looking for ASPM and 10G at the same time, then Intel X710 is a nice choice.

It is a nice Choice, although more expensive and less available good Deals IMHO (especially in Europe) .

 

[celemine1gig]

Thank you for your Reply . I doubted it could be Bifurcated since there was no BIOS Option for it. And not quite sure what the link you pointed to shows besides being required to

It can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following website:

How exactly did you enable Bifurcation then ?

Simple and short answer: I am an engineer with the privilige to access all those tools. Fun fact: Even worked for Fujitsu for a short time.
Hint: You can get the tools online, although it is not legally OK. Buzz-words: "CSME System Tools".
Edit: That only applies to the 1x4 slot. For the 2 x8 PEG slots, you need to connect CPU LGA lands, in order to change the bifurcation.

Well, I expected them to have some "Logic" behind it. But I guess it's because the black PCIe Slot is actually only x1 Electrical or ?

It would have been better if they just did a Block Diagram in their Documentation like every other Motherboard Manufacturers on the Planet (e.g. Supermicro or even ASUS/ASRock Rack).

Of course you can say that Fujitsu didn't sell this as a Motherboard but rather a configurable complete System, but still ...

Same here. I also did not know excatly what they did, partly due to the seemingly color coded slots, until I tested in detail.

Uhm, lspci -tv (so you also get the Description in Text as well) works, but it does NOT tell you where the PCIe Slot goes (e.g. the Logical Mapping to e.g. PCIe Root Port XX will probably be different than the Silkscreen on the PCB or the "Slot XX" in the Manual), wouldn't it ?

So how exactly does that Help you ? Trial & Error and keep moving the Card you have plugged in the Slot until you achieve the desired Output in lspci -tv, so you can reverse engineer their Layout ?

I don't need to reverse engineer. If you look closely at the lspci decoded output, you will see that the PEG and the PCH can be differentiated by the naming, as well as the PEG being listed with a lower bus/device/function number.
So, if you have the "tree" output and see where your devices are connected, then you can deduct, which one is a PEG connection and which one is a PCH-connection.

Well, I haven't been proven wrong on that and also other People experienced the same. It's maybe NOT conclusive (hence the "Usual" Word) and I didn't test 300'000 different Motherboards Makes & Model, but on the Supermicro X10SLL* and X11SS* that I tested, turbostat / powertop would show PC6 Package C-State when plugged into a PCH/DMI/Chipset connected Slot.

That was Mellanox ConnectX-4 LX with latest Firmware from Mellanox .

Can it be a "Fake" / Buggy Reporting ? Possibly. Not sure if I checked the Kill-a-Watt Meter for those Test to see if the Power Consumption was close to Baseline.

But for sure with the Mellanox ConnectX-4 in a CPU connected PCIe Slot, it would NOT go below PC3.


It is a nice Choice, although more expensive and less available good Deals IMHO (especially in Europe) .

To be honest I never bothered with anything newer than ConnectX-3.
However, if you want cheap X710 cards, look for the HP 562SFP+. This can be found at VERY reasonable prices, throughout Europe.

 

[luckylinux]

Simple and short answer: I am an engineer with the privilige to access all those tools. Fun fact: Even worked for Fujitsu for a short time.
Hint: You can get the tools online, although it is not legally OK. Buzz-words: "ME System Tools".
Edit: That only applies to the 1x4 slot. For the 2 x8 PEG slots, you need to connect CPU LGA lands, in order to change the bifurcation.

Aaah alright .

I don't need to reverse engineer. If you look closely at the lspci decoded output, you will see that the PEG and the PCH can be differentiated by the naming, as well as the PEG being listed with a lower bus/device/function number.

So, if you have the "tree" output and see where your devices are connected, then you can deduct, which one is a PEG connection and which one is a PCH-connection.

Because if in PCH slot it would have an extra Level / Layer in the tree view, instead of "directly" attached to the PCIe Root Port ?

However, if you want cheap X710 cards, look for the HP 562SFP+. This can be found at VERY reasonable prices, throughout Europe.

It doesn't seem *that* cheap at all unless it's a really bad Period. 80+ EUR from Germany ? That's very close to the "Normal" X710 at 100EUR.

Hp 562sfp online kaufen | eBay

Tolle Angebote bei eBay für Hp 562sfp. Sicher einkaufen.

www.ebay.de

A Week ago or so I managed to order 2 x Intel XXV710-DA2 from the US for 45 EUR each (including import Charges / VAT / Shipping / etc).

That's why the X710 (or HP 562SFP+) does NOT, at least right now, seem so interesting in Terms of Price within Europe, unless imported (US or possibly UK, although UK I didn't see any good deal on the X710, only Mellanox ConnectX-4 LX Deals from UK)..

 

[celemine1gig]

I don't have the system anymore, so I cannot give you a direct example for lspci.
However, if you post an example/examples for an "lspci" and also an additional "lspci -t -s xx:xx.x" for your desired device, I can explain what I meant.

And concerning the X710: If 80,-EUR is too expensive in your opinion, then of course, the 562SFP+ does not cut it. I obviously already considered this a good deal. Of course, this cannot compete with your 45,-EUR bargain example. Nice buy!

 

[luckylinux]

I don't have the system anymore, so I cannot give you a direct example for lspci.
However, if you post an example/examples for an "lspci" and also an additional "lspci -t -s xx:xx.x" for your desired device, I can explain what I meant.

Ah, OK, I guess you moved to something more Powerful. I'm also starting setting up some AMD Ryzen (mostly 2nd Hand Parts) Systems that have like 2-4 x more CPU Power .

NOT all PCIe Slots are populated though

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
           +-01.0-[01]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           +-14.0  Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
           +-14.2  Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem
           +-16.0  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1
           +-16.1  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2
           +-17.0  Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
           +-1c.0-[02]--+-00.0  Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)
           |            \-00.1  Emulex Corporation ServerView iRMC HTI
           +-1c.5-[03]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.6-[04]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1d.0-[05]--+-00.0  Intel Corporation I350 Gigabit Network Connection
           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
           +-1f.0  Intel Corporation C236 Chipset LPC/eSPI Controller
           +-1f.2  Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller
           \-1f.4  Intel Corporation 100 Series/C230 Series Chipset Family SMBus

Not exactly what I was expecting.

I guess everything 1X (like onboard NIC, Matrox VGA, etc) are on the Chipset and everything 0X is directly CPU connected. I would have expected an extra Tree Level / Branch for the Chipset though.

But 1c.# corresponds to the PCIe Root Port #, so that is at least an Exception to what I said above.

And concerning the X710: If 80,-EUR is too expensive in your opinion, then of course, the 562SFP+ does not cut it. I obviously already considered this a good deal. Of course, this cannot compete with your 45,-EUR bargain example. Nice buy!

Yeah from the US it's quite good at Times, if I can manage to pass the Order via eBay (I usually ALWAYS get denied, maybe it worked because I sent them an Offer instead of putting the Article in my Cart, when it ALWAYS get blocked at Checkout stating that it cannot be delivered to my Country) .

Otherwise I got 25-40 EUR / Piece Deal on the Mellanox ConnectX-4 LX from UK, which is why it would be good to use them, if I can get ASPM to work correctly .

 

[celemine1gig]

Ah, OK, I guess you moved to something more Powerful. I'm also starting setting up some AMD Ryzen (mostly 2nd Hand Parts) Systems that have like 2-4 x more CPU Power .

NOT all PCIe Slots are populated though

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
           +-01.0-[01]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           +-14.0  Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
           +-14.2  Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem
           +-16.0  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1
           +-16.1  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2
           +-17.0  Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
           +-1c.0-[02]--+-00.0  Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)
           |            \-00.1  Emulex Corporation ServerView iRMC HTI
           +-1c.5-[03]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.6-[04]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1d.0-[05]--+-00.0  Intel Corporation I350 Gigabit Network Connection
           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
           +-1f.0  Intel Corporation C236 Chipset LPC/eSPI Controller
           +-1f.2  Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller
           \-1f.4  Intel Corporation 100 Series/C230 Series Chipset Family SMBus

Not exactly what I was expecting.

I guess everything 1X (like onboard NIC, Matrox VGA, etc) are on the Chipset and everything 0X is directly CPU connected. I would have expected an extra Tree Level / Branch for the Chipset though.

But 1c.# corresponds to the PCIe Root Port #, so that is at least an Exception to what I said above.

It does look a bit weird. Maybe depending on the actual "lspci" version.
I would have expected this for the PEG ports (so both x8 ports):

00:01.0 PCI bridge: Intel Corporation 6th-9th Gen Core Processor PCIe Controller
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller

With the focus on two things:
1.) The PEG ports are listed with lower functions, than the PCH ports! So, "00:01.x"=PEG vs. "00:1x.x"=PCH.
2.) It actually says "Core Processor PCIe Controller" in the description text (if available), which is the PEG.
It is not THAT clear in your example, but you can still see that your Mellanox card is connected to "00:01.0", or in other words, PEG0.
As the PEG can be bifurcated into 3 parts (1x8 and 2x4) at max, you can also have PEG1 (00:01.1) and PEG2 (00:01.2).
The unused ports just don't get displayed.

And like you already realized, the PCH ports have 2 points to identify:
1.) The PCH ports are listed with higher functions, than the PEG ports (00:1x.x)!
2.) It actually says " C230 Series Chipset Family PCI Express Root Port" in the description text, which is a PCH port.
See here for details:

https://cdrdv2-public.intel.com/332691/100-series-chipset-datasheet-vol-2.pdf

Starting page 830, here is a quote for a better understanding:

12.1 PCI Express* Port Configuration Registers
Summary
There are twenty sets of the following configuration registers used for PCH PCI
Express* Port Configurations. Each PCH PCI Express* Configuration Register set covers
a single PCI Express* Port and maps out as the following Device/Function:
D28/F0 = Port1
D28/F1 = Port2
D28/F2 = Port3
D28/F3 = Port4
D28/F4 = Port5
D28/F5 = Port6
D28/F6 = Port7
D28/F7 = Port8
D29/F0 = Port9
D29/F1 = Port10
D29/F2 = Port11
D29/F3 = Port12
D29/F4 = Port13
D29/F5 = Port14
D29/F6 = Port15
D29/F7 = Port16
D27/F0 = Port17
D27/F1 = Port18
D27/F2 = Port19
D27/F3 = Port20

In your example lspci output it shows "1c.0", "1c.5", "1c.6" and finally "1d.0".
That means (with reference to the intel document above) you see:
- "1c.0" = 16 + 12, Function 0 = 28, Function 0 = Port 1 : Internal Matrox BMC graphics chip
- "1c.5" = 16 + 12, Function 5 = 28, Function 5 = Port 6 : On-board Intel i210 #1
- "1c.6" = 16 + 12, Function 6 = 28, Function 6 = Port 7 : On-board Intel i210 #2
- "1d.0" = 16 + 13, Function 0 = 29, Function 0 = Port 9 : Intel i350, as connected by you.

Yeah from the US it's quite good at Times, if I can manage to pass the Order via eBay (I usually ALWAYS get denied, maybe it worked because I sent them an Offer instead of putting the Article in my Cart, when it ALWAYS get blocked at Checkout stating that it cannot be delivered to my Country) .

Otherwise I got 25-40 EUR / Piece Deal on the Mellanox ConnectX-4 LX from UK, which is why it would be good to use them, if I can get ASPM to work correctly .

I am not sure, as I never owned a ConnectX-4, however from what I read (before getting the X710 in the end), the cards simply don't seem to support it. Why, no idea! But so far I did not find a single story, where someone used ConnectX-4 with active and working (!) ASPM.
The following reads somewhat ambiguous. At first it mentions working ASPM, but then says it doesn't do anything, concerning lower power consumption:

Connectx-4 Lx 2x 25GbE with ASPM support + idle power consumption measurements

I’ve recently got myself 2 Connectx-4 Lx cards. I wanted to replace very sketchy connectx-3 cards. (OCP2.0 cards on pcie adapters which negotiate only pcie3x2 D:) I’ve found decent cards on aliexpress for 62 EUR without taxes. It’s a steal at that price. Out of the box card didn’t support...

forum.level1techs.com

Weird stuff!