Fujitsu TX1320 M3 - Cheap low power server (barebone)

[luckylinux]

Don't misunderstand me, or interpret the answer as arrogant, but yes, I am sure. Edit: Scratch that. I just realized, that I wrote port 3 (as I counted from the top), while in the Fujitsu documentation this is actually port "2". So, I am talking about the 1x4 port.
(About said 1x4 port):
It is a PCH port, which can be bifurcated through low-level settings in the Intel Flash Descriptor (first 4K of SPI storage space in the Boot SPI Flash). Also the Intel corresponding Intel "fitc" tool lets you modify it via a GUI.
You can then reconfigure the port to no longer be a 1x4 port, but rather a 2x2 (or maybe even 4x1; don't remember) port. And for a dual function Optane, like the H10 or H20, you will need a split port with 2x2.
As I said, as proof I had the mentioned Intel Optane running in there, with both of its functions active and working.
Edit: The two 1x8 ports are actually the already bifurcated PEG-slot. So, the 1x 16 PEG lanes, split into 2x8. Theoretically, one of those ports (which one, depends on the question of enabling lane reversal or not) could be further split into 2x4, as the PEG supports 1x8 and 2x4. This would need to be done via CPU Straps. See here: S/W strap for x16 or x8/x8 on PEG

Thank you for your Reply . I doubted it could be Bifurcated since there was no BIOS Option for it. And not quite sure what the link you pointed to shows besides being required to

It can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following website:

How exactly did you enable Bifurcation then ?

Sorry, but do you really think, that differentiation by color coding is a safe way of telling the connection type, when you can actually analyze the function in an OS or the EFI-shell?

Well, I expected them to have some "Logic" behind it. But I guess it's because the black PCIe Slot is actually only x1 Electrical or ?

It would have been better if they just did a Block Diagram in their Documentation like every other Motherboard Manufacturers on the Planet (e.g. Supermicro or even ASUS/ASRock Rack).

Of course you can say that Fujitsu didn't sell this as a Motherboard but rather a configurable complete System, but still ...

A simple "lspci -t" in Linux will tell you where a device connection actually goes, port-wise. Very helpful stuff.

Uhm, lspci -tv (so you also get the Description in Text as well) works, but it does NOT tell you where the PCIe Slot goes (e.g. the Logical Mapping to e.g. PCIe Root Port XX will probably be different than the Silkscreen on the PCB or the "Slot XX" in the Manual), wouldn't it ?

So how exactly does that Help you ? Trial & Error and keep moving the Card you have plugged in the Slot until you achieve the desired Output in lspci -tv, so you can reverse engineer their Layout ?

"Usually" seems to be one of your favorite words. I would say that "usually" no recent Connect-X card with a recent firmware version has enabled ASPM, unless you somehow force it.

Well, I haven't been proven wrong on that and also other People experienced the same. It's maybe NOT conclusive (hence the "Usual" Word) and I didn't test 300'000 different Motherboards Makes & Model, but on the Supermicro X10SLL* and X11SS* that I tested, turbostat / powertop would show PC6 Package C-State when plugged into a PCH/DMI/Chipset connected Slot.

That was Mellanox ConnectX-4 LX with latest Firmware from Mellanox .

Can it be a "Fake" / Buggy Reporting ? Possibly. Not sure if I checked the Kill-a-Watt Meter for those Test to see if the Power Consumption was close to Baseline.

But for sure with the Mellanox ConnectX-4 in a CPU connected PCIe Slot, it would NOT go below PC3.

If you are looking for ASPM and 10G at the same time, then Intel X710 is a nice choice.

It is a nice Choice, although more expensive and less available good Deals IMHO (especially in Europe) .

 

[celemine1gig]

Thank you for your Reply . I doubted it could be Bifurcated since there was no BIOS Option for it. And not quite sure what the link you pointed to shows besides being required to

It can be found when you are logged into your Resource & Design Center (RDC) privileged account at the following website:

How exactly did you enable Bifurcation then ?

Simple and short answer: I am an engineer with the privilige to access all those tools. Fun fact: Even worked for Fujitsu for a short time.
Hint: You can get the tools online, although it is not legally OK. Buzz-words: "CSME System Tools".
Edit: That only applies to the 1x4 slot. For the 2 x8 PEG slots, you need to connect CPU LGA lands, in order to change the bifurcation.

Well, I expected them to have some "Logic" behind it. But I guess it's because the black PCIe Slot is actually only x1 Electrical or ?

It would have been better if they just did a Block Diagram in their Documentation like every other Motherboard Manufacturers on the Planet (e.g. Supermicro or even ASUS/ASRock Rack).

Of course you can say that Fujitsu didn't sell this as a Motherboard but rather a configurable complete System, but still ...

Same here. I also did not know excatly what they did, partly due to the seemingly color coded slots, until I tested in detail.

Uhm, lspci -tv (so you also get the Description in Text as well) works, but it does NOT tell you where the PCIe Slot goes (e.g. the Logical Mapping to e.g. PCIe Root Port XX will probably be different than the Silkscreen on the PCB or the "Slot XX" in the Manual), wouldn't it ?

So how exactly does that Help you ? Trial & Error and keep moving the Card you have plugged in the Slot until you achieve the desired Output in lspci -tv, so you can reverse engineer their Layout ?

I don't need to reverse engineer. If you look closely at the lspci decoded output, you will see that the PEG and the PCH can be differentiated by the naming, as well as the PEG being listed with a lower bus/device/function number.
So, if you have the "tree" output and see where your devices are connected, then you can deduct, which one is a PEG connection and which one is a PCH-connection.

Well, I haven't been proven wrong on that and also other People experienced the same. It's maybe NOT conclusive (hence the "Usual" Word) and I didn't test 300'000 different Motherboards Makes & Model, but on the Supermicro X10SLL* and X11SS* that I tested, turbostat / powertop would show PC6 Package C-State when plugged into a PCH/DMI/Chipset connected Slot.

That was Mellanox ConnectX-4 LX with latest Firmware from Mellanox .

Can it be a "Fake" / Buggy Reporting ? Possibly. Not sure if I checked the Kill-a-Watt Meter for those Test to see if the Power Consumption was close to Baseline.

But for sure with the Mellanox ConnectX-4 in a CPU connected PCIe Slot, it would NOT go below PC3.


It is a nice Choice, although more expensive and less available good Deals IMHO (especially in Europe) .

To be honest I never bothered with anything newer than ConnectX-3.
However, if you want cheap X710 cards, look for the HP 562SFP+. This can be found at VERY reasonable prices, throughout Europe.

 

[luckylinux]

Simple and short answer: I am an engineer with the privilige to access all those tools. Fun fact: Even worked for Fujitsu for a short time.
Hint: You can get the tools online, although it is not legally OK. Buzz-words: "ME System Tools".
Edit: That only applies to the 1x4 slot. For the 2 x8 PEG slots, you need to connect CPU LGA lands, in order to change the bifurcation.

Aaah alright .

I don't need to reverse engineer. If you look closely at the lspci decoded output, you will see that the PEG and the PCH can be differentiated by the naming, as well as the PEG being listed with a lower bus/device/function number.

So, if you have the "tree" output and see where your devices are connected, then you can deduct, which one is a PEG connection and which one is a PCH-connection.

Because if in PCH slot it would have an extra Level / Layer in the tree view, instead of "directly" attached to the PCIe Root Port ?

However, if you want cheap X710 cards, look for the HP 562SFP+. This can be found at VERY reasonable prices, throughout Europe.

It doesn't seem *that* cheap at all unless it's a really bad Period. 80+ EUR from Germany ? That's very close to the "Normal" X710 at 100EUR.

Hp 562sfp online kaufen | eBay

Tolle Angebote bei eBay für Hp 562sfp. Sicher einkaufen.

www.ebay.de

A Week ago or so I managed to order 2 x Intel XXV710-DA2 from the US for 45 EUR each (including import Charges / VAT / Shipping / etc).

That's why the X710 (or HP 562SFP+) does NOT, at least right now, seem so interesting in Terms of Price within Europe, unless imported (US or possibly UK, although UK I didn't see any good deal on the X710, only Mellanox ConnectX-4 LX Deals from UK)..

 

[celemine1gig]

I don't have the system anymore, so I cannot give you a direct example for lspci.
However, if you post an example/examples for an "lspci" and also an additional "lspci -t -s xx:xx.x" for your desired device, I can explain what I meant.

And concerning the X710: If 80,-EUR is too expensive in your opinion, then of course, the 562SFP+ does not cut it. I obviously already considered this a good deal. Of course, this cannot compete with your 45,-EUR bargain example. Nice buy!

 

[luckylinux]

I don't have the system anymore, so I cannot give you a direct example for lspci.
However, if you post an example/examples for an "lspci" and also an additional "lspci -t -s xx:xx.x" for your desired device, I can explain what I meant.

Ah, OK, I guess you moved to something more Powerful. I'm also starting setting up some AMD Ryzen (mostly 2nd Hand Parts) Systems that have like 2-4 x more CPU Power .

NOT all PCIe Slots are populated though

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
           +-01.0-[01]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           +-14.0  Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
           +-14.2  Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem
           +-16.0  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1
           +-16.1  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2
           +-17.0  Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
           +-1c.0-[02]--+-00.0  Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)
           |            \-00.1  Emulex Corporation ServerView iRMC HTI
           +-1c.5-[03]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.6-[04]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1d.0-[05]--+-00.0  Intel Corporation I350 Gigabit Network Connection
           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
           +-1f.0  Intel Corporation C236 Chipset LPC/eSPI Controller
           +-1f.2  Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller
           \-1f.4  Intel Corporation 100 Series/C230 Series Chipset Family SMBus

Not exactly what I was expecting.

I guess everything 1X (like onboard NIC, Matrox VGA, etc) are on the Chipset and everything 0X is directly CPU connected. I would have expected an extra Tree Level / Branch for the Chipset though.

But 1c.# corresponds to the PCIe Root Port #, so that is at least an Exception to what I said above.

And concerning the X710: If 80,-EUR is too expensive in your opinion, then of course, the 562SFP+ does not cut it. I obviously already considered this a good deal. Of course, this cannot compete with your 45,-EUR bargain example. Nice buy!

Yeah from the US it's quite good at Times, if I can manage to pass the Order via eBay (I usually ALWAYS get denied, maybe it worked because I sent them an Offer instead of putting the Article in my Cart, when it ALWAYS get blocked at Checkout stating that it cannot be delivered to my Country) .

Otherwise I got 25-40 EUR / Piece Deal on the Mellanox ConnectX-4 LX from UK, which is why it would be good to use them, if I can get ASPM to work correctly .

 

[celemine1gig]

Ah, OK, I guess you moved to something more Powerful. I'm also starting setting up some AMD Ryzen (mostly 2nd Hand Parts) Systems that have like 2-4 x more CPU Power .

NOT all PCIe Slots are populated though

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
           +-01.0-[01]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
           +-14.0  Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
           +-14.2  Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem
           +-16.0  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1
           +-16.1  Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2
           +-17.0  Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
           +-1c.0-[02]--+-00.0  Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)
           |            \-00.1  Emulex Corporation ServerView iRMC HTI
           +-1c.5-[03]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.6-[04]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1d.0-[05]--+-00.0  Intel Corporation I350 Gigabit Network Connection
           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
           +-1f.0  Intel Corporation C236 Chipset LPC/eSPI Controller
           +-1f.2  Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller
           \-1f.4  Intel Corporation 100 Series/C230 Series Chipset Family SMBus

Not exactly what I was expecting.

I guess everything 1X (like onboard NIC, Matrox VGA, etc) are on the Chipset and everything 0X is directly CPU connected. I would have expected an extra Tree Level / Branch for the Chipset though.

But 1c.# corresponds to the PCIe Root Port #, so that is at least an Exception to what I said above.

It does look a bit weird. Maybe depending on the actual "lspci" version.
I would have expected this for the PEG ports (so both x8 ports):

00:01.0 PCI bridge: Intel Corporation 6th-9th Gen Core Processor PCIe Controller
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller

With the focus on two things:
1.) The PEG ports are listed with lower functions, than the PCH ports! So, "00:01.x"=PEG vs. "00:1x.x"=PCH.
2.) It actually says "Core Processor PCIe Controller" in the description text (if available), which is the PEG.
It is not THAT clear in your example, but you can still see that your Mellanox card is connected to "00:01.0", or in other words, PEG0.
As the PEG can be bifurcated into 3 parts (1x8 and 2x4) at max, you can also have PEG1 (00:01.1) and PEG2 (00:01.2).
The unused ports just don't get displayed.

And like you already realized, the PCH ports have 2 points to identify:
1.) The PCH ports are listed with higher functions, than the PEG ports (00:1x.x)!
2.) It actually says " C230 Series Chipset Family PCI Express Root Port" in the description text, which is a PCH port.
See here for details:

https://cdrdv2-public.intel.com/332691/100-series-chipset-datasheet-vol-2.pdf

Starting page 830, here is a quote for a better understanding:

12.1 PCI Express* Port Configuration Registers
Summary
There are twenty sets of the following configuration registers used for PCH PCI
Express* Port Configurations. Each PCH PCI Express* Configuration Register set covers
a single PCI Express* Port and maps out as the following Device/Function:
D28/F0 = Port1
D28/F1 = Port2
D28/F2 = Port3
D28/F3 = Port4
D28/F4 = Port5
D28/F5 = Port6
D28/F6 = Port7
D28/F7 = Port8
D29/F0 = Port9
D29/F1 = Port10
D29/F2 = Port11
D29/F3 = Port12
D29/F4 = Port13
D29/F5 = Port14
D29/F6 = Port15
D29/F7 = Port16
D27/F0 = Port17
D27/F1 = Port18
D27/F2 = Port19
D27/F3 = Port20

In your example lspci output it shows "1c.0", "1c.5", "1c.6" and finally "1d.0".
That means (with reference to the intel document above) you see:
- "1c.0" = 16 + 12, Function 0 = 28, Function 0 = Port 1 : Internal Matrox BMC graphics chip
- "1c.5" = 16 + 12, Function 5 = 28, Function 5 = Port 6 : On-board Intel i210 #1
- "1c.6" = 16 + 12, Function 6 = 28, Function 6 = Port 7 : On-board Intel i210 #2
- "1d.0" = 16 + 13, Function 0 = 29, Function 0 = Port 9 : Intel i350, as connected by you.

Yeah from the US it's quite good at Times, if I can manage to pass the Order via eBay (I usually ALWAYS get denied, maybe it worked because I sent them an Offer instead of putting the Article in my Cart, when it ALWAYS get blocked at Checkout stating that it cannot be delivered to my Country) .

Otherwise I got 25-40 EUR / Piece Deal on the Mellanox ConnectX-4 LX from UK, which is why it would be good to use them, if I can get ASPM to work correctly .

I am not sure, as I never owned a ConnectX-4, however from what I read (before getting the X710 in the end), the cards simply don't seem to support it. Why, no idea! But so far I did not find a single story, where someone used ConnectX-4 with active and working (!) ASPM.
The following reads somewhat ambiguous. At first it mentions working ASPM, but then says it doesn't do anything, concerning lower power consumption:

Connectx-4 Lx 2x 25GbE with ASPM support + idle power consumption measurements

I’ve recently got myself 2 Connectx-4 Lx cards. I wanted to replace very sketchy connectx-3 cards. (OCP2.0 cards on pcie adapters which negotiate only pcie3x2 D:) I’ve found decent cards on aliexpress for 62 EUR without taxes. It’s a steal at that price. Out of the box card didn’t support...

forum.level1techs.com

Weird stuff!

 

[hmartin]

Slot 3 is a PCH port and can be further bifurcated. If you have full access to the BIOS flash descriptor (i.e. Programmer ready and direct flash access). This way I had an Intel Optane H20 drive in PCIe 2x2 config running in their.

Do you have a source for a PCIe x4->x2x2 bifurcation adapter? Or it's just a function of the H20 to split the NVMe lanes internally to x2x2?

Does somebody know if it support 32GB UDIMM ECC modules ?

It does not, see my blog post:

For those looking to install 32GB unbuffered DIMMs, unfortunately there is no support for that on the TX1320 M3. The capacity is recognized, and I was able to open UEFI Setup, as well as boot the Arch Linux installer, however if the CPU addresses beyond 16GB a non-maskable interrupt (NMI) is generated and the system halts.

Fujitsu TX1320 M3 does NOT like Xeon E3 CPUs with Integrated Graphics (E3-12x5 v6)

The iGPU is not supported and cannot be enabled, see https://www.reddit.com/r/homelab/comments/1138pdy

Theoretically, one of those ports (which one, depends on the question of enabling lane reversal or not) could be further split into 2x4, as the PEG supports 1x8 and 2x4.

The PEG can be split into x8x4x4. Slot 3 PCIe x8 becomes x4x4. No lane reversal is required.

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) (rev 07)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #6 (rev f1)
00:1c.6 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 (rev f1)
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]  # <------------ PCIe Slot 4 (x8)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (DRAM-less) # <------------ PCIe Slot 3 (x4)
03:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller BG4 (DRAM-less) # <------------ PCIe Slot 3 (x4)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
04:00.1 Co-processor: Emulex Corporation ServerView iRMC HTI
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

[luckylinux]

The iGPU is not supported and cannot be enabled, see https://www.reddit.com/r/homelab/comments/1138pdy

I know that.

But when operated with the Intel Xeon E3 with Integrated Graphics, it would NOT work reliably. Neither iKVM neither Boot itself. Very weird. Could be faulty CPU since a User on this Thread reported having no Issues with Intel Xeon E3 12X5 CPUs. Of course no integrated Graphics would work, but the CPU would work as an Intel Xeon E3 12X0 CPU. In my Case, it would basically NOT work at all.

 

[celemine1gig]

Do you have a source for a PCIe x4->x2x2 bifurcation adapter? Or it's just a function of the H20 to split the NVMe lanes internally to x2x2?

I also had that thought, but sadly could not find anything, that would be ready to use, right away.
The Intel H20 Optane is a special case: It is two NVMes on one M.2 PCB, basically. Both occupying a x2 link.

What might work somehow is getting two like this

https://www.aliexpress.com/item/1005008678151438.html

and then rewiring/resoldering them from one x4 slot connector into two physical x2 (electrical, but physical x4) ports.

It does not, see my blog post:

This here might be interesting:

[Guide] How to make Z370 chipset MBs compatible with 128GB RAM

@bios_guy May Mac OS need Intel ME not been disabled. But ME on 22nm PCH now won’t let us use 128G RAM.

winraid.level1techs.com

It might even work for the C236 on the TX1320. As far as I remember, the CoffeTime tool for modding a BIOS with CoffeeLake support (which by the way works on the TX1320), can add this patch for 32GB DIMM support. If it works though, I don't know, as I never tried.

The PEG can be split into x8x4x4. Slot 3 PCIe x8 becomes x4x4. No lane reversal is required.

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) (rev 07)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #6 (rev f1)
00:1c.6 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 (rev f1)
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]  # <------------ PCIe Slot 4 (x8)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (DRAM-less) # <------------ PCIe Slot 3 (x4)
03:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller BG4 (DRAM-less) # <------------ PCIe Slot 3 (x4)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
04:00.1 Co-processor: Emulex Corporation ServerView iRMC HTI
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Lane reversal is not required, that is right. However, what I meant is: Without lane reversal you get x8x4x4, repesenting x8 on the first slot and splitting the second into two times x4. While with lane reversal enabled, you could get x4x4x8. Which means that you could use the first slot as two times x4 and the second one as x8. Of course assuming, that both are actually routed with x8, which I am not absolutely sure about anymore. Anyway. Also theory, as I did not try it and just thought about the possible options.

 

[hmartin]

However, what I meant is: Without lane reversal you get x8x4x4, repesenting x8 on the first slot and splitting the second into two times x4. While with lane reversal enabled, you could get x4x4x8. Which means that you could use the first slot as two times x4 and the second one as x8.

According to the Intel Datasheet, lane reversal would mean that all the lanes are reversed:


Without lane reversal
Slot 4: PCIe x8 (01234567)
Slot 3: PCIe x4x4 (0123/0123)

With lane reversal
Slot 4: PCIe x4x4 (3210/3210)
Slot 3: PCIe x8 (76543210)

I suspect that normal PCIe devices will not work with reversed lanes, unless you have a custom adapter to reverse the lanes back into the expected ordering.

Of course assuming, that both are actually routed with x8, which I am not absolutely sure about anymore.

Slot 3 and Slot 4 are both routed for x8 from the CPU in the TX1320 M3.

 

[celemine1gig]

...
I suspect that normal PCIe devices will not work with reversed lanes, unless you have a custom adapter to reverse the lanes back into the expected ordering.
...

Actually - from my experience - most devices, that use the full lane width don't really care if the lanes go 0-3 or 3-0, as long as the count is full and the order is consecutive.

But obviously you have no chance with devices trying to use only a x2 or x1 link, if they try to link up on something, that isn't the lowest ordered lane.