This thread is, so far, about the M3, which does not have any native M.2 sockets.
For the M4, in the D3673 datasheet it says that the PCIe slots are x8, x4, and x1. So Fujitsu has bifurcated the 16 PEG lanes into x8x4x4 with the remaining x4 going to the NVMe slot. The processor supports this natively and there's no additional PCIe switch.
Hmm I'm not sure I'm getting you right. Here's the board photo and excerpt from the brochure:
So 2x8 is already x16 (PEG or not), then we have x4 + x1 (not sure whether x1 is coming from the chipset) and another x4 in NVMe slot?
Also I wasn't able to find system diagram. Does it exist for M3?
So 2x8 is already x16 (PEG or not), then we have x4 + x1 (not sure whether x1 is coming from the chipset) and another x4 in NVMe slot?
Also I wasn't able to find system diagram. Does it exist for M3?
Sorry, I guess I need to finish my coffee before I reply. The datasheet for the TX1320 M4 states that there are indeed two PCIe x8 slots. That datasheet doesn't mention how the NVMe slot is wired:
PCIe x8 to two NVMe cards don't have a PCIe switch. They require bifurcation support on the motherboard to split the x8 slot into x4x4. I would doubt this motherboard supports it in BIOS though. You could pick up a PCIe to U.2 adapter and use one of these (it is a PCIe switch) to support 4 NVMe drives. Be aware that they do add ~7W of consumption and require some form of active cooling.
Sorry, I guess I need to finish my coffee before I reply. The datasheet for the TX1320 M4 states that there are indeed two PCIe x8 slots. That datasheet doesn't mention how the NVMe slot is wired.
PCIe x8 to two NVMe cards don't have a PCIe switch. They require bifurcation support on the motherboard to split the x8 slot into x4x4. I would doubt this motherboard supports it in BIOS though. You could pick up a PCIe to U.2 adapter and use one of these (it is a PCIe switch) to support 4 NVMe drives. Be aware that they do add ~7W of consumption and require some form of active cooling.
Thanks! Interesting adapter. I won't go this route now since I've already blindly went with 9500-8i for 100 bucks from China... hardware costs are adding up really fast.
Once server arrives I'll test local M.2 with what I have on hand and report here.
It seems I can find it cheaper. Indeed, pairing it with M.2 to U.2 adapter would allow me to still have NVMe but also free up PCIe slots. I wonder if these 7W come from using all 4 drives: https://forums.servethehome.com/ind...arrier-2-5-inch-form-factor.25533/post-413050 . If it draws 3W for 2 drives I think I will accept my fate. I also wonder if that switch supports ASPM, then it would be a very interesting upgrade. Thanks again @hmartin
Hello everyone, i have a problem with my Dell Intel X520-DA2 0942V6 NIC. I get link-flapping in my TX 1320 M3. I checked the modules with ethtool on TrueNAS, which is running on that machine. Seems to be fine. Anyone else is using that card and may have some hints for me about PCI Slot, Bios Setting or other settings?
If you are testing with Linux, give it a try with the Kernel parameter "pcie_aspm=off". I know that the 82599 doesn't really behave too well with ASPM, so give it a try. If that is the case, you can try a more selective approach, that doesn't generally disable ASPM for all devices, but just for the one.
If you are testing with Linux, give it a try with the Kernel parameter "pcie_aspm=off". I know that the 82599 doesn't really behave too well with ASPM, so give it a try. If that is the case, you can try a more selective approach, that doesn't generally disable ASPM for all devices, but just for the one.
OK I finally got M4 and can report some findings here.
The server arrived with Xeon E-2236, 16+16 GB of RAM, 1TB Samsung 970 PRO and NVIDIA GT 710. It also has two redundant 450W PSUs. I've installed latest Ubuntu 24.04.3 just to test the hardware and power consumption.
root@ubuntu-PRIMERGY-TX1320-M4:/home/ubuntu# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.1 Communication controller: Intel Corporation Device a361 (rev 10)
00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1c.3 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
03:00.0 PCI bridge: Emulex Corporation x1 PCIe Gen2 Bridge[Pilot4]
04:01.0 Co-processor: Fujitsu Technology Solutions Device 1228
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
First observations: right from the start, it's impossible to turn it on until iRMC finished booting and ID led blinked (around 20 secs after power is applied). Maybe it's a standard server thing which I already forgot...
Also, sometimes (every other time) I'm unable to enter the BIOS or boot menu after hard power off, keyboard just doesn't work (I've tried 3 different ones). After I boot the OS and then reboot from it, I can enter BIOS just fine. Weird.
PSU
Since I've got dual redundant PSUs, BMC gives a warning when one of them is not connected, which's okay but IMO overkill for a home lab use. I disassembled one of them to check if it's viable to do Noctua swap as @Gruenschnabel wanted to do, but I'm not really sure it's worth it: https://www.fansco.com/uploads/2024/04/MGT4012UB-W28-Protechnic.pdf this is the fan inside (but 4 pin), and per suitable Noctua NF A4-20PWM specs, Noctua's flow rate is ~6CFM (vs 23 CFM) and RPM is also 5K vs 13K. I think what will happen is that BMC will try to force full 13K RPM on startup/check and won't be able to do that. All of this of course if PWM signal is the same. Last nail to the coffin for me is the pricing: new Noctua fan is 16 EUR, while non-redundant PSU is 17 EUR on eBay... the only downside of standard PSU for me (apart from max power, but I doubt I'll go over that) is that it's not rated as Premium, so efficiency will be lower. You win some, you lose some...
Lastly, noise: redundant PSUs are not loud per se, but this small fan is running all the time and produces quite audible low-frequency hum, which only goes away when everything's off and the plug is pulled. This hum is very well transmitted over the floor and I can tell the difference when the server is switched on in the other room.
Drives backplane and controller
Per @hmartin 's blog suggestion I've replaced the backplane with a unified one (where single board can handle 8 drives). I wish I had read further about SATA card
So long story short, I've decided to buy Broadcom HBA 9500-8i SAS/SATA/PCIe(NVME) 12Gb/s SAS3808 controller for 100 EUR from China. It works, but the problem is that it has Slimline SAS x8 connector. The only matching cable I was able to find to plug into backplane has straight connectors, which prevents backplane fan shroud from closing. That's very disappointing. I've looked into Fujitsu spare parts list, and their controllers based on the same chip (like EP520i) have "standard" SFF-8643 connectors, which allows plugging cheap and vastly abundant Mini SAS HD (SFF-8643) cables. Unless I find an angled cable or adapter I won't be able to use the this controller. Also, backplane fan is quite noisy and power hungry: Delta DS07025B12U, 12V 0.7A. Fan from M3 is 0.3 or 0.4A, but is also thinner by approx 5mm, so it won't fit in the same frame... and of course 70x70mm is pretty non-standard size and apart from Delta and random Chinese vendors you won't find anything. Ah yes, fan shroud + fan for M4 costs 46 EUR on eBay...
I've also had a random LSI 3008 controller laying around which I wanted to compare. As you can see, 9500 has ASPM support enabled, while 3008 does not. Also 3008 gets really got even when nothing is plugged in.
Also I able to confirm that 2nd M.2 slot doesn't recognize NVMe SSDs. I don't have any MSATA M.2 SSDs around, and since I wanted to have NVMe mirror, I plan to utilize that Viking U.2 adapter. The only question is where to put it, and there are not many options: either PCIe slot (but then it will take 1.5 slots, and won't be able to cool it properly), or try to construct some kind of holder to put into RDX drive slot, which has very weird 1.75x4 inch dimensions. Maybe I'll buy some old RDX drive just for the parts...if this works out, two small Noctuas should cool it reliably, but then again total solution cost is quite high. (Viking 20 + RDX frame 30 + U.2 to M.2 cable + 20 == 70 EUR).
Power consumption Finally, some numbers (taken from the wall).
Switched off (iRMC booted up): 7W (I blame redundant PSU fan)
Booted into desktop (NVMe SSD and GT710): 24W
9500 controller installed: 28.4W (no drives attached)
3008 controller installed: 33W (no drives attached)
These are without any powertop tuning. All C-states are enabled in BIOS, but I feel like at least 10W can be saved if those Delta fans were replaced.
First observations: right from the start, it's impossible to turn it on until iRMC finished booting and ID led blinked (around 20 secs after power is applied). Maybe it's a standard server thing which I already forgot...
Yes, this is sadly the case. Fujitsu also did not optimise iRMC for fast boot times at all. I guess enterprise customers don't complain about this enough.
I don't know why you don't consider an ASM1166 based card? If you plan to use the 8 drive backplane, you can only connect one half of the BP to the PCIe controller and leave the other one with the onboard ports, and you won't suffer any performance penalty. As a bonus, they are really cheap compared to a "big boy HBA" and consume basically no power.
This is typical, replacing the redundant PSU with the fixed model will not improve it more than 1 or 2 watts. The PSU needs to maintain the +11VSB and the BMC is not power optimised in the slightest. Looking at the smart plug on mine, it's drawing 6W powered off (so only iRMC powered).
I don't know why you don't consider an ASM1166 based card? If you plan to use the 8 drive backplane, you can only connect one half of the BP to the PCIe controller and leave the other one with the onboard ports, and you won't suffer any performance penalty. As a bonus, they are really cheap compared to a "big boy HBA" and consume basically no power.
Yeah I understand it now. I'll probably sell that HBA and cable then.
EDIT: @hmartin could you share a link to the cable you've used with this card? There are plenty of MiniSAS HD to 4 SATA cables, but the only angled SAS connector (so that backplane fan shroud can close) I am able to find has some additional connector which I don't need: Supermicro CBL-00129-01-A-R
EDIT: @hmartin could you share a link to the cable you've used with this card? There are plenty of MiniSAS HD to 4 SATA cables, but the only angled SAS connector (so that backplane fan shroud can close) I am able to find has some additional connector which I don't need: Supermicro CBL-00129-01-A-R
This is typical, replacing the redundant PSU with the fixed model will not improve it more than 1 or 2 watts. The PSU needs to maintain the +11VSB and the BMC is not power optimised in the slightest. Looking at the smart plug on mine, it's drawing 6W powered off (so only iRMC powered).
I'm happy to report that the system with standard (non-redundant) PSU is pulling 3.5W in a switched off state (I guess mainly because PSU fan is not running).
I doubt it has a dedicated part number, but it might explain why "Fujitsu Primergy 1320 M4 PSU" was 16 EUR and exactly the same one but looked up by the part number costs 130 EUR...
The Mounting Frame should be Part of the Chassis, NOT the PSU, as it shouldn't be required to be replaced during a PSU Replacement (it's just a Metal Part after all).
I only see 2 Options:
- Design and 3D Print some Bracket (make sure you use some relatively high Temperature Material such as PETG)
- The TX1320 M3 Operating Manual seems to indicate that the Bracket is the same (is it really though ???), so you might be able to get a Barebones Chassis without even Motherboard
I think you're right and it's a part of the frame; if the unit is ordered with redundant PSUs from the factory, they probably don't include the bracket anyway because this kind of reverse PSU swap doesn't make sense for a customer. I spent few hours yesterday trying to find the frame but I think it's a lost cause.
I don't have a 3D printer and I think I will just reinstall redundant frame back and just use one of the PSUs, so iRMC doesn't give me an error when only one PSU is plugged in (but both are installed). Maybe try the Noctua mod, since I have two PSUs anyway
I think I will also revert to the old backplane (which is comprised of two halves) because ultimately I need CD power and Molex for NVMe connectors, and included backplane power cable conveniently has all of them. I don't mind crimping my own cable, but I trust Fujitsu more
I decided to check drive prices again and small capacity (< 4 TB) SAS are really cheap, cheaper than HDDs and I think it's guaranteed they're not SMR. I've never used SAS drives before and I really don't know what to expect, but if I get 7200 RPM drive I think it shouldn't be much noisier? I've also checked power ratings and they're consuming 2-3 W more than regular 2.5" SATA HDDs, but if spin down command works I don't think it's going to be a problem. Also as far as I understand, they should still work with that ASM1166 card.
In my old NAS I'm going to migrate I currently have 6x1TB 2.5" WD Reds, but since I'll have 8 slots in the backplane now I think I can put 2 SAS drives and see how it goes.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
