LGA 1700 Alder Lake "Servers"

[nva]

I just bought Asrock Industrial IMB-1230 and got into problem. Initially it can't boot with 13th gen and I had to borrow a 12th gen to upgrade BIOS. Now the 13th gen CPU can boot into BIOS just fine but can't boot into any drive, including UEFI flash drives.

Specs say DDR4 3200 SO-DIMM but i'm running on single SO-DIMM stick of DDR4 2666. It shouldn't be the problem, right?

I think there's a bent pin in the socket, but since CPU can go into BIOS it also wouldn't be the cause.

I found the problem. This Asrock industrial board boots just fine without X710-DA2 plugged in. The NIC is x8 and was working fine. I don't know what's the reason it refuses to work in x16 slot in this board.

This X710-DA2 is working with other boards. Other NICs, including old Mellanox ConnectX-2 cards, boot with this IMB-1230 board. But 2 of them does not boot together.

 

[Qiuzman]

So for a small esxi build PFsense/windows server/windows 11/etc would the imb-1231 not be a good option due to the LAN card? I am fine living with ddr4 and non-ecc ram. It’s only 300 dollars which seemed like a decent price.

 

[Stankyjawnz]

So for a small esxi build PFsense/windows server/windows 11/etc would the imb-1231 not be a good option due to the LAN card? I am fine living with ddr4 and non-ecc ram. It’s only 300 dollars which seemed like a decent price.

For a small pfsense/virtualization machine have you seen the minipcs?

https://www.servethehome.com/almost-a-decade-in-the-making-our-fanless-intel-i3-n305-2-5gbe-firewall-review/

If you're ok without ecc I'd look into these you can get a whole system for around the price of that motherboard.

 

[zir_blazer]

I'm looking for W680 boards that have Intel Boot Guard disabled and ME in Manufacturing Mode (Albeit I recall than on some boards this was a Jumper, so it may potentially change) to propose as possible targets to port Coreboot to them. My prefered target was the ASUS PRO WS W680-ACE (IPMI or normal), but the more information, the better.

If you want to help, you need to:
Install whatever package in your Linux distribution provides msr-tools. Then run from a Terminal:

sudo modprobe msr
sudo rdmsr 0x13a

And report back output. If Boot Guard is enabled, it is not possible to do it without vendor support.

There is a far more convoluted procedure to use intelmetool that also tells whenever the ME is in Manufacturing Mode, but is not as simple: BootGuard status

Boot with iomem=relaxed.
sudo modprobe msr
git clone --depth=1 https://review.coreboot.org/coreboot
cd coreboot/util/intelmetool && make
sudo ./intelmetool -b

But this is overkill.

 

[unwind-protect]

Any DDR4 taking boards for $350? Can't see any.

 

[reasonsandreasons]

I got my Asrock IMB-X1314 for around that, but they're hard to come by. Works well, though, plus or minus IPMI (currently using the AMT firmware loader).

 

[vamega]

I think the GigaIPC mITX-Q67EB would work.
DDR4, i226, ~$250.

 

[ServerSemi]

I'm thinking of getting a 14900K and I wonder who makes the most stable mobo nowdays?

 

[zir_blazer]

I think the GigaIPC mITX-Q67EB would work.
DDR4, i226, ~$250.

Q670E does not support ECC which is the big selling point of the W680. The only other Chipset that does is R680E, but you sacrifice overclocking capabilities, and not sure if there are any other reelevant features: Intel product specifications

 

[vamega]

I got my Asrock IMB-X1314 for around that, but they're hard to come by. Works well, though, plus or minus IPMI (currently using the AMT firmware loader).

Does the X1314 allow for CPU undervolting?

I'm thinking of pairing this with a 13600K, and know these chips can run hot.
I've read that undervolting the chip can help reduce the temps, but not sure if industrial boards like these from ASRock support undervolting.

 

[RolloZ170]

I'm thinking of pairing this with a 13600K, and know these chips can run hot.

what ? not hotter than other CPU with same clocks. disable turbo and be cool.
btw: if undervolting is good for stability, Intel had done at factory.

 

[infuriatedream]

New ASUS W680 ACE BIOS Update v2802 is out, date according to website: 2023-10-06, date of CAP file in zip: 2023-09-27
Seems like 14th Gen Raptor Lake Refresh will also be supported on the W680 ("Update Microcode for next-gen Intel Processors"). Props to ASUS for that.

PRO WS W680-ACE BIOS 2802
Version 2802
12.48 MB
2023/10/06
"1. Update Microcode for next-gen Intel Processors
2. Update ME to version 16.1.30.2264 for next-gen Intel Processors
3. Improve system stability
Before BIOS update, please download the Intel ME update tool from the ASUS support site and update the ME firmware to Version 16.1.30.2264 to ensure optimized system settings.

There is also a new ME firmware also dated 2023-10-06 on the website:

MEUpdateTool
Version 16.1.30.2264
7.84 MB
2023/10/06

 

[custom90gt]

To add to the information about power usage, I have around 70W idle with a 13600k CPU, 64GB ECC RAM, 14 SATA SSDs, 1 HBA, 1 10gb NIC, the IPMI card, 2 Coral TPUs and fans (case, CPU, added small fans to the HBA and NIC, since the HBA was overheating and I use a PC case). I run TrueNAS with a Linux VM and will soon add a Windows VM (haven't setup everything yet, so power usage may yet go up).

One thing to note about power usage and C-States is that C-States is linked to PCI devices. If you have at least 1 PCI device, which doesn't have ASPM support, you will never see lower(higher) CPU C-States (even if you have them enabled) because it doesn't allow the CPU to go to lower power states. In my case even things from the motherboard don't have ASPM support, but also the IPMI card doesn't seem to have ASPM support, almost no HBA card has it and only specific NICs have ASPM support.

So if you want to have low power usage, your best bet is to have your NAS storage on a separate machine (if you need one) and use this with as few PCI devices as possible and have them support ASPM to see true power savings (no IPMI card, no HBA and NIC with ASPM support). Otherwise it doesn't matter what you add to it, because it will already have "wasted" power usage and you can shave around 5W with some settings from BIOS and powertop.

About the IPMI card I too used it mostly for initial setup, since I didn't have an additional monitor and the only complaint I have from it is that its sensor readings aren't correct, which means controls from it aren't reliable for fans (also the temperature sources are limited) and you need to use some software defined sources and curves (haven't done this yet, since the current kernel for TrueNAS doesn't support this IPMI model). Also you can only control 4-pin PWM fans and since most case and additional fans are 3-pin, you need a separate fan hub, otherwise they will run at full speed.

Sorry to bump this older post, what 10GB NIC are you using for these power numbers?

 

[memphis2k]

Hey all,

Having issues with my w680 boards, mostly memory from what I can tell.

This all started earlier this year but I wanted to build a new server. 13900k CPU and thought ECC DDR5 would be great. I have been building computers for 25+ years so I have been through a lot, seen a lot and tried a lot.

First bought an Supermicro W680 board & 64GB of RAM (Micron ECC one), Samsung 990 Pro, Windows Server 2022, 3x Gold 16TB Disks etc etc. A ton of issues, turns out I had a bad 13900k CPU. Before concluding that the CPU was bad, I had bought another Supermicro board & the ASUS W680 ACE IPMI board, 64GB of ECC DDR5, Hynix brand I believe... bought directly from Supermicro. Returned both Supermicro boards and kept the ASUS W680. PSU is an new 850 Gold. Tried another 850 watt PSU, no change.

Then I started getting random blue screens after around 5-6 days of uptime. Months of trying different combinations, found it to be my 2TB Samsung 990 Pro SSD. What the heck right? Bought an WD 850 2TB and no more blue screens. I had perfect documentation of what I tried and and what worked, what didn't etc.

Been really happy with the ASUS W680, but now I'm trying to add more memory. During all the troubleshooting, I just left 1 stick in the mobo, just 32 GB. But now I'm adding more and all I get are blue screens. Then tried booting off an USB stick with memtest86. No issues found with 128GB installed.

A year of struggling with an W680. Re-built my server OS's multiple of times, spent way more money that I had thought and waste plenty of time.

I feel like if I reboot my server now, it may or may not blue screen. I mean heck, its been blue screening on a single stick after adding 1,2,4 sticks. Followed the manual on the order of installing RAM. Havn't tried tweaking much for uEFI settings. I turned down anything saying overclocking.

Should I just go buy a used HP or Dell Server? All the Xeon classes make no sense to me, always googling. Clueless. Willing to spend about 1k-2k

I thought maybe just go Z790, but people are saying its hard getting 4x 32GB = 128GB stable

Been researching the W790 Chipsets. Either ASUS or ASRock look good. CPU & boards look about twice as much... hmm. One person said they populated all DIMM slots and it won't boot. I alsmost feel like I"m better off with DD4 ECC.

I would like an Desktop class CPU and a workstation MOBO that would work, that's why I went with W680.

Wants:
Windows Server 2022 with a couple of VM's for Ubuntu & Home Assistant, Plex, Blue Iris and 14x 4k Cameras and analytics. Getting into Proxmox and containers.
Hope for something with similar power as my 13900k
Need 3x NVMe, Boot Drive, VM's SSD, & 4TB for camera storage
Not really an AMD guy (since early 2000's) but whatever is reasonable. Someone found an older EYPC for a good price, under $1k
Would like IPMI but an PiKVM would work. Couldn't really get my ASUS IPMI working with HDMI out.
HDMI output please (view cameras on an remote HDMI monitor)

Thanks to anyone who responds.

 

[infuriatedream]

Having issues with my w680 boards, mostly memory from what I can tell.

Perhaps your CPU socket was damaged during all those CPU and board swaps. Or the CPU just needs to be reseated because one of the tiny spring maybe is not correctly making contact (I heard of this mostly regarding large server cpus with ~4000 pins but this might also happen on normal cpus). Regarding what the issue is, you need to pay attention to the actual error messages on the bluescreen and that is the only thing that you did not include. If by " I started getting random blue screens" you mean that the actual bluescreen error randomly changes each time, then yes that is likely memory related. If by that you mean that the occurance of the bluescreen happens at random times and intervals, you need to pay attention to the actual error codes. If your nvidia display driver causes memory bluescreens or your boot ssd suddenly disappears and windows has no drive anymore, then you can swap memory as long as you like. The error messages tell you where the problem is, at least most of the time.

My ASUS W680 is running with 4x32GB@4800 without increase in voltage or any special OC settings. The only initial problems I had were related to the Proxmox kernel not properly working with the i225/i226 ethernet but that was resolved by the newer proxmox kernel that has since become standard with current versions. So my initial reduction of memory speed due to those proxmox crashes were not actually needed. And then my CPU died but so far zero problems with the replacement cpu.

 

[memphis2k]

I have been building PC's since the early 90's. Been through a lot, but never my life had this amount of issues. You make very good points. I have been doing deep dives into each blue screen. And its good to know that someone can get 128GB working as the manual claims. Memtest86 passed fine with all 4, but random blue screens upon boot, then I remove 2 sticks and still getting them, then down to one stick, still getting them, and then after the 3rd try, complete power off, then maybe Server 2022 will successfully boot. Blue screens are mostly hardware related IMO.

Would be nice if there was a way to test for a bent pin. Verify that all 1700 pins are making contact. Never had any benchmark fail. Its a server, I don't want to cross my fingers if it will boot or not. There's a new uEFI out and a ME update I will do, and just maybe that will help.

Do you know what brand / model of 4x RAM sticks you have? ECC?

Did you document any uEFI changes you made? From say Factory defaults.

The i225/i226... I had the issue with my mini PC for my firewall.

 

[JanR]

I discovered this thread while researching for my next server build. Since I run an X13SAE-F machine at work (13900K, 4 x 32 GB Kingston ECC RAM), I have already some experiences with the LGA 1700 platform but I also see some issues that needs to be resolved.

With respect to stability, the X13SAE-F runs rock stable with semi-passive cooling (Noctua Noctua NH-P1 located in the airstream of two 140 mm fans running at 140-980 rpm). However, I set the power limits to something more reasonable (PL1 and PL 2 to 125 W) since this is not an OC gaming box.

These limits can also be adapted while running linux - just consider the writable files in /sys/devices/virtual/powercap/intel-rapl/intel-rapl:0 . Doing this, I made some experiments with kernel compilation (individual configuration, so no comparison possible, make -j 64):

Power limit always means PL1 and PL 2 which I configured the same way:

125 W: 1m12.844s
95 W: 1m21.445s
65 W: 1m50.487s
150 W 1m7.935s
220 W 1m2.876s
240 W 1m1.690s

Therefore, it is obvious that it makes no sense to go beyond 125 W PL 1 and PL 2. However, I'm disappointed by the performance at 65 W since I hoped for a better efficiency curve.

With respect to ECC, dmidecode reports SEC as on all ECC-capable Intel consumer platforms.

Unfortunately, there is no EDAC driver for alderlake or raptor lake. The igen6-driver mentioned before in this thread is NOT for ECC but for reporting on errors resulting from on-die ECC on some very special SKU. See driver code drivers/edac/igen6_edac.c for details:

The initial comment states:

/*
* Driver for Intel client SoC with integrated memory controller using IBECC
*
* Copyright (C) 2020 Intel Corporation
*
* The In-Band ECC (IBECC) IP provides ECC protection to all or specific
* regions of the physical memory space. It's used for memory controllers
* that don't support the out-of-band ECC which often needs an additional
* storage device to each channel for storing ECC data.
*/

Later comes a list with supported SKU. I tried to include the SKU I used here but this was not successful. This was to expect since the CPUs do not show the corresponding bits in the PCI config space.

However, the memory controller of a LGA 1700 cpu WITH real ECC RAM (72 or 80 bit) should log errors as documented in intels files. The only issue is: So far nobody wrote an EDAC driver for this. Does anybody knows more on that?

One last question: Are there any experiences on idle power consumption for X13SAE vs. X13SAE-F in order to figure out how much the BMC really needs?

 

[James C. Owens]

I wanted to alert folks that Supermicro has released BIOS version 3.0 for the X13SAE(F). They did not provide a changelog, but my guess is that it enables compatibility with 14th gen Raptor Lake CPU's. I also noticed it added higher speeds in the memory speed selection drop down, up to 5600 MHz. I think the limit was 4400 before. I now am running my ECC memory at 4800 (the rated speed for the Micron modules). No trouble whatsoever.

Note that the BIOS upgrade erases the BIOS settings, including any custom fan settings, so you need to make note of them before upgrading.

 

[James C. Owens]

There is no compatible EDAC driver for the W680 chipset for Linux yet. I think severe memory errors will get logged through ACPI and/or SMI/machine check alerts though.

 

[JanR]

There is no compatible EDAC driver for the W680 chipset for Linux yet.

Unfortunately, it looks like no one is working on them. I hope this will become better once Intel releases the Xeon E-24xx series based on LGA 1700.

I think severe memory errors will get logged through ACPI and/or SMI/machine check alerts though.

I'm not sure. At least I was not able to use error injection on X13SAE-F to force such an error in order to figure out if this really works. Additionally, I'm also interested in CORRECTED errors since these give some impression on the quality of RAM.

For example, my previous machine at work (Xeon E5-1650-v4) produces several corrected errors per month (24/7 operation) but the number stays the same since installation of the machine (seven years ago).