Thermal throttling (?) with 2xQL2M (Xeon 8160 ES) on X11DPH-T

[tesla100]

There are no cards whatsoever connected to the motherboard, I use onboard VGA.

But I'll try the other options you have suggested, thanks.

 

[romanstingler]

can you show me a bios screen of the cpu page.

and can you try a linux live usb stick and open a terminal and enter
dmesg
and send me the output(very long) pm or link

 

[tesla100]

Hi there and thanks for all your help! Here come the bios screens. I tried disabling the virtualization and enabling LLC Prefetch as you advised, but it didn't have any impact on the performance.
As for the linux usb stick, I will try my best and let you know.
And by the way, I got a response from the seller, this is what he says:

Hi,Someone just told me,
this item is special,
you just only can use 1pcs on the board, if you put 2pcs QL2M, the speed is very low,
but if only put 1pcs, must be faster,
and also you can buy 1 more QL1K with 1pcs QL2M, there must be faster than 2pcs QL2M.
you can try 1pcs first.
and any news, let me know

Obviously I answered him, that I only run it in 1P configuration and since then, no answer again...

 

[tesla100]

dmesg output:

dmesg - Pastebin.com

 

[bsjln]

hey chrgrose and tesla100!

any progress on this? I have exactly the same setup as you ie X11DPH-T with 2x 8160 QL2M ES chips (with fabric) and see about 250-300 MHz speed on all cores - core temperature according to coretemp in Linux is around 20C but package temperature is 103C both in Linux and when looking at sensors in IPMI web interface. Tried BIOS versions from 1.0 up to 3.0 (but 3.0 doesnt support ES I noticed so you cannot even get into BIOS) but no difference. would love to get these system up to full speed!!

 

[tesla100]

hey chrgrose and tesla100!

any progress on this? I have exactly the same setup as you ie X11DPH-T with 2x 8160 QL2M ES chips (with fabric) and see about 250-300 MHz speed on all cores - core temperature according to coretemp in Linux is around 20C but package temperature is 103C both in Linux and when looking at sensors in IPMI web interface. Tried BIOS versions from 1.0 up to 3.0 (but 3.0 doesnt support ES I noticed so you cannot even get into BIOS) but no difference. would love to get these system up to full speed!!

Hi there! After many days of looking desperately for a solution (btw. many thanks to @romanstingler for all the efforts and help) I bought another X11DPH-T motherboard and with this one I am able to run one QL2M in the socket 1. If I insert the second QL2M in the socket 2, once again its frequency won't go above 700MHZ and the whole system gets really slow (although the first CPU runs at its normal frequency). So ordered two QL1Ks and I am going to try to mix one QL2M and one QL1K together and hope for the best.

 

[bsjln]

Good to hear! Unfortunately both of mine are behaving the same when running alone in the first socket - and I see the same thing when running both also - but then mine doesn't even go to 700 MHz. I guess on all the dual 3647 boards with sockets for fabric there is only one of the sockets that are for fabric CPU with the retention for the OmniPath cable and perhaps then something (power supply lines for fabric or so?) is missing on the second socket.

Also found this on SuperMicro FAQ which worried me a bit:

Question
I would like to use Intel Xeon Gold 6138P (FPGA) on X11DPH-TQ.
Please confirm the compatibility, thanks.
Answer
All X11 Purley MBs already removed FPGA and -F (Omni-Path) CPUs support.

(FAQ Entry | Online Support | Support - Super Micro Computer, Inc.)

 

[tesla100]

Well, on the first X11DPH-T motherboard I bought I had the same problem like you (both CPUs, either running alone in the first socket or all together, were running at 700MHz max). But with the second X11DPH-T motherboard at least in the first socket they work as expected. The first X11DPH-T motherboard I bought is rev 1.10 and the second one is rev 1.01. Maybe they really suspended the support for OMNI path Xeons in the newer revisions, because that retention for the Omni Path cable you are talking about is missing on the revision 1.10 motherboard.

 

[tesla100]

I think it all starts to make sense to me. According to this article, if there is no metal retention clip on the socket, it probably won't support the omni path xeons:

https://www.servethehome.com/intel-...oes-not-work-on-every-server-and-motherboard/

On the revision 1.10 motherboard I have here there are no metal clips on neither of the sockets, on the revision 1.01 motherboard this metal retention clip it is present in the socket one only. That's why my Omni paths xeon I have run well in socket one of the rev 1.01 motherboard, but not in the second socket and obviously it won't run in neither socket of the rev 1.10 motherboard!

 

[bsjln]

I think you are absolutely right. I bought this board because it was supposed to have support for OmniPath CPU and retention clips on socket 1 but mine is also revision 1.10 and do not have any retention clips.

 

[tesla100]

Yes, exactly, that was what I thought as well.

 

[bsjln]

Confirmed now with Supermicro tech support that X11DPH-T socket with retention clips was replaced for one without already sometime October-December 2017 and that then later revision 1.10 PCB does not support F-series CPU at all. The reason for this according to Supermicro is that Intel dropped support for F-series CPU in microcode and everything - so it could be good to stick to BIOS and microcode from when they actually were supported... Also I would really like to find a couple of pcb revision 1.01 motherboards if anyone has any clues - or actually anything they could run on

 

[Edwin812]

Hi All,
I got exactly the same issue with Supermicro and QL2M.
Should there any problem for mix QL1K and QL2M?

 

[tesla100]

Hi All,
I got exactly the same issue with Supermicro and QL2M.
Should there any problem for mix QL1K and QL2M?

Hi, what motherboard do you have? And what revision?

Cheers

 

[Edwin812]

Hi, what motherboard do you have? And what revision?

Cheers

I got X11DPFR-SN v1.01.
It's always 103c and run around 700MHz.

 

[Edwin812]

Hi there! After many days of looking desperately for a solution (btw. many thanks to @romanstingler for all the efforts and help) I bought another X11DPH-T motherboard and with this one I am able to run one QL2M in the socket 1. If I insert the second QL2M in the socket 2, once again its frequency won't go above 700MHZ and the whole system gets really slow (although the first CPU runs at its normal frequency). So ordered two QL1Ks and I am going to try to mix one QL2M and one QL1K together and hope for the best.

As on this post. I would like to know that did you try to mix QL1K and QL2M.
And how was the result?
I am thinking to buy QL1K or not.

 

[tesla100]

I got X11DPFR-SN v1.01.
It's always 103c and run around 700MHz.

Unless the CPU sockets (or at least one of them) on your motherboard have these metal retention clips, the QL2M CPUs (with Omni path fabric) are not supported.

I had to mix the QL2M with the QL1K because my motherboard had that metal retention clip on the CPU socket 1 only. So I put the QL2M in that socket and the QL1K in the other one.

It worked flawlessly after that.

Cheers

 

[Edwin812]

icic~ My motherboard did not have the metal retention clip. So It may not work for me even I try to mix QL1K and QL2M.
I think I may also buy a QL1K and try. Will reply here even success or not.
Anyway thanks for your reply.

 

[tesla100]

icic~ My motherboard did not have the metal retention clip. So It may not work for me even I try to mix QL1K and QL2M.
I think I may also buy a QL1K and try. Will reply here even success or not.
Anyway thanks for your reply.

Buy two QL1Ks, or even better, any Cascade Lake-SP ES chips... QL1Ks have a really low base frequency...

 

[Edwin812]

Buy two QL1Ks, or even better, any Cascade Lake-SP ES chips... QL1Ks have a really low base frequency...

Many thanks for suggestion. Where you buy those CPUs?