overclocking Xeon E7 8894 v4 (x8) inside Lenovo X3950 X6 - "all core boost" ?

[ootronicsnazleaki]

Interesting thread.

I've got a partially built x3950 which is having some issues.

Currently running 6 E7-8895 v3 with 5 cpus having 8 16GB 2xR4 DDR4 2133 sticks running at 1600Mhz in independent mode.
One cpu is throwing some SMI Link errors and only seeing 6 sticks.


None the less I installed Wine and ran Cinebench R23.

Getting 39,350 with bios set to use 12 core instead of 18 so that their turbos stay up.

I am hoping once the memory is fully sorted and the additional 2 cpus are installed that I can post a comparison for the 8894 above.

Memory not being 8x 8x has significant performance implications according to the documentation.

 

[ootronicsnazleaki]

It might be really amazing if you could just slot v4's in the DDR3 books, since both the books and DDR3 ram are so much cheaper.

From my reading of the docs on the DDR3 books, they won't run the DDR3 ram at 1600Mhz in performance mode, fastest the ram runs is 1333MHz.

System x3950 X6 (3837) Product Guide (withdrawn product)

The System x3950 X6 server is an eight-socket 8U rack-mounted server that represents the sixth generation of the Enterprise X-Architecture. It delivers fast application performance, is based on an agile system design, and is a resilient platform that is needed for mission-critical databases...

lenovopress.lenovo.com

 

[MichalPL]

Did you figure out the best slot to use for them? I'll also probably populate the front with some 12Gbs SAS HDD's.

Yes, if you are talking about 1 or 2 NVMe it's not a problem and everywhere will achieve 3.4-3.6GB/s but still closer to CPU1 is better

https://lenovopress.lenovo.com/sg248208.pdf

Page 22 (page 40 in PDF)


So slots connected to the CPU1 are the best:
Any slot in "Storage book" (one of them should be available, second one taken for RAID controller) or Slot9.
if not: any PCIe connected to CPU2 ,CPU3 or CPU5 (so the CPU's directly connected to CPU1, for example CPU4 is not good and connected via CPU3).


12Gbs SAS HDD's:
Tell us the results, I have no experience with them

 

[MichalPL]

I was thinking about something else lately also. You mentioned, I think, that you first got the system with DDR3 compute books and upgraded to DDR4 books for the v4 Xeons. Did you ever try using the v4's in the DDR3 books? v4 CPU's appear to not be officially supported in the Lenovo manual, but I noticed that the official specifications for the CPU's themselves DO support DDR3:

Complicated.
Yes, I ran 4x 8894 v4 on DDR3 but inside HP DL580 G8/G9 with SPI board from G9 (also in theory support DDR4 only).
DDR3 was maybe 1% slower than on DDR4 in benchmarks (in R23 even slightly faster- lol, ~62000 points).

When having HP DL580 G8 and G9 at the same time you can also swap DDR3 RAM cassettes into new model and it will work fine, DDR4 will fit into G8 but will not work (until SPI board upgrade).

but with Lenovo x3950 X6 I didn't tested it, and my play was to buy just 1 and because it's too heavy so bought DDR4 variant.

Lenovo System x3950 X6 (3837) - DDR3, v2/v3, no info if boot from NVMe, not sure if support v4
Lenovo System x3950 X6 (6241) - DDR4, v3/v4, booting from NVMe

for example HP DL580 G8 - was not booting from NVMe, almost identical G9 support's it. Based on the HP experience I am >>> guessing <<< that new model might support older DDR3 CPU books with v4 inside and DDR3, but old board with old UEFI will not see v4 CPU's almost for sure.

Fortunately, DDR4 is just slightly more expensive now (talking about slow 2133MHz 32GB DIMMs ECC that nobody want vs 1866MHz DDR3)- so no huge difference here. DDR4 is also slightly faster here (1600 vs 1333).

 

[MichalPL]

btw. complete Lenovo System x3950 X6 (6241) - DDR4 for 999EUR!

Lenovo X3950 X6 Server 8x Xeon E7-8890 v3 18-Core CPU 0GB RAM 4x SFF 2,5" | eBay

Find many great new & used options and get the best deals for Lenovo X3950 X6 Server 8x Xeon E7-8890 v3 18-Core CPU 0GB RAM 4x SFF 2,5" at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[MichalPL]

Currently running 6 E7-8895 v3 with 5 cpus having 8 16GB 2xR4 DDR4 2133 sticks running at 1600Mhz in independent mode.
One cpu is throwing some SMI Link errors and only seeing 6 sticks.

I think it can be thermal paste on CPU pads or LGA pins. SMI is between CPU and Jordan Creek chipset, swap CPU's between books and check.

I have experience just with 2 of them:
4x 8895 v2 - about 37000 in R23
4x 8894 v4 - about 62000 in R23

8x 8894 v4 - not working in CB R23 (too many threads).
I have also 8890 v3, but they are slower than 8895v2 so not testing them carefully and never using.

What is funny 8895v2 and 8894v4 are almost identical in single-core performance (other than floating points).

RAM:
yes 8x8 DIMMs optimal minimum, in CB will not make big difference (R23 don't need bandwidth and is checking what GPU should do nowadays ) but in other benchmarks it will make a difference.

 

[ootronicsnazleaki]

I've inspected the pins with camera and cannot detect anything out of shape or any debris. I've also swapped out known RAM and CPU from good book into SMI Link error book and the results are same. Good book works fully, and bad book retains SMI Link errors in same Dimms (which are paired front back of board same channels?)

The only idea I have remaining is that possible there is some imperceptible film on the pins which may be removed somehow with spray alcohol or something. Not sure if I would try it though.

I'm also interested in trying a E7-8895 v2 in a Proliant which takes E5-26xx chips, I think it 'might' work...

 

[RussianE39]

No, E7 wont work in E5 platform. E5 has internal memory controllers, while E7 has Scalable Memory Buffers instead of "raw" Memory Controllers.

 

[chrgrose]

I'm trying to configure my x3950 x6 (6241-AC4) with DDR4 books (each with 8890v4 and 4x32GB 2400T on 1,6,9,10 slots) and am having issues booting into UEFI. I have no issues with the IMM. I suspect/hope it is a firmware issue. I currently have:

UEFI v4.50 (latest)
IMM2 v5.13 (not latest, but newer than the dependency for UEFI v4.50 )

but I keep getting CPU BUS FAULT errors on the front panel ("Uncorrectable Error has occurred on CPU's" in IMM Events) when the UEFI tries to boot.

I have also tried:

UEFI v4.50 (latest)
IMM2 v5.9 (latest)

but the system either hangs on UEFI boot or the same CPU BUS FAULT occurs. With older firmware (which the enclosure came with) the system hangs on UEFI boot (UEFI START PEI on front panel).

I have 8 equivalent compute books. I've tried using only 4 books (in the two leftward slots, top and bottom) as well as switching out different books to see if a book is faulty, but no change. I think this is the minimum configuration for the machine. No faults on the lightpath diagnostic are ever triggered, and I don't see any info in the IMM about which cpu the error occurred. I've also tried the two modes where the upper/lower nodes are unified or not (stand-alone vs. partition mode)

I've only used the IMM2 to update firmware, and update both upper and lower nodes equally (the XClarity software hasn't worked well with the machine so far).

Anyway, it might help if anyone can tell me the firmware/hardware configuration any of your working DDR4 machines are using, or if anyone has had this issue before. I fear I may need to buy another midplane and/or the two i/o boards to see if it is a hardware issue.

Thanks!

 

[RussianE39]

Check FRU numbers of the books you're using, I had exactly same issue with different part numbers of books in one server.

 

[chrgrose]

Check FRU numbers of the books you're using, I had exactly same issue with different part numbers of books in one server.

The good news: All of my Compute books are 00D0402. Both i/o books are 00FN850, and both storage books are 00D0055.

The bad news: I figured I might as well check the midplane FRU, and it is 00D0056. This is not the expected part for my 6241-AC4 enclosure. This means that the midplane must have been downgraded/replaced at some point... my understanding is that this midplane either cannot be used with the DDR4 books or at least the v4 CPU's. So this must be the source of the problem..

The really bad news: The correct/needed part FRU seems to be 00FN638. I only see a single part on ebay... from Germany... which with shipping would cost almost twice what I payed for the enclosure/io/storage books.

Man, I am sad

 

[ootronicsnazleaki]

My x3950 is running the DDR4 books as a 3837 AC1 with the (latest?) firmware.
I'll have to check to see what actual version it is, I'm not familiar with the machine in general yet, but I've been advised that the firmware makes it effectively a 6241 machine as that mostly designates the crossover from IBM to Lenovo, and not much else.

I'll try to post more info later.
One thing though is, the primary and secondary front storage books may need to be in the order they were configured to be, if you have torn it apart and put it back together.

 

[chrgrose]

My x3950 is running the DDR4 books as a 3837 AC1 with the (latest?) firmware.
I'll have to check to see what actual version it is, I'm not familiar with the machine in general yet, but I've been advised that the firmware makes it effectively a 6241 machine as that mostly designates the crossover from IBM to Lenovo, and not much else.

I'll try to post more info later.
One thing though is, the primary and secondary front storage books may need to be in the order they were configured to be, if you have torn it apart and put it back together.

Do you know what the part number is for your midplane? It is behind the top storage book. You also mentioned that you are running v3 xeons, which might mean that problem is the v4 xeons, not the DDR books. This is also consistent with the lenovo instructions here:

Lenovo System x3950 X6 (6241) Product Guide (withdrawn product)

The Lenovo System x3950 X6 server, machine type 6241, is an eight-socket 8U rack-mounted server that represents the sixth generation of the Enterprise X-Architecture. Now with Intel Xeon E7 v4 processors, the x3950 X6 delivers fast application performance, is based on an agile system design, and...

lenovopress.lenovo.com

• "Upgrades to E7 v4 processors are only supported with machine type 6241. If you want to upgrade an X6 server with machine type 3837, contact your local representative."

So the old midplane must be able to use DDR4 books, but for some reason v4 xeons are not supported.

 

[ootronicsnazleaki]

My FRU PNs:

Both Top & Bottom Storage Books: 00D0055
Midplane: 00D0056
8x Compute Books: 00D0402
Both Top & Bottom IO Book: 00FN847
4x Half Length IO Books: 00D0053

I see what you are saying there with the"Upgrades to E7 v4 processors are only supported with machine type 6241. If you want to upgrade an X6 server with machine type 3837, contact your local representative."

I had read that too and maybe I read it too fast, but I had formed the idea that it meant that you needed DDR4 books for V4 and that you cannot run DDR4 books in the 3837 without the firmware upgrade.
I could be wrong, re-reading it now it's not clear.
But I had intended to leave myself the ability to use V4 later, so hopefully that is still possible.


How did you come to this information? : "The correct/needed part FRU seems to be 00FN638."

The images of the midplane show its basically just a bunch of connectors on a PCB, not really anything else.
I don't see any SMD resistors, capacitors, or anything.
So if there were differences it would seem that it'd have to be in trace routing for tighter tolerances, or perhaps missing or added traces.

But since DDR4 books run the DDR4 ram in the 3837, and runs it at the speed faster than the DDR3 books can do, I think it's still very likely that you don't need that midplane PN specifically.
But the other bad news is, if your midplane is broke. The 00D0056 looks even more expensive on ebay.

Firmware:
Mine has
UEFI Pri:A9E164AUS
v5.30 Date:2022/09/14

UEFI Alt:A9E154AUS
v4.80 Date:2020/04/09

IMM Pri:TCOO60A
v5.90 Date:2021/07/21

IMM Alt:TCOO58B
v5.80 Date:2021/01/13


Possible try just to get past booting:
take out all ram except 1 dimm per book in the first slot that is to be populated (not sure if it's actually labeled 1 or if it's some other order)
update UEFI to v5.30

I did read somewhere that at least in my case for 3837, in order to update the firmware to use DDR4 books, you'd need to do that before installing the books, hence you'd need to have/borrow DDR3 books to perform the firmware upgrade.

 

[ootronicsnazleaki]

No, E7 wont work in E5 platform. E5 has internal memory controllers, while E7 has Scalable Memory Buffers instead of "raw" Memory Controllers.

Interesting, that makes sense. I got the idea from this post which they explored using an 46xx processor, looks like E5 now that I reread it.

https://forums.servethehome.com/index.php?threads/xeon-e5-4650-on-x9dri-ln4f-dual-socket-mainboard.25148/

I have a Proliant G8 which has V1 in it, if I put a V2 in it it could do 1866Mhz on the memory if cpu also does.
There are some E5-46xx cpus on ebay for good prices.

Makes me wonder though, I've noticed that the 4 socket and 8 socket machines do not get up there in RAM speed like the dual socket machines, maybe it's due to the Scalable Memory Buffers...
It does look like all the 4 socket and 8 socket machines from the different vendors are kind of all in orbit of a same reference machine.

 

[RussianE39]

Interesting, that makes sense. I got the idea from this post which they explored using an 46xx processor, looks like E5 now that I reread it.

https://forums.servethehome.com/index.php?threads/xeon-e5-4650-on-x9dri-ln4f-dual-socket-mainboard.25148/

I have a Proliant G8 which has V1 in it, if I put a V2 in it it could do 1866Mhz on the memory if cpu also does.
There are some E5-46xx cpus on ebay for good prices.

Makes me wonder though, I've noticed that the 4 socket and 8 socket machines do not get up there in RAM speed like the dual socket machines, maybe it's due to the Scalable Memory Buffers...
It does look like all the 4 socket and 8 socket machines from the different vendors are kind of all in orbit of a same reference machine.

If your proliant is DL560Gen8 - upgrading to E5-4657Lv2 is a great idea, v2 cpus are WAY better. From my experience, DL560Gen8 has such a poorly designed cooling system, it would overheat if you install four E5-4650 in it, it's actually one of the worst platforms for a quadsocket E5 enthusiasts (Although, quadsocket E5 is also very bad for general computing (niche use cases) comparing to a E7 ones). If you have 2socket machine (DL380 Gen8)- there is absolutely no reason paying extra for E5-4xxx series of CPUs. And yes, E7 has different memory organization and speeds because of SMB.

 

[MichalPL]

So the old midplane must be able to use DDR4 books, but for some reason v4 xeons are not supported.

This is my FV:


Model 6241, came with 8x E7-8880 v3 and 256GB DDR4 from Germany.
8880 v3 are super super slow, so replaced to 8x 8894v4 - and just worked, no issues at all (and upgrade to 2TB, 64dimms), no firmware upgrade.

---
for example upgrading HP DL580 G8 4x E7-8895v2 to "G9" 4x E7-8894v4 is quite problematic (still can do it below 1h) and require G9 SPI board (or complex flashing) so I believe same issues with x3950 x6 but in the "lenovo style"

 

[MichalPL]

Makes me wonder though, I've noticed that the 4 socket and 8 socket machines do not get up there in RAM speed like the dual socket machines, maybe it's due to the Scalable Memory Buffers...

It's not that bad, SMI almost double the speed.

E5 v0: 1600MHz factory
E5 v2: 1866MHz factory
E5 v2 overclocked (1650v2/1680v2): 2133MHz (Samsung 1866MHz DDR3 ECC)
E5 v3: 2133/2400MHz factory (~2666MHz overclocked DDR4)
E7 v2: 2666MHz (SMI)
E7 v4: 3200MHz (SMI2)

Trick is you need 2 dimms per SMI channel, so the memory is working at 1333MHz max in v2 and 1600MHz while connected via SMI2 to v4 CPU.
so E7 have 4 fast SMI channels (8 slower "DDR" channels - but 8 not 4 like E5).

 

[MichalPL]

v2 cpus are WAY better.

True, v0 was good up to... 4 cores (for example 1620 is almost identical like 1620v2)
6, 8 cores much better in v2 (1650 v2 is much faster than 1650 or 1660)

10,12,15 cores not available in v0.

 

[chrgrose]

The nightmare continues.

I have now replaced the midplane with the updated part # 00FN638, but the same problem persists.

I was doing more research and found that also the primary I/O books have CPU limits: Standard I/O book FRU dependencies - Lenovo x3850 X6 and x3950 X6 Systems - Lenovo Support US

So my part 0FN850 will not work with v4 Xeons. What a pain. In fact, after reading this the midplane might not have been the issue.

I also can't figure out which I/O books part numbers actually support v4 Xeons. I suspect that the part listed here (#00YA701) supports v4: Lenovo System x Documentation

but this one is expensive on the used market and I don't know if others can also work. For example, I can get part #00FN856 for cheap on ebay.

Does anyone have an x6 system with v4 Xeons, and knows what their Primary i/o book part number is?