Does anyone have Supermicro H12SSL series motherboard experience?

[mattlach]

Anyway, in order to validate that everything was working the way it should, I started googling around for known benchmark numbers I could replicate on this thing in its benchtop configuration (without installing Windows)

All I could find was a launchtime Geekbench 4 result (I thought geekbench was for phones?) with a multicore score of ~116k. Well I ran mine and got ~128k, so I am going to call that a success. I'm going to guess that the tests at launch were run with slower memory, not the 3200MT/s stuff I have.

If anyone is curious:
Supermicro Super Server - Geekbench

Then just for shits and giggles I ran a Geekbench6 intending to compare it to other results of the same CPU in their browser. I landed on a multicore score of 17139, which seems about right, but it is tough to tell, because the benchmark browser has results all over the place in it.

Again, if anyone is curious:
Supermicro Super Server - Geekbench

Mine lands among the higher results for a single Epyc 7543, so I'll take that as an indication that at least I wasn't scammed, and I did indeed get the RAM and CPU I was supposed to. There were some that were a couple of hundred points higher, but it looks like they were running some sort of Asus workstation board. I'm guessing they had a bigger cooler than this little 92mm thing, and benefited from better boost clocks.

Now I am going to run Memtest. That's probably going to take... ...a while.

I've never done a memtest with this much RAM before.

 

[sth]

Sorry, forgot to quote for context. @mattlach asked, "Mind sharing what you did wind up using that worked for you?"

Micron 7450 NVMe 1TB I recall.

 

[mattlach]

Now I am going to run Memtest. That's probably going to take... ...a while.

I've never done a memtest with this much RAM before.

Heh, yeah, 10 hours and 20 minutes in, and we are at 48% complete for one pass

 

[sth]

Is that the pro version?

 

[mattlach]

Is that the pro version?

This is whatever version was included on a Linux image I had handy on a USB stick I had already written. It was Linux Mint 21.2 from a previous install. I'm guessing it must be the open source version?

Though looking at the version number (5.31b) this is not the new Version 6 that was recently released for the first time in forever.

The closed source Pro version you linked mentions ECC error injection, which might be important. The open source version does not mention ECC or Error Correction at all from what I can tell.

I'm really at the limit of my understanding of the memory pipeline and the capabilities of both tools here. I do vaguely remember there being news reports years ago about one version that should not be used anymore, and one version that was good, but I honestly cannot remember which was which, and I can't seem to find those stories in my googling right now.

While I generally avoid commercial software like the plague, if the commercial version offers a functionality I need, I may have to use it. I spend most of my time in consumer client land, so I don't know if the addition of ECC adds a wrinkle here. I'd imagine some problems may be masked by the ECC doing its job, but I am not quite an expert here. Appreciate any thoughts on the matter.

 

[sth]

That was my understanding too. Given the cost of $1k of RAM and the time/energy consumed in testing it, I went with the pro version which was a small bump in overall system cost for peace of mind. Also justified it yo myself because i've used memtest for years and was good feelz to give back. I noticed when researching my system a pretty noticeable number of Epyc stability issues seemed to be RAM related too.

 

[mattlach]

That was my understanding too. Given the cost of $1k of RAM and the time/energy consumed in testing it, I went with the pro version which was a small bump in overall system cost for peace of mind. Also justified it yo myself because i've used memtest for years and was good feelz to give back. I noticed when researching my system a pretty noticeable number of Epyc stability issues seemed to be RAM related too.

Since my last post I found this writeup by Passmark (the authors/owners of the closed version of Memtest86)

It seems to suggest that ECC Error injection is really meant for the engineering of CPU's memory controllers, motherboards and memory modules, not for users, either consumer or enterprise. It seems like it won't hurt, but that it is largely unnecessary.

In addition to just writing patterns to RAM and reading them back to ensure they are the same, Memtes86 apparently reads the memory controller which will log ECC errors, both corrected and uncorrectable as well, to cover testing of ECC RAM.

So, the free version should suffice for our purposes here, unless you are doing something way more fancy than I am

All of that said, it is unclear if the open source version (memtest86+) that I used has the ability to do this. It presumably does, but it would need to understand the system chipset used, so it can read the ECC data. Judging by my screenshot (duplicated here) it says "Chipset: Unknown" and "Memory Type: Unknown" which suggests to me that the version I am using can't read the ECC data from the memory controller in the Milan Epyc's



The new Version 6 might be able to. I don't know.

This could also be because I failed to quickly hit F2 right after launch, so the test defaulted to a "safe mode" test, where it disables SMP and just runs on one core. I have read lots about how if you hit F2 it will use all cores and run MUCH faster but I wonder if this "safe mode" also causes it to not probe the chipset/RAM.

At this point I am ~13.75 hours into a ~20 hour test. I think I am going to let this test finish, and then try again, first by hitting f2 with the open source version (maybe even version 6, if I can easily find it and write it to a USB stick)

I may also try the closed source version by PassMark to see if it is any different.

I wonder how well the SMP aware version scales compared to running on a single core. If it scales linearly with core count (unlikely, very few things ever scale linearly), this 20 hour test could be a 38 minute test next time


Also, can I just say that the naming is really confusing here. For two different projects that look very much the same and do very similar things, authored by different people, one commercial, the other open source for one to be called Memtest86 and the other to be called Memtest86+ seems like it is just asking for there to be confusion.

Edit:

Alright, I resorted to wikipedia for some clarity. Let me paraphrase/summarize.

Unsurprisingly Memtest86 was first, and Memtest86+ came later.

The original was released in 1994 under an open source license, but after hit hit version 3 in 2002 the one man project kind of died, and the release stayed unchanged for a few years. At that point, another person forked the code to create memtest86+ to continue the work and add support for new chipsets and CPU's. It was maintained until version 5 in ~2013 then it also died and became stagnant.

At that point Passmark bought the rights to the original Memtest (the non-plus version) and brought it current.

Passmarks version reportedly does not utilize the positron-independent code of the original, and can thus not test ALL of the RAM like the original could. I can't seem to confirm whether or not Memtest86+ does or does not do this.

And that's the way it was until 2022, until someone out of the blue (or the original maintainer, not sure) revived Memtest86+ and launched memtest86+ version 6, and more recently 6.2


I'm not sure if that summary leaves me more or less nonplussed than I was before (pun definitely intended)

I think I am going to create a Memtest86+ 6.20 usb stick and test it in SMP mode next. Hopefully it will identify the chipset and RAM.

But being almost 14 hours in now, I am going to let this version finish first. I wish I had done my research before I started the first one...

 

[mattlach]

Running with Memtest86+ 6.20 in SMP mode.

The newer version does seem to recognize my hardware. SMP seems to be the default now, so pressing F2 on start disables rather than enables SMP.

SMP mode thus far just seems to mean it alternates cores for the test, presumably to test the reliability of the links between all DIMM's and all cores instead of just one. It does not appear to run them in parallel.

That said, at least initially the test seems to be running MUCH faster. In just a few minutes we are up to 11%. The address ranges are also zipping by much faster.



In addition to running faster, the CPU also seems to run hotter. (the CPU cooler is noticeably louder) so I do think it is actually doing things faster, as opposed to just running a slower and somehow less stringent test, though I can't know for sure

I'll let this one go overnight, and then do PassMarks latest free version as well, jsut for comparison purposes.

 

[mattlach]

Memtest86+ 6.2 completed a pass in about 5 hours, compared to the 22 or so hours the old version took in fail-safe mode:



Going to try PassMarks version as well, just for shits and giggles.

 

[mattlach]

Well, meh. The PassMark image does not appear to be bootable for me.

I've tried writing th eimage with both the "USB Image Writer" tool in Linux Mint, and with dd, and neither produce a bootable USB stick.

No time to troubleshoot now. Maybe have to use Windows for this? The Readme has instructions for linux, but they don't appear to work.

Edit:

Oh never mind. I figured it out. It's an EFI boot, whereas I had to force the board into traditional boot for Memtest86+

 

[mattlach]

And we are off. Sure looks a little different:



It is also a much heavier image. 1.1GB vs the 6.2MB of Memtest86+

It does appear to have more in the way of specific system information though.

I just ran it with defaults. Looks like it only uses 16 cores by default.

It also seems faster yet than the latest Memtest86+

 

[mattlach]

Happened to glance at the BMC before leaving for work. Glad I did. RAM temps were getting shockingly high using Passmarks Memtest98. BMC was alerting.

DIMM's A-D were hitting 72C.

I figured an open air test was fine just for a RAM test, but apparently not.

Got some airflow over the RAM now in a very crude manner, lol




With the lowest setting on the blower they are down to 40C again.

I guess that's my daily reminder that server parts are designed with constant airflow in mind.

 

[mattlach]

Also worth noting.

Per my Kill-A-Watt, power at the wall was ~150W with the old Memtest86+, about 178W with the version 6.2 Memtest86+ and hovers between 200 and 250W with the PassMark version, so it is definitely hitting it harder, and it shows. it's tearing through those passes.

So, yeah, temps definitely went up with the PassMark version.

 

[mattlach]

So,

It seems like the PassMark version tears through the first ~half of each pass, and then gets slower after that, with each pass completing after ~11.5 hours on my system with 512GB of registered DDR4-3200

By default it seems it runs 4 passes. I imagine there must have been some sort of rationale behind that choice, so I am going to stick with it and complete 4.

In order to make my office usable in the meantime though I had to get rid of that noisy blower fan.



This is much easier on the ears.

RAM temps did go up by about 16C over the blower though! Those things sure do move a lot of air!

 

[RolloZ170]

It seems like the PassMark version tears through the first ~half of each pass, and then gets slower after that

sure, as described in the readme/instructions.
it helps to find issues, if you get errors in the first quicker pass, no need to check further.
you can disable this in the cfg file.

#do full first pass
PASS1FULL=1
# nr. of passes
NUMPASS=2

 

[mattlach]

sure, as described in the readme/instructions.
it helps to find issues, if you get errors in the first quicker pass, no need to check further.
you can disable this in the cfg file.

#do full first pass
PASS1FULL=1
# nr. of passes
NUMPASS=2

In my experience the first and second passes took about the same amount of time. In each pass, I found that the first half of the pass went by rather quickly, and the second half of the pass took rather long.

So I didn't find that the first pass was faster.

 

[RolloZ170]

In my experience the first and second passes took about the same amount of time. In each pass, I found that the first half of the pass went by rather quickly, and the second half of the pass took rather long.

So I didn't find that the first pass was faster.

the memtest86 pro version i have use PASS1FULL=0 as default,
maybe your version uses PASS1FULL=1

from the manual

Specifies whether the first pass shall run the full or reduced test. By default, the first pass
shall run a reduced test (ie. fewer iterations) in order to detect the most obvious errors as
soon as possible.
0 – Reduced test (default)
1 – Full test

 

[mattlach]

the memtest86 pro version i have use PASS1FULL=0 as default,
maybe your version uses PASS1FULL=1

from the manual

Specifies whether the first pass shall run the full or reduced test. By default, the first pass
shall run a reduced test (ie. fewer iterations) in order to detect the most obvious errors as
soon as possible.
0 – Reduced test (default)
1 – Full test

Yeah, I had read that it would do this by default too, but that just didn't seem to be the case for me. Each pass seems to take roughly 11.5 hours.

I didn't go into the configuration to mess with the settings though. I just let it run at default.

While I have decades of experience with both enterprise and consumer hardware, I figured that I don't know enough about the particulars of RAM, memory channel, and controller design to intelligently pick which settings to run to perform an adequate test. So I went ahead and assumed that people who know much more than me came up with the defaults based on some sort of reasoning, so I ran with those.

 

[mattlach]

the memtest86 pro version i have use PASS1FULL=0 as default,
maybe your version uses PASS1FULL=1

from the manual

Specifies whether the first pass shall run the full or reduced test. By default, the first pass
shall run a reduced test (ie. fewer iterations) in order to detect the most obvious errors as
soon as possible.
0 – Reduced test (default)
1 – Full test

Yeah, I had read that it would do this by default too, but that just didn't seem to be the case for me. Each pass seems to take roughly 11.5 hours.

I didn't go into the configuration to mess with the settings though. I just let it run at default.

While I have decades of experience with both enterprise and consumer hardware, I figured that I don't know enough about the particulars of RAM, memory channel, and controller design to intelligently pick which settings to run to perform an adequate test. So I went ahead and assumed that people who know much more than me came up with the defaults based on some sort of reasoning, so I ran with those.

So, I may have to take this back. The times for each cycle have increased. It's just not by as much as I would have expected, so it didn't show up as obvious in my early calculations, and it is thrown off by the progress indicator definitely not being linear with respect to time.

05 -> ~58% goes really fast, then it sits at 58% forever, and then it speeds up to like 92% where it sits for a VERY long time, so if I try to make an estimate at either 58% or 92% based on time passage, the numbers are all over the place.

Here is where I am now:



Interestingly it seems to have misidentified my system as having 16 channel RAM, when in reality it is 8 channel.

 

[mattlach]