EXPIRED HYVE Edge Metal G10 - Epyc 7642 - 250£/offer

[luckylinux]

Did anybody else experience DIMM Channel Issues ?

On 2 separate Servers populated with 16 x 32GB DIMMs each (RDIMM PC4-2666V), I get an entire Channel NOT detected neither in BIOS nor dmidecode.

In one Case it seems to be Channel B (2x32GB missing), in another Case it seems to be Channel F0 (2x32GB missing).

Sure some DIMMs might be bad (I tried to reseat them once already), but this seems to be a bit of a Coincidence.

In BIOS I set RAM Overclock to enabled and manually set 1333MHz as the target Frequency (so the DIMM will run at their rated Speed, i.e. 2x1333MHz = 2666MHz).

EDIT 1: reseating the affected DIMM (and also saving BIOS Settings forcing RAM Frequency to 1333MHz, i.e. 2x1333MHz = 2666 MHz Operation) seems to have fixed that on one Server (the one where Channel B was affected).

I still need to try on the other Server where Channel F was affected.

 

[Buraz39]

Hi, im looking for some memory speeds and sizes rdimm or lrdimm that definitely work on this specific motherboard, whatever you guys tested that boots with this mbo and cpu, thanks in advance

 

[Johan Kooijman]

Did anyone find a way to simply reboot these machines from OS? Mine goes to blank screen, fans 100% and just sits untill I do a power cycle.

 

[luckylinux]

Did anyone find a way to simply reboot these machines from OS? Mine goes to blank screen, fans 100% and just sits untill I do a power cycle.

No, same issue .

You need to remove Power.

Not sure if you can do that remotely with ipmitool using e.g. a Raspberry Pi (or whatever remote machine by sending the Command via ipmitool over the LAN Network).

Otherwise only Option is to have a Relay/Contactor/Smart Plug or possibly (better) a Controllable PDU.

 

[Cruzader]

Whenever i see a host getting stuck there on reboot i log on BMC and do the "reboot BMC" then both BMC and host come back up.
So guessing its BMC glitching that prevents it from booting back up.

Got 3-4 different firmware versions across the ones i got, its not all of them that this happends with but have not looked further at what version those that do tend to get stuck run.
If its just some versions doing it or not.

I did notice that they did not like lrdimm tho.
And with my luck 14tb out of 16tb in the 2 bladecenters i gutted was lrdimm...

 

[luckylinux]

Whenever i see a host getting stuck there on reboot i log on BMC and do the "reboot BMC" then both BMC and host come back up.
So guessing its BMC glitching that prevents it from booting back up.

Got 3-4 different firmware versions across the ones i got, its not all of them that this happends with but have not looked further at what version those that do tend to get stuck run.
If its just some versions doing it or not.

I did notice that they did not like lrdimm tho.
And with my luck 14tb out of 16tb in the 2 bladecenters i gutted was lrdimm...

Really ? I had no Issues with 8x64GB LRDIMM.

I didn't try with 16x64GB LRDIMM (yet ?) though.

EDIT 1: Complain about your "Luck". With the current RAM Prices you can still make a HUGE Profit out of those. I think I have less than 8tb of DDR4 .

EDIT 2: These RAMs I tested and work

LRDIMM: 8 x Micron DDR4-RAM 64GB PC4-2400T ECC LRDIMM MTA72ASS8G72LZ-2G3B2

RDIMM: 8-16 x SAMSUNG DDR4-RAM 32GB PC4-2666V ECC RDIMM M393A4K40CB2-CTD6Q or CTD6Y or CTD7Y etc

EDIT 3: @Cruzader isn't it maybe an Issue with EPYC not liking SK Hynix RAM ? I remember reading something about it before at least ...

 

[Cruzader]

EDIT 3: @Cruzader isn't it maybe an Issue with EPYC not liking SK Hynix RAM ? I remember reading something about it before at least ...

Might be, it was all from blades bought at the same time with same model/revision but i dont remember what brand it was.

Took 4-5 attempts to get a boot and would freeze after sitting just 5-10minutes with proxmox booted.
Consistently across the 4 i had on the workbench prepping then.

DIT 1: Complain about your "Luck". With the current RAM Prices you can still make a HUGE Profit out of those. I think I have less than 8tb of DDR4 .

Amusingly the 2 bladecenters are just jammed stuck in the racks in one of our DCs, the replacement hardware above both were left with all its weight down on them.

Otherwise they would have been thrown away with the 16tb in them a long time ago.

We did not need the space and nobody wanted to start shifting the weight above so they were just left there unplugged full of blades.
I pulled the blades and the chassises are still sitting there, probably will be intil we replace racks.

 

[luckylinux]

Might be, it was all from blades bought at the same time with same model/revision but i dont remember what brand it was.

Took 4-5 attempts to get a boot and would freeze after sitting just 5-10minutes with proxmox booted.
Consistently across the 4 i had on the workbench prepping then.


Amusingly the 2 bladecenters are just jammed stuck in the racks in one of our DCs, the replacement hardware above both were left with all its weight down on them.

Otherwise they would have been thrown away with the 16tb in them a long time ago.

We did not need the space and nobody wanted to start shifting the weight above so they were just left there unplugged full of blades.
I pulled the blades and the chassises are still sitting there, probably will be intil we replace racks.

I didn't try Proxmox, just Ubuntu (or was it Debian, I cannot remember).

Could be you are one of those "rare Cases" as Proxmox Forum Staff usually replies, who experience an Issue they cannot replicate (and is probably related to Ubuntu Kernel and/or one of their Patches on top of it).

 

[Johan Kooijman]

For what it's worth.. on my boxes proxmox was stable. Which makes sense since it's simply debian 13 + ubuntu kernel.

 

[luckylinux]

For what it's worth.. on my boxes proxmox was stable. Which makes sense since it's simply debian 13 + ubuntu kernel.

"Simply". Way too many Ubuntu Kernel Bugs seen on Proxmox.

Sure, they don't always affect the same Users, but they DO happen.

When everything else fails, running the Debian Kernel usually fixes the Issue (even though it's not supported / recommended, it's not like they give Users any Alternative).

 

[Johan Kooijman]

Found a way around the "won't come back after reboot issue".
At least for my Proxmox 9 installed machines.

When you add "reboot=pci" to you grub config, /etc/default/grub on Debian & Ubuntu, and update your grub files, you get a nice reboot.