Epyc 9755 ES and MZ73-LM2 problems

[terryww]

The 2500W PSU is new. Ok, so that brings us back to transient load ChatGPT mentioned. It says server PSUs are designed to handle 2x specified amps for a few ms so 2 socket systems can power up without triggering the PSUs over current protection stuff. That's the only thing left to test - find 2x 1600W CRPS PSUs with a backplane that can handle +1600W (and hopefully the power load is distributed between the 2 PSUs).

 

[RolloZ170]

Ok, so that brings us back to transient load ChatGPT mentioned

no, i wrote the PSU is bad.
( or the cables are too thin whatever something is wrong)

 

[terryww]

Hm... the thing this happened before with a used LC Power 1600W (130A@12V) and now also with a 2500W PSU (200A@12V). Or perhaps sth with the motherboard, but 1 CPU worked so... no idea. And it's expensive to test with diff HW :/

 

[RolloZ170]

Hm... the thing this happened before with a used LC Power 1600W (130A@12V) and now also with a 2500W PSU

ok your point.
i have no idea now.

 

[terryww]

Tnx for all the help. I'll try to find a used server PSU and see.

 

[RolloZ170]

btw: the CPU does not eat direct +12V. there is a VRM unit which converts +12V to Vcore( 0,6 - 1.3V)
and those VRM controllers have a soft start to prevent high current transitions.

 

[TrashMaster]

ASRock Rack ? you can kill them also from your list.
TURIN2D24G-2L+/500W proprietary form factor. i doubt you can buy a bare mobo.
no BIOS to DL. you have to beg for it. they are late with motherboards and for that reason almost no ES support.

I have that exact board. These guys get them in batches of 10 for around 1200 and you can use standoffs to build an adapter out of lexan.

https://www.tech-america.com/item/asrock-mb-turin2d24g-2l-500w-soc-socket-sp5-lga6096-max512gb-ddr5-proprietary/turin2d24g-2l-500w

If someone needs a specific bios hit me up i have a few saved BUT only the latest bios seems to work well with TURIN chips. Genoa work great in it though.

You do need to buy custom power cables and I can send you a link to that as well if you want to rock an ATX power supply. The board doesnt use standard EPS power, its got those microhi connectors asrock rack likes.

Also I ran my 9334 genoa ES pair in it for a while and worked great on any bios.

 

[RolloZ170]

The board doesnt use standard EPS power

it is a std. but not consumer class.

 

[TrashMaster]

it is a std. but not consumer class.

ATX 12V Power Connector 8 Pin Power Cable for ASRock Rack and Seasonic is what I bought for my PSUs, they just call it the "ASRock Rack proprietary special 8-pin" and worked perfect for me.
Others do use/reference the connector but I am not aware of the actual standard doc for the specs.

 

[RolloZ170]

they just call it the "ASRock Rack proprietary special 8-pin" and worked perfect for me.

...Dell, HPE and and and use that. not proprietary,
Micro-Hi is successor/improvement of Molex Micro-Fit

 

[TrashMaster]

...Dell, HPE and and and use that. not proprietary,
Micro-Hi is successor/improvement of Molex Micro-Fit

Well, for anyone who comes along later and reads this, Dell especially is famous for using a connector that fits existing cables, but changes the pins/voltages to suit their needs. Like this little Dell jbod controller that hijacks a 12v pin and expects 5v on it. Just because the cable fits does not mean plugging it in wont release magic smoke.

 

[RolloZ170]

Like this little Dell jbod controller that hijacks a 12v pin and expects 5v on it. Just because the cable fits does not mean plugging it in wont release magic smoke.

that is ATX 4Pin instead of ATX24 used by ASRock,Supermicro and others.
my SPC741D8UD-2TX550 uses that too.

 

[terryww]

I have that exact board. These guys get them in batches of 10 for around 1200 and you can use standoffs to build an adapter out of lexan.

https://www.tech-america.com/item/asrock-mb-turin2d24g-2l-500w-soc-socket-sp5-lga6096-max512gb-ddr5-proprietary/turin2d24g-2l-500w

If someone needs a specific bios hit me up i have a few saved BUT only the latest bios seems to work well with TURIN chips. Genoa work great in it though.

You do need to buy custom power cables and I can send you a link to that as well if you want to rock an ATX power supply. The board doesnt use standard EPS power, its got those microhi connectors asrock rack likes.

Also I ran my 9334 genoa ES pair in it for a while and worked great on any bios.

Thanks for the link! Couldn't find it with a quick search. Also surprisingly affordable at $1235. The Gigabyte LM2 was $1300 from China. Wonder if Zen5 ES CPUs would work at all. Can I ask what PSU you're using?

 

[TrashMaster]

Thanks for the link! Couldn't find it with a quick search. Also surprisingly affordable at $1235. The Gigabyte LM2 was $1300 from China. Wonder if Zen5 ES CPUs would work at all. Can I ask what PSU you're using?

I am running 5 Seasonic 1600w PSUs (mix of platinum and titanium) for the inference rig. Usually I apply a pretty hefty safety factor to them, and localize groups of cards electrically (like 4x3090 and their risers together) or (2 x blackwell cards and their risers together) I dont cross PSUs between groups of components and have different 20 amp circuits feeding different PSUs. I am actually in the process of re-designing my aluminum structure that holds all this stuff in order to shrink MCIO lengths (my 80cm cables are borderline with gen5) to fix some AER stuff or instances where I have to force pcie gen5 cards down to gen4.

Also, sorry to the haters but I might be the reason why a certain SKU of ddr5 rdimms, or 1600w PSUs are extra high on ebay. Every time one hit my 'buy' price, or somebody accepted my offer, i snaked it.

 

[RolloZ170]

Wonder if Zen5 ES CPUs would work at all.

there are some ebay listings with MZ73-LM2 + 2x 9755 ES like yours.

 

[terryww]

I am running 5 Seasonic 1600w PSUs (mix of platinum and titanium) for the inference rig. Usually I apply a pretty hefty safety factor to them, and localize groups of cards electrically (like 4x3090 and their risers together) or (2 x blackwell cards and their risers together) I dont cross PSUs between groups of components and have different 20 amp circuits feeding different PSUs. I am actually in the process of re-designing my aluminum structure that holds all this stuff in order to shrink MCIO lengths (my 80cm cables are borderline with gen5) to fix some AER stuff or instances where I have to force pcie gen5 cards down to gen4.

Also, sorry to the haters but I might be the reason why a certain SKU of ddr5 rdimms, or 1600w PSUs are extra high on ebay. Every time one hit my 'buy' price, or somebody accepted my offer, i snaked it.

I wonder how you connected the PSUs. Is it like you use 1 PSU for powering the motherboard + CPUs and then other, chained together for GPUs? So if you can power 2 CPUs with your PSU, I assume in my case it's not about power stepping (i.e. an ATX PSU is fine). Which leaves the other option ChatGPT mentioned - transient currents being higher than what my 2500W PSU offers (200A@12V) and potentially this triggers the PSU's over current protection circuit. Or sth on the motherboard.

Have you thought about buying a case that supports 8-10 GPUs? Sth with 4x 2000W and an old motherboard for $1700? So using this motherboard and finding the right PSU + power cables, the only thing left is a rack case for a killer ML setup. But haven't seen cases for less than $1k. I assume the cheapest DYI option would be to just find a big case that allows placing the GPUs face (where the GPU vents are) down, remove the GPU fans, stack GPUs without the fans on top of each other and use the big case fans for cooling.

 

[DanRR]

Have you solved your problem with the power supply and other issues?

 

[terryww]

Have you solved your problem with the power supply and other issues?

No. Still searching for a server PSU to test instead of using Cannon 2.5kW. But it's strange because other users report it's working fine with ATX PSUs (a company making workstations is using Super Flower ATX 1600W), another reported that it worked for him with Asrock PG1600 PSU, so searching for +2kW server PSU with right power cables.
Also not sure anymore it's the PSU as the last OCPIOK LED starts blinking when the PSU shuts down after spinning the fans for 2s. Which I assume means the motherboard signals to the PSU to shut down (so perhaps not a PSU issue). If the PSU would trigger it's over current protection circuit, I guess it would just shut down without the motherboard noticing anything. So perhaps the second CPU socket is having power problems.

 

[DanRR]

Indeed, the power supply is the hell problem.
Does your power supply have regular 12V cables or it has these new 12+4pin ones what this motherboard does not have? Do we have to use adapters to connect? I just bought SuperFlower like the other guy here (Venturi) and found no information what is supplied via regular 12V cables and what via these new ones. The supplied "manual" or their website is a total garbage with strange drawings showing cables with 3 rows and columns of pins. Do not like to ruin my money, the board, processors and PSU itself connecting them blindly

 

[terryww]

Tnx for confirming, have thought I'm going crazy with thinking PSU would be an issue. What's your experience?
I guess you're talking about the new PCIe power connector - 12VHPWR. So the PSUs I tried, if they had 12VHPWR, it would be the same connector on both sides so you can't plug it in into the 3x6pin power sockets on the motherboard. I'd always plug in PCIe 6pin in there (and for modular PSUs connect that to the output labeled VGA). But yeah, the motherboard's manual does share the pin layout (page 20) and from what I checked, it is the standard PCIe 6pin power cable pinout.

Having a quick chat with grok.com, the power draw doesn't add up:
- 24pin has max draw of 300W
- 2x 8pin for CPU, max 300W each
So the 6pin PCIe power sockets I guess are mostly used to power PCIe slots. It looks like any PSU with 2 8pin CPU + 1x PCIe power cables should be able to handle it. So what's left is transiatn current and power stepping handling on the PSU's side (according to AI). But it's hard to find a server PSU supplier and delivery dates for Alibaba are like 3 months.