EPYCD8-2T: two questions about Linux and IPMI

balnazzar

Member
Mar 6, 2019
105
7
18
If you start/run your dockers with --restart=always -- google for docs, then they will restart automatically on boot. I think this is more easy to solve than suspend/resume on *server* board.
It seems you are telling me that server boards often experience issues with suspend/hibernate.. I wasn't aware of that but mine is my first server board..
Yes I could restart the containers automatically at every reboot, but think about the models, the jupyter notebooks inside these containers..

Ultimately I think I can live with restarting everything every time, but it will be boring, and it makes using the ipmi via the internet even more important, since not being able to suspend would equate not to being able to use wake on lan...

Thanks, anayway.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Mates, apologies in advance if I try and push my other issue a bit.

Suspend and hibernation just don't work on my setup, and that's truly a ballbreaker, since I run a lot of dockers and cannot restart all of them every time I need to power off the machine.

I tried several linux distributions (strarting from debian stable to ubuntu 20.04 test), compiled even the most recent kernel manually and optimizing for AMD (EDAC, etc etc), and tried even the userspace suspend (`uswsusp`). Nothing. The screen blanks, and that's all. The system refuses to power off.

It is very awkward, since it is rock stable under any load with every tested distro. Furthermore, prior to purchasing the epycd8-2t, I read that review: ASRock Rack EPYCD8-2T Makes For A Great Linux/BSD EPYC Workstation - 7-Way OS AMD 7351P Benchmarks - Phoronix

It says "These days any Linux distribution released in the past year or two is working fine with AMD EPYC processors. I personally tested this ASRock EPYCD8-2T with Fedora Workstation 29, CentOS 7, Debian 9.8, Clear Linux 27910, and openSUSE Leap 15.0. The experience was pleasant and without any issues to report on the Linux side."

I'm pretty sure I am not setting the BIOS properly. If any of you has the epycd8-2t with linux, could suggest something?
I wish I could help with this but I honest I have little experience with suspend/hibernation in Linux. I do not suspend/hibernate my servers. They run 24/7.
 

balnazzar

Member
Mar 6, 2019
105
7
18
I wish I could help with this but I honest I have little experience with suspend/hibernation in Linux. I do not suspend/hibernate my servers. They run 24/7.
Thanks anyway ;-) Any special setting you would suggest for rome/epycd8 with Linux? Both BIOS, OS, and kernel compilation flags?
 
Last edited:

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Thanks anyway ;-) Any special setting you would suggest for rome/epycd8 with Linux? Both BIOS, OS, and kernel compilation flags?
There's was much I had to change in my BIOS. The only things I changed are:

  1. Set OnBoard Video to Primary (so that IPMI video works even with add-on GPU's installed)
  2. Enable IOMMU (for virtualization support)
  3. Configure cTDP as seen in the below screen shot (this is set for better performance, but you will sacrifice some power efficiency. This is a personal choice)
 

balnazzar

Member
Mar 6, 2019
105
7
18
Thanks, I already did point 2. For point 3, I'd retain a more efficient setup, even at the cost of some performance.. That was the point of 7282.

Regarding point 1, suppose that I use a physical monitor attached to the onboard VGA output, and such monitor has a certain resolution. Will that resolution be used even when I redirect the video output via the IPMI remote control?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Thanks, I already did point 2. For point 3, I'd retain a more efficient setup, even at the cost of some performance.. That was the point of 7282.

Regarding point 1, suppose that I use a physical monitor attached to the onboard VGA output, and such monitor has a certain resolution. Will that resolution be used even when I redirect the video output via the IPMI remote control?
Couldn't answer that. I haven't had a monitor hooked up to any of my servers going on for many years now.
 
  • Like
Reactions: balnazzar

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,266
428
83
Harking back to earlier, it's very rare for server motherboards to support S3 suspend/S2R and from a quick look at the manual, the Epyc D8 boards don't.

Hibernate/S2D might be a different matter as IIRC it's significantly easier to support than S2R but there's no mention of support for that in the manual either.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
105
7
18
Harking back to earlier, it's very rare for server motherboards to support S3 suspend/S2R and from a quick look at the manual, the Epyc D8 boards don't.

Hibernate/S2D might be a different matter as IIRC it's significantly easier to support than S2R but there's no mention of support for that in the manual either.
Thanks for your reply. The manual is rather incomplete, as often happens with motherboards manuals, sadly.
That being said, I really wonder what that "soft shutdown" mentioned in the IPMI really is.

I'm trying different combintions of sleep/hibernate commands (e.g. uswsusp - ArchWiki) but with no success... One could say that I bought a server board, something explicitly thought to be on 24/7, but then I could reply that there is nothing available on the workstation market segment that supports cpus with more than 64 lanes.
 

TXAG26

Active Member
Aug 2, 2016
274
81
28
Soft shutdown in the bios refers to the power button sending a shutdown request to the underlying operating system (Windows/Linux), which, if it works, promptly has the OS orderly shutdown. The other bios option immediately cuts the power when the power button is pressed.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
105
7
18
I tried it (it's in the IPMI management interface, not in the BIOS). It actually shuts it down (presumably forcing some suspend, since if I power on subsequently, the OS won't awake.. I got to hard reset)
 

TXAG26

Active Member
Aug 2, 2016
274
81
28
I think I would move past trying to get suspend to work on a server grade board. It's usually a crap-shoot trying to get that to work on desktop motherboards that claim to fully support it. Most server boards likely would not support a true suspend feature, as there would always likely be at least one peripheral that kept it from cleanly suspending.
 
  • Like
Reactions: balnazzar