EPYCD8-2T: two questions about Linux and IPMI

balnazzar

Member
Mar 6, 2019
104
7
18
Hi, I just built a system with a 7282 and an Asrock Rack EPYCD8-2T.

It seems to be rock solid with Lubuntu 18.04 but here are a couple of question I really hope you could answer since I'm driven crazy with them.

1. OS-independent: I just cannot manage to have the IPMI working over the internet, and the manual (http://asrock.pc.cdn.bitgravity.com/Manual/IPMI/EPYCD8-2T.pdf) is not helping me.
As you may know, the board had a dedicated management NIC with its own IP. There is no problem reaching it from inside the LAN just by entering the NIC ip in the browser (default port 80). If I try to reach it from the internet it refuses to connect, but port forwarding on the router works perfectly, since I reach other services on this very machine, all of them listening upon the regular 10G X550 NICs.
Any clue? o_O

2. Linux-related: As I said the board seems to be rock solid with Linux.. But suspension just doesn't work.. If I try and suspend, the screen blanks out, but the machine does not power off. If I hit a key, nothing happens. If I push the power button, it "wakes up", in the sense that I am presented with my system as I left it.
Maybe someone has a similar config with linux and can help me.

I hope this is the right section to post such kind of questions, but should I be wrong, I kindly ask the moderators to move my post to more appropriate sections.

Thanks!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,298
553
113
Hi, I just built a system with a 7282 and an Asrock Rack EPYCD8-2T.

It seems to be rock solid with Lubuntu 18.04 but here are a couple of question I really hope you could answer since I'm driven crazy with them.

1. OS-independent: I just cannot manage to have the IPMI working over the internet, and the manual (http://asrock.pc.cdn.bitgravity.com/Manual/IPMI/EPYCD8-2T.pdf) is not helping me.
As you may know, the board had a dedicated management NIC with its own IP. There is no problem reaching it from inside the LAN just by entering the NIC ip in the browser (default port 80). If I try to reach it from the internet it refuses to connect, but port forwarding on the router works perfectly, since I reach other services on this very machine, all of them listening upon the regular 10G X550 NICs.
Any clue? o_O

2. Linux-related: As I said the board seems to be rock solid with Linux.. But suspension just doesn't work.. If I try and suspend, the screen blanks out, but the machine does not power off. If I hit a key, nothing happens. If I push the power button, it "wakes up", in the sense that I am presented with my system as I left it.
Maybe someone has a similar config with linux and can help me.

I hope this is the right section to post such kind of questions, but should I be wrong, I kindly ask the moderators to move my post to more appropriate sections.

Thanks!
What port are you forwarding to access via the Internet. And personally, I would not recomment making you server accessible over the Internet without a VPN.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
What port are you forwarding to access via the Internet. And personally, I would not recomment making you server accessible over the Internet without a VPN.
Thanks for your reply. You are totally right about security concerns.. But once I manage to have it working, I wanted to use SSL certificates and ditch the user/pw authentication. Other options would be ssh tunneling. A vpn seems to be overkill, but I may be wrong or too trusty towards mankind..

I just forwarded port 80 on the router. The IPMI is listening on that port, as confirmed by the fact that I reach it flawlessly from inside my lan just by entering the IMPI IP (which is 192.168.1.4) in the browser (that is, without specifying the port by IP:))port).

EDIT: I left the ipmi/bmc settings on their default. Looking in settings, the "web" service says that it's configured over the interface "bond0". Now I am a total noob, but I thought that nic bonding was a manner of teaming them to improve perfomances.. Why is the impi nic bonded with something else..? Did you have the same config as default in your epycd8?
Thanks!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,298
553
113
Thanks for your reply. You are totally right about security concerns.. But once I manage to have it working, I wanted to use SSL certificates and ditch the user/pw authentication. Other options would be ssh tunneling. A vpn seems to be overkill, but I may be wrong or too trusty towards mankind..

I just forwarded port 80 on the router. The IPMI is listening on that port, as confirmed by the fact that I reach it flawlessly from inside my lan just by entering the IMPI IP (which is 192.168.1.4) in the browser (that is, without specifying the port by IP:port).

Thanks!
Is it not redirecting to port 443 though? Are you getting a cert error warning when you access it locally?
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
Is it not redirecting to port 443 though? Are you getting a cert error warning when you access it locally?
I tried to connect to 443, and it just refuses. No cert error. Mind that the port to connect with certs should be 4433 (labeled as mutual port), while 443 is listed as secure port and *maybe* it could be for ssh tunneling?

EDIT: I just changed the non-secure port to 7777 just to make an attempt... It doesn't work. It continues to listen upon port 80. I rebooted, opened the ipmi management, the port is correctly set on 7777, but the ipmi keeps to be available just on port 80 from inside the lan.
I'm starting to think I have a bugged ipmi.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,298
553
113
I tried to connect to 443, and it just refuses. No cert error. Mind that the port to connect with certs should be 4433 (labeled as mutual port), while 443 is listed as secure port and *maybe* it could be for ssh tunneling?

EDIT: I just changed the non-secure port to 7777 just to make an attempt... It doesn't work. It continues to listen upon port 80. I rebooted, opened the ipmi management, the port is correctly set on 7777, but the ipmi keeps to be available just on port 80 from inside the lan.
I'm starting to think I have a bugged ipmi.
If you put in https://ipaddress it doesn't work? Very strange.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
https uses 443 by default, it won't show the port after the IP address. I would try and forward port 443 instead and try to connect via https.
Bloody hell, it can be that! I was taking for granted that since I used https un-securely, it was the same as http (80)!

I'll get back to home soon (I had to bring the dog out) and try it, then I'll report back. But I am sure as hell you pinpointed my flaw... Thank you!
 

balnazzar

Member
Mar 6, 2019
104
7
18
https uses 443 by default, it won't show the port after the IP address. I would try and forward port 443 instead and try to connect via https.
And indeed you were right, now it works. I'm twice stupid for not having thought about such an obvious (in the hindsight) thing..

Let me ask you another couple things I noticed.. I keep finding this kind of log:

scr.png

Detail:

scr2.png
Do you have any idea about what does it mean?

Furthermore, since (at least for now) I cannot manage to suspend the machine via the OS, I was wondering if I could do it with that:

scra.png

Is "soft shutdown" equivalent to suspend? I'd try it, but I thought it better to ask before trying stuff at random. Thank you!! ;)
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,263
428
83
ACPI shutdown is the equivalent of pressing the soft shutdown button. It's rare to find a server board that supports S3.

Also note that just changing the port/protocol to HTTPS doesn't really get you much in the way of security. The biggest security threat with exposing an IPMI directly to the internet isn't that your password might get easily snooped, but by drive-by exploit or automated brute-force attacks. IPMI on the internet is analogous to putting a spare keyboard and monitor plugged in to your computer on a busy high street - anyone walking past can have a crack at it. Putting it behind a VPN will keep out all but the most determined crackers.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
ACPI shutdown is the equivalent of pressing the soft shutdown button. It's rare to find a server board that supports S3.
Thanks for your reply. I don't really understand what S3 really is, but I'll search. For now, it seems it is NOT equivalent to sleep/suspend, that is, I won't find my work where I left it.

Also note that just changing the port/protocol to HTTPS doesn't really get you much in the way of security. The biggest security threat with exposing an IPMI directly to the internet isn't that your password might get easily snooped, but by drive-by exploit or automated brute-force attacks. IPMI on the internet is analogous to putting a spare keyboard and monitor plugged in to your computer on a busy high street - anyone walking past can have a crack at it. Putting it behind a VPN will keep out all but the most determined crackers.
I do understand that changing to https doesn't really add too much security. I just wanted to connect over the internet for testing purposes.
Coming to VPNs, I understand that the BMC should provide all the means to act as a vpn server. I'll read the manual and let you know.
But I was under the impression that with SSL certificates and passwordless login you could attain a level of security similar to VPNs.. From the manual:

Screenshot from 2020-01-28 01-15-26.png

What do you think about it? Thanks!!
 
Last edited:

balnazzar

Member
Mar 6, 2019
104
7
18
Uhm.. It seems the IPMI/BMC doesn't provide any means to act as a VPN server. It seems I have to use a machine (maybe a raspberryPI?) to keep the VPN up.. Am I right?

Thanks!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,298
553
113
Uhm.. It seems the IPMI/BMC doesn't provide any means to act as a VPN server. It seems I have to use a machine (maybe a raspberryPI?) to keep the VPN up.. Am I right?

Thanks!
Correct you need some other device such as your router/firewall or another standalone device such as an RasPi.
 

KarelG

Member
Jan 29, 2020
30
8
8
Thanks.

I'm reading this thread: https://forums.servethehome.com/ind...commended-setup-for-ipmi-over-internet.18121/

I'm not understanding it perfectly, but they mention cheap dedicated devices.. Should I go for them?
The thread is good, you should try to understand at least a gross meaning which is "Your IPMI is usually outdated and/or broken and/or support outdated/or broken crypto" That's at least how the *usual* situation with IPMIs are. So do not trust IPMI from security viewpoint hence you need to add "cheap" box in front of it which you trust from security point. How you do that exactly is than another question and depends on your capability to use IPMI's functionality -- e.g. are you OK with just serial over lan (hence ssh tunnel is enough) or do you need full web (here ssh tunnel may not work, but you will need VPN). The notes about VLAN is then cream on top of cake -- to isolate your IPMI even on your LAN...
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
The thread is good, you should try to understand at least a gross meaning which is "Your IPMI is usually outdated and/or broken and/or support outdated/or broken crypto" That's at least how the *usual* situation with IPMIs are. So do not trust IPMI from security viewpoint hence you need to add "cheap" box in front of it which you trust from security point. How you do that exactly is than another question and depends on your capability to use IPMI's functionality -- e.g. are you OK with just serial over lan (hence ssh tunnel is enough) or do you need full web (here ssh tunnel may not work, but you will need VPN). The notes about VLAN is then cream on top of cake -- to isolate your IPMI even on your LAN...
Thanks! Much clearer now... No, I do not want to isolate IPMI on my LAN, but I'd like to have full web access. If I'm understanding it correctly, you are saying that ssh won't be enough for full web since its overhead in encrypting all the traffic a web access would generate?
At this point, I think I'll buy a RPi 4B, it will make a good wireguard server and maybe I can manage to have fun with it and use it as a spare media player...
Or, if you want, please suggest a dedicated device in the range of 80/100$. Thanks!!
 

KarelG

Member
Jan 29, 2020
30
8
8
Thanks! Much clearer now... No, I do not want to isolate IPMI on my LAN, but I'd like to have full web access. If I'm understanding it correctly, you are saying that ssh won't be enough for full web since its overhead in encrypting all the traffic a web access would generate?
[...]
No, you don't as I've not been too clear about ssh tunneling and web access. Generally speaking it depends on IPMI vendor and if it did good or bad job. What you need from web is it to have it working well over whatever port you like to have -- e.g. your ipmi web UI can't hard-wire any port or any part of URL (host or IP addr). If they do that well, then you will be able to use web even throught ssh tunnel and if you like even using RPi which should be perfectly capable for it.

So just try it! :)
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
104
7
18
Mates, apologies in advance if I try and push my other issue a bit.

Suspend and hibernation just don't work on my setup, and that's truly a ballbreaker, since I run a lot of dockers and cannot restart all of them every time I need to power off the machine.

I tried several linux distributions (strarting from debian stable to ubuntu 20.04 test), compiled even the most recent kernel manually and optimizing for AMD (EDAC, etc etc), and tried even the userspace suspend (`uswsusp`). Nothing. The screen blanks, and that's all. The system refuses to power off.

It is very awkward, since it is rock stable under any load with every tested distro. Furthermore, prior to purchasing the epycd8-2t, I read that review: ASRock Rack EPYCD8-2T Makes For A Great Linux/BSD EPYC Workstation - 7-Way OS AMD 7351P Benchmarks - Phoronix

It says "These days any Linux distribution released in the past year or two is working fine with AMD EPYC processors. I personally tested this ASRock EPYCD8-2T with Fedora Workstation 29, CentOS 7, Debian 9.8, Clear Linux 27910, and openSUSE Leap 15.0. The experience was pleasant and without any issues to report on the Linux side."

I'm pretty sure I am not setting the BIOS properly. If any of you has the epycd8-2t with linux, could suggest something?
 

KarelG

Member
Jan 29, 2020
30
8
8
Mates, apologies in advance if I try and push my other issue a bit.

Suspend and hibernation just don't work on my setup, and that's truly a ballbreaker, since I run a lot of dockers and cannot restart all of them every time I need to power off the machine.
[...]
If you start/run your dockers with --restart=always -- google for docs, then they will restart automatically on boot. I think this is more easy to solve than suspend/resume on *server* board.