BMC/IPMI Best Practices

[ReturnedSword]

Here's a subject that I'm only obliquely exposed to at work, that seems to be in major discussion in light of recent events (i.e. Bloomberg Businessweek Supermicro story).

I'm in the slow process of redoing my home lab, mostly downsizing/consolidating to fewer systems. Currently I manage each system by accessing it directly as my homelab is partially located in my home office. I'll be moving almost everything to my garage, so it'd be a pain to run back and forth from home office/garage. I'll also be picking up more IPMI enabled systems as time goes on to replace my current hardware.

The high level view of my current setup is everything goes through a pfSense router, split between WAN, LAN, WiFi, GUEST, OPT4. OPT4 is currently not being utilized. WiFi clients must VPN into LAN to access LAN resources. I also use OpenVPN for my laptop/mobile to access LAN when I'm road warrioring.

Here are some questions for network security gurus:

1. What are the best practices for isolating the management network (IPMI) from my other networks?
2. Is it enough to place the management network on a separate VLAN, or on the physical OPT4 network, with all outbound traffic from the management network blocked?
3. Is it necessary to have a completely different router for the management network? I'd also need to be able to VPN into the management network occasionally from separate physical locations.
4. What are the best practices for local management? A VM existing on a system residing on LAN/WiFi that's VPN'd into OPT4, or a physical system (e.g. NUC/laptop or something similar) connected directly to the management network?

 

[Patrick]

I have a content note to do get an article done on this.

I am not a big fan of covering this Bloomberg stuff as I prefer we focus more on the technical end. It is creating havoc in our publishing queue.

 

[ReturnedSword]

I agree on the sensationalist nature of the Bloomberg article, and think their reporters could have made some more effort to explain their allegation. Instead it read to laypeople like "The Chinese have implanted superchips into the cloud." These BMC/IPMI vulnerabilities have been popping up occasionally since the IPMI standard was implemented AFAIK, and won't be going away anytime soon.

At work, I do more enterprise design stuff so from my point of view things like BMC/IMPI is "black magic" that just makes things work. We usually delegate these things to the network guys, and it's embarrassing to say, but even our enterprise security people usually don't pay as much attention to this particular area.

My intention of this discussion is to give myself a better understanding of best practices relating to BMC/IPMI management. Do you have any high level pointers in mind to get me started?

 

[herby]

I've been thinking about this a bit with my machines at home too. I currently just lump my IPMI IP addresses with my hypervisors and network management in one VLAN. Connections in from other VLANs are fairly limited, but I don't block any outgoing connections. I might need further separate into more networks and tighten up some firewall rules.

 

[RTM]

Fundamentally I see no reason to treat IPMI much different than (the ideal of) other management interfaces.

No "full" direct internet access, if servers need internet access such as for updates it is via a proxy that uses whitelisting or even better a local repository.
A management interface should only be accessible via a jumphost (ideally something like cyberark, thycotic secret server but a decent bastion host should do the trick), and is not able to access other management interfaces.

The important thing is not just to protect against outgoing traffic, but also against traffic flowing to and from management interfaces.
We need to make it harder for an attacker to do lateral movement on our networks, we have to accept that server/system/whatever do get compromised, but preventing further success of the attacker is key to a successful defense.

My preferred method of implementing this is by doing segmentation via a firewall, with a /30 network per service (and of course adequate firewall rules), but you could probably implement it with port isolation on a switch or "direct" connections between jumphost and management interface (virtual or physical).

 

[WANg]

Here's a subject that I'm only obliquely exposed to at work, that seems to be in major discussion in light of recent events (i.e. Bloomberg Businessweek Supermicro story).

I'm in the slow process of redoing my home lab, mostly downsizing/consolidating to fewer systems. Currently I manage each system by accessing it directly as my homelab is partially located in my home office. I'll be moving almost everything to my garage, so it'd be a pain to run back and forth from home office/garage. I'll also be picking up more IPMI enabled systems as time goes on to replace my current hardware.

The high level view of my current setup is everything goes through a pfSense router, split between WAN, LAN, WiFi, GUEST, OPT4. OPT4 is currently not being utilized. WiFi clients must VPN into LAN to access LAN resources. I also use OpenVPN for my laptop/mobile to access LAN when I'm road warrioring.

Here are some questions for network security gurus:

1. What are the best practices for isolating the management network (IPMI) from my other networks?
2. Is it enough to place the management network on a separate VLAN, or on the physical OPT4 network, with all outbound traffic from the management network blocked?
3. Is it necessary to have a completely different router for the management network? I'd also need to be able to VPN into the management network occasionally from separate physical locations.
4. What are the best practices for local management? A VM existing on a system residing on LAN/WiFi that's VPN'd into OPT4, or a physical system (e.g. NUC/laptop or something similar) connected directly to the management network?

The answer will be synonymous with "hey, suppose if some random guy with some technical knowledge now has physical access to your machine through the internet. How would you limit the damage that he/she can inflict on your infrastructure?"
It's basically the multi-layer approach - toss up VLANs, do point-to-point VPN access with multi-factor authentication (Yubico Yubikeys via a PAM module, for example) isolate networks to least privileges, encrypt the important stuff, and as always, if it's a homelab, treat it as ultimately disposable - it should not be a perpetual money drain that consumes way too much time and money off your downtime. The second you start getting too paranoid to the point where you go boxing with shadows...is the second you should shove your phone into a faraday pouch and get some outdoor time. That's pretty much my biggest takeaway from CISSP - your defenses should be proportional in terms of cost and time consumption to the net worth of whatever you are planning to protect.

 

[laserpaddy]

I have a content note to do get an article done on this.

I am not a big fan of covering this Bloomberg stuff as I prefer we focus more on the technical end. It is creating havoc in our publishing queue.

I bet thats an understatement--I really enjoyed the article insights...

 

[BLinux]

perhaps a bit specific, but I think appropriate to this topic: With Supermicro IPMI interfaces, I've used OpenVAS to scan for vulnerabilities and found reports of an "anonymous" account. For production type of environments, I will usually remove the "ADMIN" account and add a new account with admin privileges. But, how does one get rid of the "anonymous" account and what kind of privileges can be gained with this "anonymous" account? What is the best practice in terms of configuring the Supermicro IPMI to be secure as possible? (other than external things like putting them on a protected segmented network)

 

[fohdeesha]

What is the best practice in terms of configuring the Supermicro IPMI to be secure as possible? (other than external things like putting them on a protected segmented network)

If you want my honest opinion after the last month or so of being buried in BMC firmware reverse engineering (idracula etc): move away from supermicro gear

 

[BLinux]

If you want my honest opinion after the last month or so of being buried in BMC firmware reverse engineering (idracula etc): move away from supermicro gear

that seems a bit dramatic, no? that's like saying the internet is so scary, we should not use it and live off grid and lock up all our computers and never use them again? like you found out with idracula, all these things with any brand has problems. even if one decides these BMC/IPMI things are too dangerous to have around, you can disable it with a jumper on the motherboard; at least with the Supermicro boards. i believe idrac can also be disabled on Dells, not sure of others...

 

[fohdeesha]

not saying that, but it became pretty clear that even with all the "good admin" preventative measures you can think of, you're still left with firmware that's inherently much less secure than the competitors. Personal choice, of course. Just don't put it anywhere on the internet (inbound OR outbound) and don't let anyone you don't trust near it, you'll be fine (assuming nothing happened in the supply chain, but you'd have to be an important target for that to happen)

for a point of reference: idracula took 2 to 3 weeks of nightly work to discover/make happen, with supermicro, 30 to 60 seconds, depending, and I don't know how much I should say publicly but it's also not something that I believe they can patch with software, only way around it I see is full keys embedded in processor + sig checking. Not trying to sound like a dell salesmen here (really) but that's my experience the last month or two

 

[ReturnedSword]

It would be sufficient to place the IPMI interface on an isolated VLAN/physical network though to mitigate issues right? That way any vector would need to physically be there in front of your server. I could do without remote access via VPN, but completely disabling the BMC kind of invalidates the usefulness of the BMC to begin with.

 

[kapone]

We all assume (in general) that the BMC is isolated from the host OS and can only be accessed from the network interface that it is bound to.

Well... You see my point. Any vulnerability or compromised hardware could throw that assumption out the window.

 

[ReturnedSword]

Is it possible for traffic to transverse via the host OS from regular interface to the management interface? I can foresee this happening with systems that have a shared management interface, however we can just configure the interface to be used only for the BMC, right?

 

[kapone]

Is it possible for traffic to transverse via the host OS from regular interface to the management interface? I can foresee this happening with systems that have a shared management interface, however we can just configure the interface to be used only for the BMC, right?

As an example...OS to iDRAC Pass through using iDRAC7 | Dell Canada

 

[Aluminum]

Minimally they must be on a VLAN that is walled off from everything, but separate physical if you want to be real.

One problem is that some BMC hardware setups seem to have the ability to use any onboard LAN port, not just the designated (typically realtek) port. In fact I'm pretty sure some of them have an auto-failback, if no cable is plugged in the "dedicated" BMC port, they will listen on one of the other (usually two) by default. I find this design ****ing insane to be honest.

Too many IT people don't seem to get this stuff in their heads either, if you showed them KVM rack drawer and said "should we have this wired to the bench outside" they would laugh, but the embedded equivalent ends up on the internet more often than people seem to realize (with a crappy web login portal at best).

 

[dswartz]

Minimally they must be on a VLAN that is walled off from everything, but separate physical if you want to be real.

One problem is that some BMC hardware setups seem to have the ability to use any onboard LAN port, not just the designated (typically realtek) port. In fact I'm pretty sure some of them have an auto-failback, if no cable is plugged in the "dedicated" BMC port, they will listen on one of the other (usually two) by default. I find this design ****ing insane to be honest.

Too many IT people don't seem to get this stuff in their heads either, if you showed them KVM rack drawer and said "should we have this wired to the bench outside" they would laugh, but the embedded equivalent ends up on the internet more often than people seem to realize (with a crappy web login portal at best).

Not only do some of them have auto-failover, but it seems like half the time, if you DO want auto-failover... it doesn't work!

 

[fohdeesha]

Is it possible for traffic to transverse via the host OS from regular interface to the management interface? I can foresee this happening with systems that have a shared management interface, however we can just configure the interface to be used only for the BMC, right?

The fun part is loading modified firmware that behaves and reports that it's using the super secure dedicated IPMI port you've spent months securing, but is also active and listening on the second NC-SI interface (shared with your regular server ports), and using that for all your "fun traffic"

NICSelection.sh (GitHub)

network.sh (GitHub)

 

[ReturnedSword]

As the consensus seems to be that BMC firmware is usually bug-ridden and full of security holes, when is the point I need to flip out and disable the BMC altogether and physically plug in a monitor and keyboard?

 

[kapone]

As the consensus seems to be that BMC firmware is usually bug-ridden and full of security holes, when is the point I need to flip out and disable the BMC altogether and physically plug in a monitor and keyboard?

When you think you're important enough for a "state actor" to take notice of you.