SuperMicro IPMI not reachable from VMs running in the local hypervisor

[iworkinnotworking]

I have bunch of SuperMicro KVM boxes on which we're seeing the behaviour described below:

Boxes have a shared IPMI in a 192.168.0.0/23 subnet, hypervisor host itself at .10 and ipmi at .11.
Host can arp and access the IPMI just fine.

Host is acting as a hypervisor and bridging virtual machines to the same 192.168.0.0/23 over the same physical port on which the IPMI is reached from outside. Virtual machines can reach the hypervisor at .10 and various other destinations inside the 192.168.0.0/23, both inside and outside of the bridge-domain (br0) of the hypervisor. However, for some reason the virtual machines cannot reach the IPMI, they do not receive arp replies from the IPMI. If I try to access the IPMI outside of the hypervisor box, it is reachable as are the VMs. So the problem appears to be localized to the IPMI subsystem/bridging.

Below is the illustration of the logical topology:

VM-->br0<--->eth0<--->Physical switch <---> Router
              |
              |
         Shared-ipmi

For reasons outside of my control there are no vlans involved in this.

So it kind of seems as if the IPMI is somehow filtering frames ingressing on the shared interface from the direction of the bridge-domain. To me this doesn't make all that much sense and I was thinking to suggest using the dedicated IPMI port. But has someone maybe seen something similar before and if so, did you happen to solve this somehow? One option I was thinking on trying is to add /32 route towards the IPMI that points to the router on one of the VMs, as the longest match wins the VM should punt the packet to the router that in turn would forward the packet right back to the hypervisor box but using its own mac address as the source. I suppose for this to work similar /32 route is needed also on the IPMI but as I do not have control to it I have not yet had the chance to try this. Any other suggestions are also welcome.

Thanks for your help/ideas.

 

[RolloZ170]

maybe completely wrong but i got in mind early reading this:
talking from host to own IPMI works over localhost or PCIe IPMI bridge(internal)
the shared ipmi LAN expects the recipient outside, packets sent outside.

 

[sko]

Just use a dedicated interface for the BMC IPMI. Shared setups often cause problems because of duplicate MAC addresses or other weirdnesses...
Also depending on the hypervisor/host OS there might be some limitations/"special behavior" of the bridge implementation in the way to get such a setup working reliably.

 

[iworkinnotworking]

maybe completely wrong but i got in mind early reading this:
talking from host to own IPMI works over localhost or PCIe IPMI bridge(internal)
the shared ipmi LAN expects the recipient outside, packets sent outside.

Hi, thanks a ton for the thought. Just so that I understand correctly, you are thinking whether the VM talking to the IPMI differs from the hypervisor talking to the IPMI from the perspective of the PCIe? That is very interesting idea and I have to admit that my expertise does not cover the PCIe bus in this context or sufficient detail.

I could be wrong, but the CPU of the hypervisor is doing the bridging (of the VM-to-physical NIC, and hence towards the IPMI) so I was thinking, isn't this basically the same thing as if CPU of the hypervisor is talking to the IPMI? But since the VM-to-IPMI doesn't work whereas hypervisor-to-IPMI does, maybe it is not quite that trivial. Alternatively the IPMI is actually implementing some filtering scheme where it drops frames destined towards the IPMI unless they actually come from either the source address of the NIC or indeed from outside the box itself? This would break the broadcast model of Ethernet quite handily but its likely completely within the realms of posibility with this kit.

Does anyone know if the network behavior of the SuperMicro IPMI is documented at sufficiently detailed level anywhere? I for one came up short whilst searching SuperMicro site.

Thanks,

 

[iworkinnotworking]

Just use a dedicated interface for the BMC IPMI. Shared setups often cause problems because of duplicate MAC addresses or other weirdnesses...
Also depending on the hypervisor/host OS there might be some limitations/"special behavior" of the bridge implementation in the way to get such a setup working reliably.

Yes, I tend to agree with what you are saying and I will certainly take the dedicated IPMI port into the discussion as to how to approach this matter. Problem with this is that it implies changing something that has worked apparently fine with previous server platform, but I guess that could be turned into a supporting argument for your suggestion considering the possible difference in behaviour and caveats between implementations/vendors.

Thanks for the suggestion

 

[RolloZ170]

VM talking to the IPMI differs from the hypervisor talking to the IPMI from the perspective of the PCIe?

afaik to access the PCIe->IPMI a VM needs passthrough.

 

[iworkinnotworking]

As it turns out, after some back and forth the dedicated ipmi port solved the issue as suspected.
Thanks for the suggestions.