Quanta LB4M MAC entry problem

[legen]

Hello everyone.

We use the LB4M in a active/passive configuration for our SAN (JovianDSS Open-E). Today we discovered a problem which we have debugged for some time. Our two SAN nodes has HA (failover) and for that it uses a virtual IP which is moved between the two head nodes in case of failure.

We discovered that this virtual IP MAC is missing from the LB4M switches mac-addr-table. This in turn causes all writes to this virtual ip to be mirrored to all ports on that vlan(!).

Have any one been running these LB4M switches in a active/passive setup and had this problem?
Do you think this could be solved by adding static mac entries for the virtual IP in question?

Or any other suggestions (we have already contacted the supplier of our SAN and are awaiting their response). We do not know why the LB4M switches fail to pick up the MAC of the virtual ip.

Thanks!

 

[aero]

No personal experience with LB4M here, but in a general sense...

Are you able to create a port mirror to look at the traffic coming from the SAN to verify that it is indeed sending traffic with the virtual MAC?

I believe static mac entries would work to solve the flooding, but will the switch allow you to create the same static mac on both of your ports? If not then you lose your HA.

 

[wildchild]

You could create a static mac to ip.. should fix you up.
Have seen similar behaviour running nlb in a vmware cluster.
Wouldnt be doing about the same thing would you ?

 

[Scott Laird]

Can you configure the SAN to fail over at L3 (IP address, via gratuitous ARP) instead of L2 (virtual MAC)? I'd expect that to work better in a lot of cases anyway.

In general, if a switch doesn't know which port to use for a MAC address, then the only thing it can do is to broadcast it to all ports. The way it learns the MAC/port mapping is by watching inbound traffic. Does the SAN send traffic using the virtual MAC, or does it use its real MAC address?

Getting MAC mappings right is pretty much the core of what switches do; I wouldn't be surprised if this was a bug in your SAN, not the switch.

If nothing else works, you could put the SAN devices onto their own VLAN, and then limit the number of ports that see traffic for that VLAN. That'll keep the broadcast-ish traffic from clogging other ports.

 

[Terry Kennedy]

We use the LB4M in a active/passive configuration for our SAN (JovianDSS Open-E). Today we discovered a problem which we have debugged for some time. Our two SAN nodes has HA (failover) and for that it uses a virtual IP which is moved between the two head nodes in case of failure.

We discovered that this virtual IP MAC is missing from the LB4M switches mac-addr-table. This in turn causes all writes to this virtual ip to be mirrored to all ports on that vlan(!).

I don't have any experience with either your SAN platform or your switch, but learning MAC addresses and putting them in the forwarding table is pretty much what switches do.

I would look to see what MAC address your SAN is using, and make sure it is a valid address. The switch may be refusing to learn an invalid address. One solution is to pick a random address with the same 3-byte prefix as one of your physical MAC addresses and see if the switch accepts it. For example, if one system is using 08-00-2B-18-29-30 and the other system is using 00-26-B9-70-A3-7F, pick a virtual MAC of 08-00-2B-45-67-89. You want the last 3 bytes to be very different from the "real" address, as systems can reserve a block of more than one address (for example, a 4-port network card or a "regular" Ethernet port and a "remote access" Ethernet port on the motherboard).

Another issue might be port security, where the switch only accepts traffic from a single MAC address, either hard-coded in the config or the first one "seen" on that port. Normally this will log an access violation and/or shut down the port, so that probably isn't the issue here.

 

[legen]

Thanks for the reply guys and sorry for the late replies!

No personal experience with LB4M here, but in a general sense...

Are you able to create a port mirror to look at the traffic coming from the SAN to verify that it is indeed sending traffic with the virtual MAC?

I believe static mac entries would work to solve the flooding, but will the switch allow you to create the same static mac on both of your ports? If not then you lose your HA.

The switches pick up the MAC of the actual SAN NICs and not the virtual IP MAC. Our xenservers do see the virtual mac however which is odd. How could the virtual IP MAC be present on the xenservers but not on the switches?

I agree that we might lose our HA with static MACs since both switches would announce that they have the wanted MAC on one of their pors :/

You could create a static mac to ip.. should fix you up.
Have seen similar behaviour running nlb in a vmware cluster.
Wouldnt be doing about the same thing would you ?

I wonder if that will really work and keep our HA intact?
If both switch 1 and switch 2 announce that they have the SAN virtual MAC i wonder what would happen?

Can you configure the SAN to fail over at L3 (IP address, via gratuitous ARP) instead of L2 (virtual MAC)? I'd expect that to work better in a lot of cases anyway.

In general, if a switch doesn't know which port to use for a MAC address, then the only thing it can do is to broadcast it to all ports. The way it learns the MAC/port mapping is by watching inbound traffic. Does the SAN send traffic using the virtual MAC, or does it use its real MAC address?

Getting MAC mappings right is pretty much the core of what switches do; I wouldn't be surprised if this was a bug in your SAN, not the switch.

If nothing else works, you could put the SAN devices onto their own VLAN, and then limit the number of ports that see traffic for that VLAN. That'll keep the broadcast-ish traffic from clogging other ports.

Il check if they can do L3 instead of L2 with my SAN supplier but i have no high hopes... I agree that it looks like the SAN might be doing it wrongly but i need to try and verify that. Also would need to check if this happens with other switches or if its the combination of our SAN and tjhe LB4M switches.

Il ask if my colleague can do some measurements on the SAN ports to see if and how the actual ARP messages look, if they use the virtual IP mac or the NICs mac or so.

I don't have any experience with either your SAN platform or your switch, but learning MAC addresses and putting them in the forwarding table is pretty much what switches do.

I would look to see what MAC address your SAN is using, and make sure it is a valid address. The switch may be refusing to learn an invalid address. One solution is to pick a random address with the same 3-byte prefix as one of your physical MAC addresses and see if the switch accepts it. For example, if one system is using 08-00-2B-18-29-30 and the other system is using 00-26-B9-70-A3-7F, pick a virtual MAC of 08-00-2B-45-67-89. You want the last 3 bytes to be very different from the "real" address, as systems can reserve a block of more than one address (for example, a 4-port network card or a "regular" Ethernet port and a "remote access" Ethernet port on the motherboard).

Another issue might be port security, where the switch only accepts traffic from a single MAC address, either hard-coded in the config or the first one "seen" on that port. Normally this will log an access violation and/or shut down the port, so that probably isn't the issue here.

Good idea. Il try to test using a MAC like the virtual ip MAC to see if the switch accepts it. Port security is off as far as i can tell so that should not be the issue here.