Intel AMT VLAN

[PigLover]

Built my first system system using Intel AMT over the last week. It is an attempt as a really small, passive cooled (no fans) ESXi server. Its built on an Intel DQ77KB mini-ITX 1155 motherboard with a I3-3470T low power CPU. Except that I haven't actually got it running passive right now it turned out to be a really good build.

I noticed something about Intel's Management Engine (ME/AMT) that I found a bit disappointing when compared to IPMI 2.0. On most IPMI implementations you can specify a VLAN for the management traffic. This is especially important when management traffic and "normal" traffic share the same physical LAN interface. Can't do this on the Intel Management Engine...it runs untagged only.

Also, even when there is a shared NIC, IPMI uses an alternate MAC address for the management engine. Intel doesn't. So if the ME gets its IP address using DHCP you can't do a separate DHCP transaction for whatever actual service runs on that port unless you go into the OS and spoof the MAC.

All of this becomes a PITA when you run your management traffic segregated onto a separate LAN but don't want to hard-dedicate one of the two ports to system management traffic.

I did manage to solve this for this build by doing something really abnormal. For the connection to the management NIC (red NIC) is set my HP switch to send VLAN 200 (the management VLAN) untagged and set the normal "default" VLAN 1 to run tagged. This allowed me to get the AMT link onto the right subnet. But I still had to put the VMware management port onto the second NIC on VLAN200 due to the common MAC address and DHCP problems it creates.

Leaves me with VLAN 200 exposed on both NICs (a potential security issue) and a strange configuration of the VLAN tags on one line (a "error prone" configuration because simple mistakes lead to unexpected traffic on VLAN 200).

I could have solved the entire problem by dedicating "red" port to management traffic only and then using static IP configs for either the AMT or VMware management interface.

All would have been much simpler if Intel had just implemented VLAN on the management engine...

 

[PigLover]

Found another surprising interaction between ESXi and vPRO/AMT...could be MB specific to the DQ77KB.

On the DQ77KB, if you do a "passthrough" of the USB3 controller in VMWare then you lose control of the virtual mouse/keyboard with AMT. You can still view the display using VNC Viewer plus, but you can't type anything (acutally - a small correction - anything you type shows up as keyboard input to the VM you passed the USB3 controller through too).

Apparently the virtual keyboard is presented to AMT on the USB3 controller - and if you do a passthrough you end up taking the controller away from AMT.

I don't need it to be the USB3 controller - I could pass through the USB2 ports, except that the current case I am using (Akasa Euler) does not provide any way to expose any of the motherboard USB ports. I just need one (ugh!).

 

[Patrick]

That is an interesting find!

 

[PigLover]

As I've been looking deeper into this it appears that the DHCP problem is by design. It is supposed to operate with both the vPRO/AMT and the host OS sharing the same IP address. In fact, AMT snoops for the host OS sending DHCP requests on the shared port and modifies its IP address to match the host if the host gets assigned a new one!

In their "known/fixed issues long" it appears that early versions of vPRO/AMT firmware got confused by virtual networks presented by most hypervisors, which end up sending separate DHCP requests for each client OS. vPRO/AMT ended up taking on the IP address of the last client OS to make a DHCP request. They fixed it by checking the MAC address presented in the DHCP request more carefully...

I'll give it a try with vPRO/AMT and ESXi sharing the same IP address and see how it goes. Its would still be nice if you could specify a VLAN for vPRO to use like you can with ESXi.

 

[Aluminum]

Found another surprising interaction between ESXi and vPRO/AMT...could be MB specific to the DQ77KB.

On the DQ77KB, if you do a "passthrough" of the USB3 controller in VMWare then you lose control of the virtual mouse/keyboard with AMT. You can still view the display using VNC Viewer plus, but you can't type anything (acutally - a small correction - anything you type shows up as keyboard input to the VM you passed the USB3 controller through too).

Apparently the virtual keyboard is presented to AMT on the USB3 controller - and if you do a passthrough you end up taking the controller away from AMT.

I don't need it to be the USB3 controller - I could pass through the USB2 ports, except that the current case I am using (Akasa Euler) does not provide any way to expose any of the motherboard USB ports. I just need one (ugh!).

So I'm not the only one with that case!

What you could do is get one of those short usb header plugs and pass it through the cutout for the expresscard. Not the most elegant thing in the world, but hey it works

I bought the expresscard riser from mini-box, but its designed for the atom board where the minicard slot is farther back (DN2800MT) so it sticks out too much to use the backplate.

 

[PigLover]

...What you could do is get one of those short usb header plugs and pass it through the cutout for the expresscard. Not the most elegant thing in the world, but hey it works ...

That's exactly what I am likely to do. I have enough parts stored from years of builds/tear-downs/rebuilds that I have quite a selection of USB header cables. There isn't much room on the thin-mini-ITX IO shield to do something pretty, so I probably will have to just pull the cable through the expresscard slot. I guess it will have to do.

Disappointing part is that I just got it all installed and mounted before I discovered this. I figured nothing would go wrong if I didn't test the USBs first. With the job I do for a living I should know better. Test everything before you install!

 

[PigLover]

Been playing around some more...

IP address sharing between the Management Engine (vPRO/AMT) and the host OS (ESXi for this machine)

Reconfigured things so that vPRO/AMT and ESXi both use the same port for management. Set my management VLAN to "untagged" on that port of my switch so that both management interfaces hit the Management VLAN (and I set the "normal" VLAN 1 to run tagged on that port so that I can still get to it from VMs that share that port. I tried all five headers with the same result (both "double" port headers and the single USB header).

Both the vPRO/AMT and ESXi get the same IP address via DHCP. The management engine seems to properly intercept traffic to its reserved ports. KVM using RealVNC+ gets to the console display. Port 16992 gets to the vPRO web interface. Everything else at that IP address goes right through to ESXi. All seems well.

There was a problem with link speed negotiation. For some reason the NIC kept setting up at 10Mbit after ESXi loaded (bad!). But I found that simply forcing the speed to GigE in the host OS (ESXi's config) caused it to reset to GigE after ESXi initialized no problem. No idea what's up with that - but something about the management engine was interfering with link speed negotiation during ESXi driver setup.

USB Passthrough in ESXi interferes with vPRO/AMT KVM keyboard

I did a little ghetto-mod and pulled an old USB-header cable through the slot in the IO shield made for the expresscard (per Aluminum's idea above). Unfortunately, all of the headers on the MB appear to be part of the same USB controller driving the back-panel USB3s, so no matter which port you use you have to "pass through" the USB controller that gets the keyboard to the management engine...so no working "K" in your "KVM" if you use ESXi passthrough on the USBs.

What is strange is that ESXi reports that there are two USB controllers available. I took a look at the block diagram for the board in Intel's technical spec. Unfortunately, it just shows all of the USB coming out of the Q77 chip. There is one more USB pathway associated with one of the expresscards - so maybe that is the other USB controller. I'll have to try and find the technical specs on the Q77 chipset itself to get more information.

Oh well. Rarely need the ESXi console anyway. Almost anything you need can be done over SSH - and I'd rather do it over SSH anyway. And if I really must have KVM to the console I can just shut down the VM with the USBs passed through and unconfigure the passthrough for a little bit. Over the last several hours I've gotten unreasonably good at making this change...

 

[Aluminum]

USB Passthrough in ESXi interferes with vPRO/AMT KVM keyboard

I did a little ghetto-mod and pulled an old USB-header cable through the slot in the IO shield made for the expresscard (per Aluminum's idea above). Unfortunately, all of the headers on the MB appear to be part of the same USB controller driving the back-panel USB3s, so no matter which port you use you have to "pass through" the USB controller that gets the keyboard to the management engine...so no working "K" in your "KVM" if you use ESXi passthrough on the USBs.

What is strange is that ESXi reports that there are two USB controllers available. I took a look at the block diagram for the board in Intel's technical spec. Unfortunately, it just shows all of the USB coming out of the Q77 chip. There is one more USB pathway associated with one of the expresscards - so maybe that is the other USB controller. I'll have to try and find the technical specs on the Q77 chipset itself to get more information.

Oh well. Rarely need the ESXi console anyway. Almost anything you need can be done over SSH - and I'd rather do it over SSH anyway. And if I really must have KVM to the console I can just shut down the VM with the USBs passed through and unconfigure the passthrough for a little bit. Over the last several hours I've gotten unreasonably good at making this change...

One more slightly crazy thing you could try:

Expresscard adapter from mini-box (~$10) though you have to remove or cut the backplate even more.
+
USB expresscard that uses a dedicated pci-e controller, not sure which type would be best but there are a couple of 2 port usb 3.0 with reneas chipset for cheap.


This also means the mSATA port is blocked if that matters.



/incoming wild speculation and unverified guesses
I think the intel usb bonding thing might have something to do with their controller able to designate the ports as either 3.0 xhci or the more compatible 2.0 ehci controller. Many other boards are not able let non-3.0 aware environments (XP, firmware interfaces, older OS installers, etc) see the ports without a preloaded driver which might not even exist. (using inside its own UEFI seems to be the exception but given how new it is, makes sense)

I think they might even be using a mux design somehow. It would explain why the board seems to be "short" 4 possible ports. (should be 4x 3.0 and 10x 2.0, I can only account for 5~6 of those 2.0 ports factoring in header pins and minicard slot)

 

[PigLover]

I actually think I'm done playing with it. It is what it is. Functionally, it is perfect. I have my ESXi host fully relocated to a little Mini-ITX box. Its dead silent with no moving parts. Its using otherwise 'wasted' space. With four full-time VMs running (FreePBX, Web/FTP Server, email server, myth-tv back-end) it draws around 17watts at the plug most of the time, rising to about 28-30watts occasionally when its "busy". Since its sitting right under my printer, I added a 5th VM with a copy of W7 as a "printer server" (which is why I wanted the USB passthrough). By hosting the printer i can now let my main machine 'sleep' when its not in use, saving even more power (I had to leave that one on because my kids computers and the various laptops all pointed to it for printing - if it went to sleep they had to wake it up before they could print).

All in all, except for this one "wart' related to remote KVM, it couldn't have satisfied what i was trying to do any better than it does.

While it was fun exploring ways to "fix" the KVM problem, for now I'm all done and I'll just live with it. If I ever actually need the KVM function with ESXi running I'll just de-configure the USB passthrough as long as needed for task.

...It would explain why the board seems to be "short" 4 possible ports. (should be 4x 3.0 and 10x 2.0, I can only account for 5~6 of those 2.0 ports factoring in header pins and minicard slot)

Addressing this one part: I think the reason they are "short" 4 USB ports has a very simple explanation: mini-ITX real estate. There just isn't more room for more headers or the traces to feed them. You'll notice they are also short 1 SATA port and 2 lanes of PCIe compared to what the Q77 chipset delivers. Also - real estate.

 

[xnoodle]

Are you running ESXi 5.0 or 5.1?

 

[PigLover]

5.0 update 1 with the "automatic startup" patch applied (actually, the entire patch rollup containing the fix for automatic startup of VMs). I've also applied a community supported driver to activate the 2nd NIC (the Intel 82579LM is not natively supported).

I can't use 5.1, has to be 5.0. 5.1 currently has two problems handling USBs that prevent me from using it. First, USB device passthrough is completely broken. You can select a USB controller for passthrough in the advanced configuration screen, but upon reboot to activate it is no longer selected. VMWare knows about this and has committed a patch to fix it, but won't release it until the first "combined updates" release.

The second problem is that normal USB handling does not yet work correctly with USB3 controllers, at least not for Windows. There is a hackish workaround for W8/Server2012, but YMMV. Besides, ESXi has to recognize your USB device first before this even works - and it doesn't recognize my printer.