I'm wrapping up a project to do something like this, and for the most part it's worked well for me.
First, I tried those USB display adapters a few years ago in ESXi, and they did not work without a real GPU (the driver said it needed a GPU with OpenGL support above what the software SVGA GPU has or something). Perhaps that's changed now though.
HDMI audio has an issue for me, and I've seen one report of the same thing happening to someone else. On my host, if you pass through the HDMI audio device to a VM, whenever you shut down the VM, the entire host just resets (not even a PSOD). It doesn't do this if I leave out the HDMI audio device when passing through. But it looks like you won't use HDMI audio anyway.
I passed through one of the onboard USB controllers, which mostly works. I seem to get extremely brief dropouts (my keyboard lights go out, and the mouse jumps). It's slightly annoying, but not a deal breaker. I have an Apple Cinema Display which has a built in webcam and speakers that plug in over USB, and both of those work fine.