I know this is not an answer, but this is something I would love to know how to do as well. I have never been able to find a way to set the port mode in ESXi . Must be a module flag you can set via the command line. Assuming they are just not left as autodetect.
As far as what mode it the port is in, that you can see. From the docs:
vmnic_ibX means the device port is in InfiniBand mode.
vmnicX means the device port is in Ethernet mode.
Edit: Oh nevermind, read your second post. If your going through a IB switch then you want IPoIB which means you want infiniband mode. As you noticed