I attempted this w/ a 3 node vSphere cluster and direct attached X540-T2's and had a helluva time getting everything to work right (vmotion, svmotion, random NFS disconnects/mounting/vmkpinging issues...it was absured). Guess what was the EASY fix, get a switch involved :-D
Sorry not much help but I am fairly handy in both the networking and virtualization world and hell if I could make this sing. 4 Hosts direct connect/attached and all hosts playing nice networking wise is gonna be tough. Maybe RTM's response is possible but if I were a betting man your gonna invest more than it's worth ('is the juice worth the squeeze' if you know what I mean?).
I have no doubt that a 2 node direct connect setup would probably work very well and in theory I thought a 3 node vSphere direct conn 10G base-T setup was gonna be pretty simple/straightforward (and I tried all sorts of shinanigans of tagging/untagging vSwitch port group/phys switch/native vlan/VMware 4095 pass all vlan's trick, promisc/mac forge/vDS trickery, etc...tons of other stuff) but I soon learned a hard lesson and swapped out my X540-T2's for 520's and DAC cables/switch upgrade. Everything was DEAD simple after I got a switch involved which I just knew was gonna be the case.
Good luck, let us know if you get it to all play nice.