Voltaire 4036E dropping packets

I think i'm missing something simple here. running ConnectX-3 40GB and have a 4036E on ESXI servers and if trying to connect to my ethernet over 10GB SFP i get about 50% dropped packets when trying to ping any device over IB. I can ping the voltaire interface ip fine, the voltaire can ping the devices fine, but from a desktop or even from an IB to another IB it drops. If I setup the IB to use another gateway i can ping internally over IB just fine. Do I need to setup separate gateways and route tables between them or is there something missing in my config to allow it to use my firewall as a gateway like a L2 normal switch. I'm not very experienced in networking so i'm hoping its a simple thing im missing.