I looking for a better understanding on the difference between IPOIB and ETH mode on ConnectX-3 adapters (40/40 IB/Eth)
Couldn't find answers to my questions online; if someone could point me towards the right direction, it would be much appreciated.
1. My understanding is the IPOIB runs on top of Infiniband so the port can be used for both protocols. If the card is set to Ethernet mode, it cannot be used for RDMA (unless you run ROCE). In this case, it seems advantageous to run IPOIB if you need both protocols. Is this assumption correct?
ConnectX-3 Ethernet mode does not work on an Inifiniband only switch so it's probably not running on top of Infiniband implying that there must the CPU, latency and bandwidth benefits.
2. Does IPOIB require additional CPU compared to the Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
3. Does IPOIB have a higher latency than Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
4. Does IPOIB have a lower bandwidth ceiling compared to Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
5. Some switches seem to bridge Infiniband to Ethernet. How does this work when the protocols are different? Or is this only applicable when using ROCE (so the bridge is just removing the Ethernet wrapping on the Infiniband protocol)? [If I'm way off base, any guidance to better documentation would be appreciated.]
6. The ConnectX-3 VF in a VM seems to be have a much lower bandwidth than the PF function on the host (testing both IPOIB directly connected and testing Ethernet mode directly connected using Proxmox, debian modules). Have others experienced the same or am I just missing tweaks/configurations that would improve performance?
Thanks in advance.
Couldn't find answers to my questions online; if someone could point me towards the right direction, it would be much appreciated.
1. My understanding is the IPOIB runs on top of Infiniband so the port can be used for both protocols. If the card is set to Ethernet mode, it cannot be used for RDMA (unless you run ROCE). In this case, it seems advantageous to run IPOIB if you need both protocols. Is this assumption correct?
ConnectX-3 Ethernet mode does not work on an Inifiniband only switch so it's probably not running on top of Infiniband implying that there must the CPU, latency and bandwidth benefits.
2. Does IPOIB require additional CPU compared to the Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
3. Does IPOIB have a higher latency than Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
4. Does IPOIB have a lower bandwidth ceiling compared to Ethernet mode on the ConnectX-3? If yes, is it significant? My tests directly connecting two servers did not show a significant difference but it may be due to bottlenecks in my hardware.
5. Some switches seem to bridge Infiniband to Ethernet. How does this work when the protocols are different? Or is this only applicable when using ROCE (so the bridge is just removing the Ethernet wrapping on the Infiniband protocol)? [If I'm way off base, any guidance to better documentation would be appreciated.]
6. The ConnectX-3 VF in a VM seems to be have a much lower bandwidth than the PF function on the host (testing both IPOIB directly connected and testing Ethernet mode directly connected using Proxmox, debian modules). Have others experienced the same or am I just missing tweaks/configurations that would improve performance?
Thanks in advance.
