Does anyone have the Mellanox ConnectX-2 EN running in FreeBSD 10.3 in passthrough under VMware ESXi?
I have the single port 10 Gbps SFP+ version, and the performance I can squeeze out of a point-to-point connection (with 9000 byte jumbo frames) between a physical Windows 7 machine and a virtual Windows 10 VM is around 7-9 Gbps in single direction tests, and 5-6 Gbps in each direction during unidirectional tests using iperf3.
Previous tests were with the ConnectX-2 card interfacing VMware using built-in driver version 1.9.7. With 9000 byte jumbo frames I reached only approximately 1.5-2 Gbps between the Windows 7 physical hardware, and a FreeBSD 10.3 VM (FreeNAS 9.10), during iperf2 testing. The same performance was mimicked during multiple re-transfers of the same large file from a ZFS based share over SMB reaching no more than 150-200 MB/s during transfer essentially from and to RAM disks.
I tried to get the ConnectX-2 card to work in passthrough to FreeBSD and built the 2.1.5 driver (latest supporting ConnectX-2 according to its release note). During boot the driver complains about "No IRQs left", but I doubt this is the real reason.
I'd like to get this NIC running either in a pfSense or a FreeNAS VM, both of which are based on FreeBSD 10.3.
I have the single port 10 Gbps SFP+ version, and the performance I can squeeze out of a point-to-point connection (with 9000 byte jumbo frames) between a physical Windows 7 machine and a virtual Windows 10 VM is around 7-9 Gbps in single direction tests, and 5-6 Gbps in each direction during unidirectional tests using iperf3.
Previous tests were with the ConnectX-2 card interfacing VMware using built-in driver version 1.9.7. With 9000 byte jumbo frames I reached only approximately 1.5-2 Gbps between the Windows 7 physical hardware, and a FreeBSD 10.3 VM (FreeNAS 9.10), during iperf2 testing. The same performance was mimicked during multiple re-transfers of the same large file from a ZFS based share over SMB reaching no more than 150-200 MB/s during transfer essentially from and to RAM disks.
I tried to get the ConnectX-2 card to work in passthrough to FreeBSD and built the 2.1.5 driver (latest supporting ConnectX-2 according to its release note). During boot the driver complains about "No IRQs left", but I doubt this is the real reason.
Code:
mlx4_core0: <mlx4_core> mem 0xfd500000-0xfd5fffff,0xe7000000-0xe77fffff irq 18 at device 0.0 on pci3
mlx4_core: Mellanox ConnectX core driver v2.1 (Oct 3 2016)
mlx4_core: Initializing mlx4_core
mlx4_core0: No IRQs left, device won't be started.
device_attach: mlx4_core0 attach returned 28