Mellanox CX4 ConnectX VPI cards and FreeNAS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

themelon

Member
Jan 26, 2016
38
6
8
43
Son of a...

I had a nice, long post written up about some Mellanox CX4 VPI goodness and FreeNAS but I managed to hit a combination of keys that sent me back a few pages. I need to create longer post write-ups in a text editor first...

Short story is that the older ConnectX VPI cards with CX4 connectors that are supposed to be dual mode but I was never able to get into Ethernet mode NOW WORK in FreeNAS 9.10 as 10GBE ports without doing anything special. I'm assuming it is because of the driver enhancements in FreeBSD 10.3. I have not yet tried the same cards in newer Linux kernel/OFED versions but I would hope it would be the same. They are not recognized in ESXi 6 as such but I have not yet switched the vMware Mellanox stack out for the upstream version.

Here is a bit of iperf'ing I did between a Linux VM(server) on ESXi6 using VMXNET3 over a vSwitch using a Myricom 10G-PCIE-8 CX4 interface plugged into a Fujitsu XG2000 switch then to a Quanta LB4 to the Freenas(client) machine:

[root@sleven] ~# iperf -c 172.23.144.42 -t 30 -i 2 -w 8M
------------------------------------------------------------
Client connecting to 172.23.144.42, TCP port 5001
TCP window size: 516 KByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[ 3] local 172.23.144.11 port 39933 connected with 172.23.144.42 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 2.0 sec 1.53 GBytes 6.55 Gbits/sec
[ 3] 2.0- 4.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 4.0- 6.0 sec 1.53 GBytes 6.56 Gbits/sec
[ 3] 6.0- 8.0 sec 1.53 GBytes 6.58 Gbits/sec
[ 3] 8.0-10.0 sec 1.52 GBytes 6.53 Gbits/sec
[ 3] 10.0-12.0 sec 1.53 GBytes 6.55 Gbits/sec
[ 3] 12.0-14.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 14.0-16.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 16.0-18.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 18.0-20.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 20.0-22.0 sec 1.53 GBytes 6.56 Gbits/sec
[ 3] 22.0-24.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 24.0-26.0 sec 1.53 GBytes 6.57 Gbits/sec
[ 3] 26.0-28.0 sec 1.53 GBytes 6.56 Gbits/sec
[ 3] 28.0-30.0 sec 1.53 GBytes 6.56 Gbits/sec
[ 3] 0.0-30.0 sec 22.9 GBytes 6.56 Gbits/sec

Sometime this weekend I plan to benchmark with a non-VM host and perhaps I will get more than the ~6.55Gbit/s wall that I hit. I was at about the same speeds when I switched the VLAN on the FreeNAS machine back to the Chelsio S320E-CXA as it's parent. Even if I don't break that wall I still have discovered a new card that actually works on FreeNAS in 10Gbit mode over CX4!! Now I won't have to chase down the usually more expensive Chelsio S320E-CXA, Myricom or the Intel CX4 cards. The Mellanox cards that I am specifically using are HP branded part number 452372-001/448397-B21 FW v2.9.1000 with the following lspci output:

06:00.0 Network controller: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)

WOOHOO!!

I thought I should pass this information on to anyone that was curious.
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I think the FreeNAS networking stack is limited to around 40Gbps max right now.
 

themelon

Member
Jan 26, 2016
38
6
8
43
It's funny, and awesome at the same time, that we are at such a point that an enthusiast can easily hit walls like that at home.