ConnectX-3 and SR-IOV on modern kernels?

[naptastic]

TL;DR: If you have a ConnectX-3 and a recent kernel, and are using SR-IOV successfully, what options are you using?

I have a CX354A-FCBT and I have not gotten SR-IOV to work at all. I'm using inbox drivers since old MLNX-OFED drivers won't work with modern kernels and new MLNX-OFED drivers don't support ConnectX-3. My current kernel version is 7.0.14 with one patch applied that gives (basically) -march=native; otherwise it's vanilla.

Here is mlxconfig -q for the card:

Device #1:
----------

Device type: ConnectX3
Device: 0e:00.0

Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 16
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 0
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 iSCSI(2)
BOOT_VLAN_P2 0
IP_VER_P1 IPv6(1)
IP_VER_P2 IPv6(1)
CQ_TIMESTAMP True(1)

So far so good: the firmware is configured for SR-IOV. But when I enable VFs in the driver, no interfaces on the card will come up; no traffic gets in or out. Here are the lines I have tried in /etc/modprobe.d/mlx4_core.conf:

# This breaks things
# options mlx4_core num_vfs=0,0,7 probe_vf=0,0,1 port_type_array=1,2

# This also breaks things
# options mlx4_core probe_vf=0 num_vfs=7

# This also breaks things
# options mlx4_core debug_level=1 port_type_array=1,2 num_vfs=0,8,0 probe_vf=0,0,0

# I don't actually know what this option does but I think it's interfering with SR-IOV
# log_num_mgm_entry_size=-1

They're all commented out since, y'know, I like to actually use the card. But SR-IOV would be really helpful, and I know the platform supports it since SR-IOV works just fine with Connect-IB and ConnectX-4 cards.

If I try to create VFs after the card has been initialized:

[/sys/bus/pci/devices/0000:0e:00.0]# echo 16 > sriov_numvfs
-bash: echo: write error: No such file or directory

...I expected this to fail but ENOENT isn't what I expected. Whatever. The official documentation is pretty much all wrong. I think maybe I need different options for mlx4_core but even reading the source code doesn't tell me enough.

Thanks in advance,
Nap

 

[Stephan]

Driver mlx4 saw real development 10 years ago. You might need a patch. Patch by somebody else, a looong time ago. Also I like KISS principle more, give VM a virtio-net device, bridge to whatever VLAN on host as appropriate, done. 56 Gbps also the last sane step-up from 1/2.5/5/10 Gbps in homelab imho.

# rmmod mlx4_ib mlx4_en mlx4_core ; modprobe mlx4_core num_vfs=0,0,16 ; lspci | grep MT27500
b3:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
b3:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:00.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:01.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b3:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

 

[naptastic]

Driver mlx4 saw real development 10 years ago. You might need a patch.

Patch applied cleanly to 7.1.4. Going to test now. Thanks!

Patch by somebody else, a looong time ago. Also I like KISS principle more, give VM a virtio-net device, bridge to whatever VLAN on host as appropriate, done. 56 Gbps also the last sane step-up from 1/2.5/5/10 Gbps in homelab imho.

Absolutely agree. EDR InfiniBand / 25gbE / 100gbE requires more expensive PBC materials, so it's probably never going to come down to current FDR / 56gbE prices. (I do wish they'd made 14gbE a thing. The transceivers exist; Fibre Channel uses them for 16GFC.)

25 Gbps also requires FEC for any link over 2 meters (I think?), which gives it worse port latency than FDR. The new Ethernet offloads consume tons of power and take up tons of die area; an IB-only card with VXLAN overlays, even with CPU emulation, is still less power than a ConnectX-4, and a lot less money.

I do like the simplicity of virtio-net with bridging, but I want separate IB GUIDs for SAN purposes. (I use SRP for block storage. I'm not crazy; I just miss Fibre Channel.) Also, I want to experiment with routing Internet traffic to my IPoIB network.

 

[klui]

I haven't done this but should. Someone here posted a set that's a bit different. The shell script gets the source from kernel.org's git page.

 

[naptastic]

Question for both of you: What's in /etc/modprobe.d/mlx4_core.conf (or your system's equivalent)? I still can't get any interfaces to pass traffic if any VFs exist. There are errors in dmesg but I'm pretty sure I'm just holding it wrong.

Edit 2: Are either of you using Infiniband at all? I really don't want to go Ethernet-only but from the patches, I think Infiniband support may not have been tested or patched. However, I'm still at square 1 because the driver configuration is still screwed up. (Guess I could try setting ETH/ETH and see if I get working links...)

Edit: I compared the two patch sets. The only in-kernel differences are whitespace. The zip file contains a patch for DKMS; I don't know enough about DKMS to know if it's relevant or not. But since I'm patching the kernel itself, I'm guessing it's not necessary.

Thanks,
Nap

 

[klui]

No, I'm not running a newer kernel and CX-3 w/ SR-IOV. I'm running an old 6.8.12 kernel.

Not using IB.

 

[naptastic]

(Am I the only homelabber in the world using Infiniband? LOL)

 

[klui]

It would help if there is a guide on how to use it. I understand IBoIP incurs performance hits and most of the services I rely on use IP like SMB. If I could get nfs working using IB I wouldn't mind trying it. But how would that work from a Windows client?

 

[naptastic]

It would help if there is a guide on how to use it. I understand IBoIP incurs performance hits and most of the services I rely on use IP like SMB. If I could get nfs working using IB I wouldn't mind trying it. But how would that work from a Windows client?

The official documentation is all but useless, so I keep my own internal documentation. It doesn't cover anything Windows-related since I have no Windows hosts left. But I know there's a Windows version of OFED. Once you have IPoIB working, you can use SMB Direct and get full speed file transfers. On Linux, you can use NFS/RDMA to get full-speed file access. For block storage, you have your choice of iSCSI+iSER, NVMe-oF, or SRP. I'm fond of SRP because it almost tastes like Fibre Channel, and the Linux target has ramdisk support built in.

If there was ever documentation, I think it must only have been provided under NDA to enterprises who spent over $10M on hardware. More likely, it never got written.

 

[klui]

Thanks for sharing. I will have to reconsider and try it out. How fast do your file transfers max out?

Most of my hosts are Linux but my one SMB is just Windows.

 

[Stephan]

I have nothing in mlx4_core.conf, not using SR-IOV or Infiniband. Just raw 56 Gbps ethernet with suitable cables (DAC and also fiber). Or 40 Gbps with the Kaiam LR4 lite transceivers. Which on Mellanox is sacrilegious, I know that. Guys were 10 years ahead of everybody else in the 2000s and 2010s. But in-box drivers in recent Linux kernels still great. NFSv4 exports are RDMA-ready but I tweaked nothing.

Nice that you can build your own kernels, a lost art. Been doing that for a long time since late 1990s when the WOLK patchset for 2.2 or 2.4 was larger than kernel itself. I recently switched to building packages as a separate user for added security, after the recent Arch AUR incident. I also build my own kernels because I want a few patches and features. So my linux-custom Arch package has the acs (pci), bridge-mac (mac addr selection), kvm-nopoll (cpu load) patches from Proxmox, ath9k liberated regulatory, powerlimits, watchdog adjustment (slower), antenna reduction patches (I have a mPCIe card with eeprom coded for big antenna), bnx2x patch to enable SGMII 2.5g for fiber SFP CPE, all GPL-only symbol drm and related reverted so Nvidia drivers work better, mlx4 as shown, qxl video driver patch because I use the kernel also in VMs which load X11 and virtio-gpu et.al. sucked, and last not least I have openzfs last stable or whatever is staged in next stable branch merged into kernel (no modules), and arch package also contains zfs-utils. Because I have machines which use zfsbootmenu to boot from ZFS and have root on ZFS. Recently pimped the booster initramfs generator to allow boot from NVDIMM optanes and not suck interacting with ZBM or when zpool.cache is missing or incorrect. It will try hard to boot from something. Also includes microcode for Intel and AMD now. Tested sd-zfs hook, abysmal buffer overruns, and I stopped when I found incompatibility with systemd 261. Refused to debug that. Normal mkinitcpio so-so with zfs hook. Dracut, like sd-zfs, a victim of moving target that is systemd. Constant breakage in new releases and then your machine no longer boots, great. So booster which is written in go-lang my choice to boot from ZFS and have root on ZFS.

Any questions. ;-)

 

[dsrhdev]

hello,
here is my conf, absolutely in-tree driver, no patches, the kernel is `6.19.12`, but the ~same with `7.0.12`-`7.1.5`

Spoiler: mlx4_core modinfo

```
root@s-251:~# modinfo mlx4_core | head -n 5
filename: /lib/modules/6.19.12-200.fc43.x86_64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko.xz
version: 4.0-0
license: Dual BSD/GPL
description: Mellanox ConnectX HCA low-level driver
author: Roland Dreier

```

dual port CX-3 card with

Spoiler: mstflint

```
⬢ [root@toolbx ~]# mstflint -d mlx4_0 q
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: f4521403007f0f40 f4521403007f0f41 f4521403007f0f42 f4521403007f0f43
MACs: f452147f0f41 f452147f0f42
VSD:
PSID: MT_1090120019
⬢ [root@toolbx ~]# mstflint -d mlx4_0 dc | head -n 19 | tail -n 12
[PS_INFO]
Name = MCX354A-FCB_A2-A5
Description = ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name = cx3-1_MCX354A_fdr_09v.prs

[ADAPTER]
PSID = MT_1090120019
pcie_gen2_speed_supported = true
pcie_gen3_speed_supported = true
adapter_dev_id = 0x1003
silicon_rev = 0x01
vdd_change_to_1_offset = 7
```

Spoiler: ibstat for PF

⬢ [root@toolbx ~]# ibstat -d mlx4_0
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0xf4521403007f0f40
System image GUID: 0xf4521403007f0f43
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xf65214fffe7f0f41
Link layer: Ethernet
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0xf4521403007f0f42
Link layer: InfiniBand
```

Spoiler: ibstat for host-visible VF

```
⬢ [root@toolbx ~]# ibstat -d mlx4_1
CA 'mlx4_1'
CA type: MT4100
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x001405006e25784a
System image GUID: 0xf4521403007f0f43
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x7c1fd0fffe3da4ed
Link layer: Ethernet
⬢ [root@toolbx ~]# ibstat -d mlx4_2
CA 'mlx4_2'
CA type: MT4100
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0014050002ef41eb
System image GUID: 0xf4521403007f0f43
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x58667afffee3c93a
Link layer: Ethernet

```

Spoiler: module parameters

```
root@s-251:~# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/boot/ostree/fedora-coreos-da7aa0584065ba26684201dec910464ba1b33ccde2574138347f1d7e956eb264/vmlinuz-6.19.12-200.fc43.x86_64 rw ostree=/ostree/boot.0/fedora-coreos/da7aa0584065ba26684201dec910464ba1b33ccde2574138347f1d7e956eb264/0 ignition.platform.id=metal console=tty console=ttyS1,115200 mitigations=off root=UUID=70009ed2-b231-43e5-8020-89cff867d724 rw rootflags=lazytime,prjquota boot=UUID=62030900-47a3-4a4b-8c24-cd9384e92842 intel_pstate=disable intel_iommu=on iommu=pt pci=realloc intremap=no_x2apic_optout intel_idle.max_cstate=0 processor.max_cstate=0 mlx4_core.num_vfs=4,0,0 mlx4_core.probe_vf=2,0,0
```
on some baseboards you need to have `pci=realloc` + `intremap=no_x2apic_optout` if BIOS doesnt have `SR-IOV` entries

Spoiler: eth/ib links

```
root@s-251:~# ip link show enp1s0
4: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether f4:52:14:7f:0f:41 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 1001, spoof checking off, link-state enable
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 3003, spoof checking off, link-state enable
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
root@s-251:~# ip link show ibp1s0d1
7: ibp1s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/infiniband 80:00:0a:20:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7f:0f:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
Truncated VF list: ibp1s0d1
```

Spoiler: lscpi

```
root@s-251:~# lspci -nnd 15b3:
01:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
01:00.1 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
01:00.2 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
01:00.3 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
01:00.4 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

```

None: in my case inbox opensm doesnt work with IB VF, i just installed nvidia's opensm

Spoiler: opensm

```
root@r-252:~# podman exec -ti nvidia-opensm bash
⬢ [root@toolbx /]# rpm -qa | grep opensm
opensm-libs-5.27.1-202605051000.ee5d4a31.2604086.x86_64
⬢ [root@toolbx /]# rpm -qi opensm-libs
Name : opensm-libs
Version : 5.27.1
Release : 202605051000.ee5d4a31.2604086
Architecture: x86_64
Install Date: Tue Jul 14 15:37:40 2026
Group : System Environment/Libraries
Size : 283916
License : GPLv2 or BSD
Signature :
RSA/SHA512, Sat May 30 20:12:32 2026, Key ID dc726c5e41b9cc50
Source RPM : opensm-5.27.1-202605051000.ee5d4a31.2604086.src.rpm
Build Date : Sat May 30 20:12:32 2026
Build Host : c-234-230-1-007.mtl.nbulabs.nvidia.com
URL : GitHub - linux-rdma/opensm
Summary : Libraries from the opensm package
[...]
⬢ [root@toolbx /]# ldd /usr/bin/opensm
linux-vdso.so.1 (0x00007fbda7b9b000)
libosmvendor.so.14 => /lib64/libosmvendor.so.14 (0x00007fbda77d4000)
libopensm.so.37 => /lib64/libopensm.so.37 (0x00007fbda77b9000)
libosmcomp.so.22 => /lib64/libosmcomp.so.22 (0x00007fbda779a000)
libibumad.so.3 => /lib64/libibumad.so.3 (0x00007fbda778e000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbda74db000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbda74ae000)
libc.so.6 => /lib64/libc.so.6 (0x00007fbda72b1000)
libm.so.6 => /lib64/libm.so.6 (0x00007fbda719a000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbda7b9d000)
⬢ [root@toolbx /]# opensm -v
-------------------------------------------------
OpenSM 5.27.1
-I- Configuration loaded
Log File: /var/log/opensm.log
```

and just drop single `opensm` binnary from `nvidia's opensm-*.rpm` package to `/usr/bin`

upd: also you might need to update selinux policies:

Spoiler: selinux

```
root@s-251:~/git/nvidia-opensm# cat nvidia-opensm-fix.te
module nvidia-opensm-fix 1.0;

require {
class infiniband_endport { manage_subnet };
class infiniband_pkey access;
type iscsid_t;
type systemd_modules_load_t;
type unlabeled_t;
}

#============= iscsid_t ==============
# Разрешаем ядру (контекст хранилища) доступ к неразмеченным ключам фабрики
allow iscsid_t unlabeled_t:infiniband_pkey access;

#============= systemd_modules_load_t ==============
# Разрешает systemd-modules управлять подсетью на неразмеченных портах InfiniBand
allow systemd_modules_load_t unlabeled_t:infiniband_endport { manage_subnet };
```

upd2: oh, yes all NFSoRDMA, iSER and NVMEoF work good over VF

 

[naptastic]

Thanks for sharing. I will have to reconsider and try it out. How fast do your file transfers max out?

Most of my hosts are Linux but my one SMB is just Windows.

The fastest drives I have in my lab are all PCIe 3.0 NVMe drives, so with NFS/RDMA or SRP, it's as fast as local.

 

[naptastic]

hello,
here is my conf, absolutely in-tree driver, no patches, the kernel is `6.19.12`, but the ~same with `7.0.12`-`7.1.5`

```
root@s-251:~# ip link show enp1s0
4: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether f4:52:14:7f:0f:41 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 1001, spoof checking off, link-state enable
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 3003, spoof checking off, link-state enable
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
root@s-251:~# ip link show ibp1s0d1
7: ibp1s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/infiniband 80:00:0a:20:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7f:0f:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
Truncated VF list: ibp1s0d1
```

Ok, the module options you provided worked and have been most helpful.

I notice your IB MTU is 2K. Do you know if it's possible to get 4K MTU with this setup? It seems like it might be an HCA limitation. If I set `mtu=5` in partitions.conf (for 4K) then the subnet manager says there's a port MTU mismatch between the HCA and the switch it's connected to. (Use `opensm -D 0x07` and have something watching /var/log/opensm.log because it becomes very noisy.) I have to turn SR-IOV off again to get the Infiniband interface to join the subnet at 4K.

Thanks,
Nap

 

[dsrhdev]

Ok, the module options you provided worked and have been most helpful.

I notice your IB MTU is 2K. Do you know if it's possible to get 4K MTU with this setup? It seems like it might be an HCA limitation. If I set `mtu=5` in partitions.conf (for 4K) then the subnet manager says there's a port MTU mismatch between the HCA and the switch it's connected to. (Use `opensm -D 0x07` and have something watching /var/log/opensm.log because it becomes very noisy.) I have to turn SR-IOV off again to get the Infiniband interface to join the subnet at 4K.

hello,
1) the module parameters triplet is important: you can use the third one only when both ports are in ETH mode
2) i guess the 2/4k MTU limitation is about of CX-3 VF using. on RDMA it has practically ~no impact, you can try to use CONNECTED mode if MTU matters (i.e. for non-RDMA interactions, like iperf/regular ip traffic)