Which (inexpensive) 10Gbe sfp+ NIC is recommended for Linux and FreeBSD?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Ciciban

New Member
Feb 5, 2025
11
0
1
As tittle says, for which NIC should I search ebay?
I need something that will work with OpnSense, Proxmox and TrueNas core. On the other end I have HORACO 2.5GbE Managed Switch 8 x 2.5G + 1 x 10G SFP+ and mikrotik CRS 310. So it needs to work with either and ideally with both. But this probably depends on transceivers?

What would be the best option and to be available for reasonable price in EU ?
 

louie1961

Active Member
May 15, 2023
384
169
43
I have one of these cheap X520 clones in my Proxmox box https://www.amazon.com/dp/B01DCZCA3O?th=1

I have been very happy with it. I can't say if it works with BSD based softwares (OpnSense/TrueNAS core) but it worked no sweat in my TrueNAS scale box as well. I have it connected to one of these https://www.amazon.com/dp/B0CCDD4CLC?th=1

I have used DAC as well as AOC and both seemed to work the same, no issues.
 
  • Like
Reactions: Ciciban

blunden

Well-Known Member
Nov 29, 2019
986
314
63
People seem to like X520 because it's dirt cheap if buying used. The downside is higher power draw, that it's limited to PCI-E 2.0 and the fact that it's basically End-of-Life or very close to it. I would expect drivers for it to be supported long after Intel drops official support, but the other limitations are worth taking into account.

Other alternatives are Mellanox ConnectX-3 and 4 as well as the Intel X710, although the latter is more expensive.

Worth knowing with Intel NICs is that they tend to be vendor locked by default when it comes to SFP+ transceivers. Most can be unlocked with either a driver flag or by flipping some bits in the card's EEPROM.

NOTE: I would double check the driver status on BSD before buying. It's not something I keep track of. :)
 
Last edited:
  • Like
Reactions: Ciciban

Ciciban

New Member
Feb 5, 2025
11
0
1
I have one of these cheap X520 clones in my Proxmox box https://www.amazon.com/dp/B01DCZCA3O?th=1

I have been very happy with it. I can't say if it works with BSD based softwares (OpnSense/TrueNAS core) but it worked no sweat in my TrueNAS scale box as well. I have it connected to one of these https://www.amazon.com/dp/B0CCDD4CLC?th=1

I have used DAC as well as AOC and both seemed to work the same, no issues.
What does it mean clone ?
Original chip, but noname vendor of the card ? Drivers and FW are still from intel ? Is fw freely available from intel ?
 

louie1961

Active Member
May 15, 2023
384
169
43
What does it mean clone ?
Original chip, but noname vendor of the card ? Drivers and FW are still from intel ? Is fw freely available from intel ?
Yes, no name manufacturer. I have no idea on the other questions. I just popped the card in and started using it. Haven't needed to update anything yet.
 
  • Like
Reactions: Ciciban

Ciciban

New Member
Feb 5, 2025
11
0
1
X520 should work on BSD without issues, according to internet.

But there are many mentions that intel cards don't play with non intel transceivers for fiber. And DA version should only work with DCA cables.
This seems to be quite a limitation
 

sko

Active Member
Jun 11, 2021
383
238
43
From a FreeBSD point of view:
1st choice: Chelsio
2nd choice: Mellanox
3rd choice: Intel

Chelsio has the best driver support and even SR-IOV "just works"™. They are a tad more expensive than Mellanox, but the T520 based models are still dirt cheap. Th 6th generation cards which are also 25G capable are still more expensive (but usually still way under 100$), but relatively uncommon in europe, i.e. harder to find as used parts and usually very expensive from EU sellers.

Mellanox Cx3 and 4 can be found dirt cheap everywhere as white-label variants from china. For homelab/non-critical systems those are perfectly fine (I never had one of those 'white box' NICs or even HBAs fail) and can be found below 30$. Their drivers also 'just work'™, but SR-IOV is currently buggy for Cx3 (had it running a year ago, so it seems to be a regression and might be fixed soon).
To switch between ethernet and infiniband you don't have to fiddle around with firmware like on linux - just load the specific driver (mlx4en or mlx4ib) and the card runs in the respective mode. You can even switch at runtime by unloading the one and loading the other kernel driver.

Intel drivers are also quite 'uneventful' (except for SR-IOV), but Intel cards are the most picky ones when it comes to transceivers and the failure modes they show with off-brand transceivers are quite annoying to say the least... i.e. they sometimes seem to work, until the NIC (or one port) just vanishes after a reboot. Hard to diagnose, especially due to the fact that it "sometimes seem to work" e.g. after switching the transceivers around. You *need* to put intel-coded transceivers in them to have them work reliably. Chelsio and Mellanox absolutely don't care what transceivers you throw at them, they just work...


Regarding 2.5Gbps: thats not part of the SFP+ standard (just as all copper transceivers because they have way too high power draw), so you need transceivers that handle the negotiation and fake a 10G link to the SFP port.
 

blunden

Well-Known Member
Nov 29, 2019
986
314
63
What does it mean clone ?
Original chip, but noname vendor of the card ? Drivers and FW are still from intel ? Is fw freely available from intel ?
Intel sells the chip used by the X520 separately as the Intel 82599ES. Manufacturers use those to create a very similar product to the official Intel X520 NICs. Drivers are from Intel and firmware also presumably so, although I'm not sure whether Intel's tools will let you flash their official firmware releases on these cards.

OEM models of Intel NICs from Dell, HP, etc. generally don't accept the generic Intel firmware unless you trick the tool to cross-flash it.

X520 should work on BSD without issues, according to internet.

But there are many mentions that intel cards don't play with non intel transceivers for fiber. And DA version should only work with DCA cables.
This seems to be quite a limitation
Like I said, you can usually unlock them. There is a thread about that here on the forum. The reason that the unlock is implemented as an EEPROM toggle is because Intel want to allow the OEMs to decide whether their cards should be vendor locked or not.

I unlocked my X710-DA2 NIC, but ended up buying Intel coded fiber transceivers from Ipolex (seems to be rebranded 10Gtek) anyway since it didn't cost anything extra.

Not sure what you mean by DCA cables. Do you mean DACs?
 

MountainBofh

Beating my users into submission
Mar 9, 2024
398
291
63
The 3rd party 82599ES cards are unlocked in my experience. They work well in Windows and Linux with Intel's standard drivers. I've tested all sorts of random transceivers in them and haven't seen any that did not work.
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,290
864
113
Stavanger, Norway
intellistream.ai
To get SR-IOV to work correctly on Linux with Mellanox ConnectX-3, do the following

Make sure SR-IOV is enabled in bios. And use mstconfig to enable SR-IOV
Code:
sudo mstconfig -d 82:00.0 set SRIOV_EN=true NUM_OF_VFS=63
Use
Code:
/sbin/lspci -d 15b3:
to find the pci address for your nic.

Add "intel_iommu=on iommu=pt" to /etc/default/grub
Example
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
Update kernel config
Debian
Code:
sudo update-grub
RHEL
LEGACY BIOS
Code:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
UEFI
Code:
sudo grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg

Edit /etc/initramfs-tools/modules
Add this line:
mlx4_core num_vfs=16,16 port_type_array=2,2 probe_vf=16,16

This will add 8 virtual ports on each physical port, total 16 ports

Update initram
sudo update-initramfs -u

Reboot

Adding this to the /etc/modprobe.d/mlx4.conf file as most examples on google tell you, will cause issues where the virtual functions doesn't get properly initialized. I am not sure exactly why, but it seems like loading the driver very early in the boot process helps making sure the virtual functions are working properly.
 
  • Like
Reactions: nexox and richardm

bugacha

Active Member
Sep 21, 2024
472
141
43
As tittle says, for which NIC should I search ebay?
I need something that will work with OpnSense, Proxmox and TrueNas core. On the other end I have HORACO 2.5GbE Managed Switch 8 x 2.5G + 1 x 10G SFP+ and mikrotik CRS 310. So it needs to work with either and ideally with both. But this probably depends on transceivers?

What would be the best option and to be available for reasonable price in EU ?

Mellanox ConnectX-4 Lx its 25 but its cheap

IMO no point buying ConnectX-3 - its too old
 
  • Like
Reactions: pimposh and nexox

TrevorH

New Member
Oct 25, 2024
28
11
3
Solarflare SFN8522's are generally dead cheap on eBay. Seem to work well for me under RHELatives.
 

richardm

Member
Sep 27, 2013
51
16
8
As tittle says, for which NIC should I search ebay?
I need something that will work with OpnSense, Proxmox and TrueNas core.
I have an 020NJD 0MRT0D which (I think) is Dell's version of the MCX4121A-ACAT (ConnectX-4). I think I paid around $25 on eBay. I can confirm everything works great with Debian so it should be fine with Proxmox as well. SR-IOV was no issue.

With FreeBSD I've used only SR-IOV passthrus from VMware ESXi -- I've never used the card "bare metal." This aspect has been flawless with 14.2-RELEASE. FreeBSD's mlx5en driver has delightfully few bug reports.

Edit: The issue with creating SR-IOV VFs in FreeBSD may have been recently fixed.
 
Last edited:

sko

Active Member
Jun 11, 2021
383
238
43
Edit: The issue with creating SR-IOV VFs in FreeBSD may have been recently fixed.
AFAIK the regression affected only the mlx4 driver for Connect X3 based cards, not mlx5en (CX4-6).
There are no freezes or other fatal error states, the driver simply doesn't create the 'mlxenN' device nodes in /dev/iov, so one could simply not create any VFs.

With intel X5xx cards the problem is more severe - one can create VFs, but as soon as those are used in any way (e.g. adding an address to a VF) the whole interface (PF) goes silent. There seems to be some bug in the firmware that is triggered by the FreeBSD driver, because it requires a reset of the NIC (reboot, sometimes cold) to get it back to a working state.

But TBH, given that CX4 cards are available at the same price point (or even less: sub-30$ offers are pretty common for CX4), allow for 25G upgrade (ACAT/XCAT only have different firmware but identical hardware) and have much better offloading and less power consumption compared to X5xx based cards, there's really no reason to still buy those ancient NICs...
 
  • Like
Reactions: blunden

BackupProphet

Well-Known Member
Jul 2, 2014
1,290
864
113
Stavanger, Norway
intellistream.ai
Mellanox ConnectX-4 Lx its 25 but its cheap

IMO no point buying ConnectX-3 - its too old
I agree, but 25G/100G switches are still slightly expensive, especially the good ones.
40/56GbE is dirt cheap today. Switches, cables, transceivers and nics. For homelab, 40/56G should be plenty.
The biggest issue with the ConnectX3 generation is RoCE/RDMA. Either it works, or not. I am wondering if some nics maybe has firmware config issue. CX4 generation works great with RoCE. Anyway, most CX3 40G nics support VPI, which means you can run IB instead, then RDMA work flawless. Combine this with Mellanox SX6036, you get great speed for great value.
Things get more interesting with CX5 and CX6 where you can get NVMe-oF and other kinds of acceleration/offload.
 

bugacha

Active Member
Sep 21, 2024
472
141
43
I agree, but 25G/100G switches are still slightly expensive, especially the good ones.
40/56GbE is dirt cheap today. Switches, cables, transceivers and nics. For homelab, 40/56G should be plenty.
The biggest issue with the ConnectX3 generation is RoCE/RDMA. Either it works, or not. I am wondering if some nics maybe has firmware config issue. CX4 generation works great with RoCE. Anyway, most CX3 40G nics support VPI, which means you can run IB instead, then RDMA work flawless. Combine this with Mellanox SX6036, you get great speed for great value.
Things get more interesting with CX5 and CX6 where you can get NVMe-oF and other kinds of acceleration/offload.

ConnectX4 Lx will happily run in 10gbe mode against 10gbe switch
 

bugacha

Active Member
Sep 21, 2024
472
141
43
SRV-IO works out of the box with ConnectX4 Lx no need for additional settings.

Lm-sensors even support reading temperature off the card too
 
  • Like
Reactions: BackupProphet

i386

Well-Known Member
Mar 18, 2016
4,647
1,768
113
36
Germany
I am wondering if some nics maybe has firmware config issue.
working rdma (I assume for smb) requires:
- correct os (windows 10/11 pro for workstations/enterprise, windows server >=2012 R2)
- correct bios settings
- correct firmware/driver settings
It doesn't helpt that mlnx oefd distro changes some firmware/driver settings on different os during installation, which makes troubleshooting a pain in the ass