How to pick a SFP+ Adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

amalurk

Active Member
Dec 16, 2016
311
116
43
102
I have a few lower power Proxmox boxes I want to connect with SFP+ (for Ceph backend) that have a PCIe3 x4 or x16 slot open (half height bracket). I will use 1 or 2m DACs.

I don't know what cards to get, Connect 3 non, pro? Solarflare? Supermicro ones? Intel 520?

Basically there are too many ebay options and I don't know how to choose one. I don't want to get stuck with an IB card or one I have to flash to get to work and don't want to have to use some special kernel or driver under Proxmox.

I would like to stay under $60/card. Maybe dual port but do not have switching capacity for that and don't need that on front end. Suggestions and why?

Thanks!
 

CyklonDX

Well-Known Member
Nov 8, 2022
784
255
63
careful with mellanox used cards from ebay.

x520 are ok


you can use this for selection of modules...

keep in mind, one goes to switch (and often they are oem locked - so you need to choose one for that switch)
most sfp+ cards like x520 are also locked to intel brand; so ones with IN letter marked on them... and so on... the web page will help you select correct ones depending on brand...


typically you will have following letters by model

cisco = null
jupiter = JU
intel = IN
ibm = IB
generic = GE
HP = HP

and so on...
 

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
The CX3 is a good choice. Excellent compatibility and extremely common. It should work with Proxmox without any special drivers, as long as its in Ethernet mode (see below)

The cards are capable of both InfiniBand and Ethernet. The "VPI" firmware is switchable using a command-line utility. The "EN" firmware is Ethernet-only. If you want to avoid this, get a card with EN firmware or one that specifically says it's in Ethernet mode. Switching the mode is pretty easy though. Being open to the mode switching will help you get the best deal.

The "Pro" version supports RoCEv2 (RDMA-over-Ethernet which can be routed between subnets. The Pro version also has VXLAN encapsulation offload, which is for overlay networking.

I got mixed results for whether Ceph supports RoCE (RDMA) but some say it does but I suspect it's rare: How to enable Ceph with RDMA

If you will never use RoCEv2 or VXLAN offload (I suspect not), the non-Pro is probably fine. Maybe someone else has an opinion on this?

The used market is good for these cards, under $60 is reasonable. Beware of fakes, this can happen. The CX3s do need some airflow to stay cool, or they can overheat and drop off the system. With DACs this is less of an issue, optics get a lot hotter.

Single vs dual port is your choice. You can do NIC teaming, or connect to 2 different boxes without a switch, or re-purpose them later. With one port inactive, the power draw should be similar to a single port card.

I run the QLogic QLE8142-IBMX for 10G. They are dual port and rock-bottom dirt cheap, and just the natural cooling in my desktop case is enough. But they are less common, may not work with ESXi past 6.7, and don't support RoCE or VXLAN or SR-IOV.
 
  • Like
Reactions: amalurk and altmind

kpfleming

Active Member
Dec 28, 2021
383
205
43
Pelham NY USA
I recently purchased an HP 560SFP+ dual-port card; it's working fine with Brocade-coded AOCs and was under US$50. I believe it's the same chipset as the Intel X520.
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
SolarFlare 7000-series (SFN7002, SFN7122, also under S7120) are super cheap, run relatively cool, and work fine at 10GbE. No RoCE of any sort (their alternative to RDMA was their OnLoad accelerated TCP stack).

I've also run CX311/312, CX353/354 with QSA, and X520; all excellent cards with broad SFP support (under Linux).

I'm pretty sure RDMA on ceph is dead, unfortunately; Mellanox/NVIDIA dropped the project.

I'm not fond of the dirt cheap QLogic or Emulex cards.
 
  • Like
Reactions: amalurk