Any downsides with ConnectX-3?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

monotux

Member
Oct 23, 2019
57
34
18
Kiruna, Sweden
www.monotux.tech
Hi,

I recently put my first 10 Gbps capable switch into use at home, a ICX6450-24p. I had two ConnectX-3 cards already and plugged those into the switch with DACs, and it seems to work perfectly fine.

I'm now looking into buying two more 10 Gbps cards for two more nodes, but before going with another set of ConnectX-3 I'm looking for advice regarding 10 Gbps adapters - the X-3 are cheap and I suspect that's for a reason.

What would I gain with going with some other adapters? Are there any obvious downsides to the X-3s?

If it matters, one of the new nodes will use DAC, and the other new node might use existing CAT5 cabling (hopefully I'll make a new CAT6 cable run) + SFP+ to RJ45 adapters on both sides. If I move my desk to my basement I might get away with another DAC run.
 

Rand__

Well-Known Member
Mar 6, 2014
6,632
1,767
113
The main thing might be that they are old and out of support (except Pro iirc).
So at some point you might not get new drivers or you might loose OS support for them.

Else they are brilliant :)
 
  • Like
Reactions: monotux

monotux

Member
Oct 23, 2019
57
34
18
Kiruna, Sweden
www.monotux.tech
The main thing might be that they are old and out of support (except Pro iirc).
So at some point you might not get new drivers or you might loose OS support for them.

Else they are brilliant :)
Would I gain anything in power consumption or so with something more modern? I've understood that 10GBase-T is more power hungry than SFP+, but not sure about the difference between generations of SFP+.

I'm looking to place one of the cards in my firewall, but it seems the X-3 has hardware support for vlans et c, which I guess will be nice.
 

i386

Well-Known Member
Mar 18, 2016
4,240
1,546
113
34
Germany
What would I gain with going with some other adapters? Are there any obvious downsides to the X-3s?
Like @Rand__ said they are end of life and driver support can be removed from newest linux kernels
Other downsides are:
- "missing" features like offload technologies for nvme, encryption etc.
- no roce 2.0 (cx-3 pro supports roce 2)
 
  • Like
Reactions: monotux

Rand__

Well-Known Member
Mar 6, 2014
6,632
1,767
113
Would I gain anything in power consumption or so with something more modern? I've understood that 10GBase-T is more power hungry than SFP+, but not sure about the difference between generations of SFP+.
Possibly, but i never measured.
Not sure its worth it vs the expected acquisition cost... for the price difference you will be able to run the CX-3 for years (even if we only assume $30 vs $60 and $10 extra power/year).
 
  • Like
Reactions: monotux

llowrey

Active Member
Feb 26, 2018
167
140
43
I have a mixture of CX-3 (non-pro) and CX-4 and have observed a couple of differences in behavior.

For some strange reason, Network Manager (at least RHEL/CentOS/Fedora) has a hard time with linking at 56GbE with the CX-3 cards. On boot, it'll link up, almost immediately go down, and will come up again about 1 minute later and stay up. I don't observe this behavior at 40GbE or with the CX-4 cards.

SR-IOV is much better with the CX-4. With the CX-3 I would regularly see VFs freeze up due to buffer errors. Another oddity is that the interface stats for the PF would be the total for the port including all the traffic from the VFs. With the CX-4 the PF stats only reflect the traffic that was directly put on the PF interface and does not include the traffic from any of the VFs.
 
  • Like
Reactions: monotux

monotux

Member
Oct 23, 2019
57
34
18
Kiruna, Sweden
www.monotux.tech
I'll be using the cards with a mix os Linux, FreeBSD/FreeNAS and potentially Windows 10. So old drivers might not be terrible.

Any recommendations for other 10 Gbps cards, reasonably priced and more modern?
 

dragonian

Member
Jan 3, 2020
47
30
18
I can't imagine linux removing support for the ConnectX-3 cards.. how many ancient generations of intel chips are still supported?
Incidentally ConnectX3 is supported by the mlx4 linux driver.

I don't feel bad picking up cheap SFP+ cards, then you have options - you can rock the 10GB-T or the DAC/Optics as needed.
 
  • Like
Reactions: monotux