Any downsides with ConnectX-3?

[monotux]

Hi,

I recently put my first 10 Gbps capable switch into use at home, a ICX6450-24p. I had two ConnectX-3 cards already and plugged those into the switch with DACs, and it seems to work perfectly fine.

I'm now looking into buying two more 10 Gbps cards for two more nodes, but before going with another set of ConnectX-3 I'm looking for advice regarding 10 Gbps adapters - the X-3 are cheap and I suspect that's for a reason.

What would I gain with going with some other adapters? Are there any obvious downsides to the X-3s?

If it matters, one of the new nodes will use DAC, and the other new node might use existing CAT5 cabling (hopefully I'll make a new CAT6 cable run) + SFP+ to RJ45 adapters on both sides. If I move my desk to my basement I might get away with another DAC run.

 

[Rand__]

The main thing might be that they are old and out of support (except Pro iirc).
So at some point you might not get new drivers or you might loose OS support for them.

Else they are brilliant

 

[monotux]

The main thing might be that they are old and out of support (except Pro iirc).
So at some point you might not get new drivers or you might loose OS support for them.

Else they are brilliant

Would I gain anything in power consumption or so with something more modern? I've understood that 10GBase-T is more power hungry than SFP+, but not sure about the difference between generations of SFP+.

I'm looking to place one of the cards in my firewall, but it seems the X-3 has hardware support for vlans et c, which I guess will be nice.

 

[i386]

What would I gain with going with some other adapters? Are there any obvious downsides to the X-3s?

Like @Rand__ said they are end of life and driver support can be removed from newest linux kernels
Other downsides are:
- "missing" features like offload technologies for nvme, encryption etc.
- no roce 2.0 (cx-3 pro supports roce 2)

 

[Rand__]

Would I gain anything in power consumption or so with something more modern? I've understood that 10GBase-T is more power hungry than SFP+, but not sure about the difference between generations of SFP+.

Possibly, but i never measured.
Not sure its worth it vs the expected acquisition cost... for the price difference you will be able to run the CX-3 for years (even if we only assume $30 vs $60 and $10 extra power/year).

 

[llowrey]

I have a mixture of CX-3 (non-pro) and CX-4 and have observed a couple of differences in behavior.

For some strange reason, Network Manager (at least RHEL/CentOS/Fedora) has a hard time with linking at 56GbE with the CX-3 cards. On boot, it'll link up, almost immediately go down, and will come up again about 1 minute later and stay up. I don't observe this behavior at 40GbE or with the CX-4 cards.

SR-IOV is much better with the CX-4. With the CX-3 I would regularly see VFs freeze up due to buffer errors. Another oddity is that the interface stats for the PF would be the total for the port including all the traffic from the VFs. With the CX-4 the PF stats only reflect the traffic that was directly put on the PF interface and does not include the traffic from any of the VFs.

 

[monotux]

I'll be using the cards with a mix os Linux, FreeBSD/FreeNAS and potentially Windows 10. So old drivers might not be terrible.

Any recommendations for other 10 Gbps cards, reasonably priced and more modern?

 

[i386]

10 Gbps cards, and more modern?

10GBE is kinda "dead" for enterprise and replaced by 25GBE.

Do you want sfp+ and its variants or is 10GBase-T also acceptable?

 

[monotux]

10GBE is kinda "dead" for enterprise and replaced by 25GBE.

Heh, I'm late to the game

Do you want sfp+ and its variants or is 10GBase-T also acceptable?

Well, both. SFP+ and DACs are convenient in my 'lab', but it's much easier to make a CAT6 run to my current office.

 

[dragonian]

I can't imagine linux removing support for the ConnectX-3 cards.. how many ancient generations of intel chips are still supported?
Incidentally ConnectX3 is supported by the mlx4 linux driver.

I don't feel bad picking up cheap SFP+ cards, then you have options - you can rock the 10GB-T or the DAC/Optics as needed.