Which is better 100G nic? Mellanox ConnectX-4 or Intel E810

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

anthros

New Member
Dec 16, 2021
20
4
3
Portland, OR USA
i dont need any of those as there is zerto windows in my infra
this will be used for server communication running linux
even when i do setup nfs/san the server and clients will still be linux

so again is the Mellanox ConnectX-4 Dual x 100G a good choice to go with that also supports 40G?
My, aren’t we testy? ;)

Maybe you’ll never use Samba, but you almost certainly want to use NFSoRDMA. If so, then the cards’ RDMA capabilities are something you need to consider. AFAIK, e810 cards support iWARP, a dying/dead RDMA implementation proprietary to Intel. Mellanox ConnectX-4 cards support RoCE, which is the leading standard and which is being implemented by more and more vendors beyond Mellanox/NVIDIA.

The Mellanox card supports both 40G and RoCE, while the intel card supports neither. The Mellanox card is a slam-dunk for your use case, at least in my opinion.

[Edit: I was mistaken: the Intel card indeed supports RoCE V2. Thanks to MountainBofh for the correction].
 
Last edited:

uberguru

Active Member
Jun 7, 2013
456
31
28
is there much of difference between connectX-4 and connectX-5?
should i just get the X4 oris the X5 worth getting?



VS

 

nexox

Well-Known Member
May 3, 2023
1,271
587
113
The ConnectX 5 has more features that you likely won't need (if they're even supported in any publicly available driver,) other than that they work very similarly for me. You need to compare the data sheets to see if anything is worth the price difference to you, then do some research to see if those features can be used on the platform you intend to run. I know I wasted quite a lot of time trying to use the supposedly-supported NAT offload on my ConnectX 5 and all I ever got was a generic error message that lead me to an impenetrable maze of function pointers in the kernel source.
 

uberguru

Active Member
Jun 7, 2013
456
31
28
i decided to go with the connectx-5 since i found a dual one for $125

now issue is theDAC cable to use, they are expensive man
and am not sure how to select DAC cable as they have different models all over i get lost trying to know which is which to buy
why i just default to mikrotik ones since they only have just one model for each speed and easy to go with

anyone know of cheap 40/100G dac cable to buy that will work from dell server to mikrotik switch
3 meters version
 

uberguru

Active Member
Jun 7, 2013
456
31
28
@MountainBofh will these work for the mikrotik switch since you mentioned something about vendor locking
can you expand on that more? the cable will not work for other brand switches?

the one i would have bought is XQ+DA0003 | MikroTik but most sellers either add shipping or total is over $60 and that is crazy for a cable. i dont mind used

 
Last edited:

MountainBofh

Beating my users into submission
Mar 9, 2024
321
239
43
100gb DAC cable should also work for 40gb. Look for the term DAC, that stands for direct attached cable.

Vendor locking refers to the practice that some switches and network cards are deliberately crippled such that they will only work with transceivers made by the same vendor. You can get around this by getting an unlock code from the vendor, or getting a transceiver that is coded to pretend to be that vendor's transceiver.


The only network cards that did vendor locking were the Intel X520 and X710, and this can be disabled.

On the switch side, a lot of the high end enterprise switches vendor lock, but most if not all of them provide for an option to turn it off.

Mikrotik does not do this, so you can use any cable or transceiver you want. Mellanox also does not vendor lock, so you can use any cable or transceiver with Mellanox products.
 

uberguru

Active Member
Jun 7, 2013
456
31
28
1. so which DAC cables are the ones that bend easily?
2. what is difference between active vs passive DAC cable?

like this ebay listing says active cable and from pics seem cable is easily bendable compared to standard dac cables

 

MountainBofh

Beating my users into submission
Mar 9, 2024
321
239
43
Active cables have some sort of amplifier to carry the signal farther. Normal passive cables only go a fairly short distance (like 5-7 meters).

Active cables are usually a fiber cable with transceivers on the end (as what you linked to from Ebay). Fiber is much more flexible, but costs more as it requires transceivers at each end.

DAC cables tend to be fairly stiff, but they're cheap as they don't require transceivers.
 

Stephan

Well-Known Member
Apr 21, 2017
1,014
783
113
Germany
now issue is theDAC cable to use, they are expensive man
Maybe old tip, read firmware release notes for your card, these should contain a list of verified cables including OEM part number. If you don't need it tomorrow, hack the numbers into ebay and wait a little. Recently snipered a 30 meter QSFP+ FDR 56G cable this way for 30 fiat including shipping.
 
  • Like
Reactions: lightsword

uberguru

Active Member
Jun 7, 2013
456
31
28
Maybe old tip, read firmware release notes for your card, these should contain a list of verified cables including OEM part number. If you don't need it tomorrow, hack the numbers into ebay and wait a little. Recently snipered a 30 meter QSFP+ FDR 56G cable this way for 30 fiat including shipping.
just curious, what is your usecase for using infiniband?
 

Stephan

Well-Known Member
Apr 21, 2017
1,014
783
113
Germany
No infiniband, I use the cable for 56G ethernet between CX3 in a Linux workstation and a SX6012 switch.
 

uberguru

Active Member
Jun 7, 2013
456
31
28
No infiniband, I use the cable for 56G ethernet between CX3 in a Linux workstation and a SX6012 switch.
but the switch is infiniband right? so what you mean no infiniband?
my main point was just your use-case, just curious of use-cases of infiniband instead of regular ethernet