Mellanox branded vs OEM

[turgin]

Looking at jumping on the cheap 10G bandwagon with the Mellanox ConnectX-2 cards. They will be used in two ESXi 6 hosts direct connected via DAC for a Starwind VSAN sync and iscsi network.

Is there any value in getting the actual Mellanox MNPA19-XTR vs a branded HP or Dell for use in a whitebox server?

Just wondering primarily for flashing firmware and such.

 

[i386]

Thee is no difference once you flash the mellanox firmware, they are all the same.

 

[_alex]

I would conaider to go straight towards 40G and skip 10G, for little difference on the investment.

 

[darklight]

According to my experience, RDMA gives a lot of features and there is a big difference with or without it. Especially with Starwind, you will definitely achieve better performance. So it would be better to use newer Connect X 4 which is RDMA capable cause even Connect X 3 doesn't support RDMA with ESXi anymore.

 

[turgin]

Thanks for the replies. This is just a small home network. I only have 6x2TB in RAID10 on an IBM M5015(LSI 9260) in each host. I currently use 4 x 1G for sync and 4 x 1G for iscsi that preforms well enough. I just want to reduce and simplify cabling.

I think the old Connect-X 2 will do well enough for my use. I'll put 2 in each host and configure sync and iscsi port groups on each one.

 

[_alex]

to be precise, it's not the CX-3 not supporting anything - it's ESXi that doesn't provide drivers ...
Anyway, also 40G Ethernet is still faster than 10G and can be had for not much more ...

 

[i386]

According to my experience, RDMA gives a lot of features and there is a big difference with or without it. Especially with Starwind, you will definitely achieve better performance. So it would be better to use newer Connect X 4 which is RDMA capable cause even Connect X 3 doesn't support RDMA with ESXi anymore.

RDMA works with cx3 and cx2 cards, if it's not enabled check your driver settings.

Edit: CX-2

 

[gzorn]

YMMV.
I bought a Dell-branded CX-312 (10gbe dual port) and flashed it with the stock firmware. After the flash, the activity LEDs on the board flash constantly regardless of port activity/link.

Otherwise, the NIC works on my Supermicro X11-SSi-LN4F but doesn't work on a Gigabyte consumer motherboard (won't complete POST). Another thread (https://forums.servethehome.com/ind...ing-boot-in-one-server-but-not-another.19787/) on the board suggests that it might be possible to adjust some firmware settings to get it working, but I haven't bothered to investigate.
(EDIT - corrected thread link)

 

[fohdeesha]

YMMV.
I bought a Dell-branded CX-312 (10gbe dual port) and flashed it with the stock firmware. After the flash, the activity LEDs on the board flash constantly regardless of port activity/link.

That's what happens when you flash pre-built stock firmware to an OEM card that's had some hardware changes (like the dell cards). You need to extract the firmware.ini settings dell uses, then build a new custom firmware image using them - then flash that image and it'll work perfectly fine.

You'll need to flash the card back with the stock dell firmware before doing this, otherwise the firmware.ini exporting step will just save out the stock (incompatible) mellanox firmware.ini settings - https://forums.servethehome.com/ind...x-3-to-arista-7050-no-link.18369/#post-178015

 

[gzorn]

You'll need to flash the card back with the stock dell firmware before doing this, otherwise the firmware.ini exporting step will just save out the stock (incompatible) mellanox firmware.ini settings - https://forums.servethehome.com/ind...x-3-to-arista-7050-no-link.18369/#post-178015

Thanks for the pointer, @fohdeesha . I might have saved the old dell firmware image before the overwrite. TBH, I don't really care about the LED's. The card works fine otherwise.

 

[fohdeesha]

You can always just download the dell firmware from dells site, you just need to get it back to the stock dell firmware to extract their version of firmware.ini - if it's different enough that the LED's are going wonky, I'd be worried it's also impacting performance - there's a TON of config registers in that file, ASIC/CPU clock, PHY clock, BAR space, etc. I wouldn't be surprised if that's why it doesn't work on the other system

 

[turgin]

Great info.

@fohdeesha You specifically mention Dell, but do you know about HP branded ones?

Sounds like it may be worth the couple dollars difference and get Mellanox branded just to save the headache.

 

[fohdeesha]

so far dell rebrands are the only cards I've ran into that were actually different, all the HP Sun and IBM cards I've used were identical and could just be flashed with prebuilt stock firmware, but I can't say for 100% sure as I haven't used ALL models ever released. Even if you do need to build firmware, its like 5 commands and 2 minutes to do so, so up to you how much your time is worth

 

[marv]

This might be unrelated, but my Dell i350 NIC refused to POST in desktop boards too. Issue is with one of the smbus pins on PCIe connector which had to be covered by tape for the card to boot in dektop motherboard (sorry, I dont remember which one and cannot find the blogspot post now).