EXPIRED Dell MLX CX4 100G QSFP28 LP €160 - Gone

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Hi,

I have been sitting on those for a while deciding whether I need/want/can afford more than the two I got but I think not so up for grabs now:) Also been waiting since I had not tested them end to end (see below) but I think they should be fine (1 month return, new item so has warranty as well).

Disclaimer: I am not affiliated and not benefiting from this but I have discussed intensively with the seller so I have a lot of info to share.

1. If the title is not clear - these are *new* Mellanox CX4 - 456 PCIe3 x16 EDR 100GBs VPI Cards with Dual Ports with LP adapter. You *do* need an electrical x16 slot for those to work properly

2. Those are the cheapest I have ever seen on dual port EDR cards, those are DELL OEM but flash easily to Mellanox standard firmware (done one of my two without issues at this point in time). The OEM part number is rather uncommon, have not been able to find it referenced anywhere, there are o/c several possible implications.
I have not actually put mine into my cluster yet (due to my issue with the Fujitsu board i posted in the mainboard section) but I have seen no issues whatsoever with these cards, so from my point of view they are ok.

2.Total 20 are available at this price point, if you get 5+ they are willing to do 5% discount but I'd request to not do bulk buying so everybody can get some :) I post this on a Saturday intentionally to hinder this;)

Shipping to EU is available on Ebay directly, international shipping possible on request.
VAT deductible on request as well. Return possible,

Dell Plus QSF MLX L CX4 100G HWTYK | eBay

Have fun.
 

im2geek4you

Member
Feb 27, 2017
30
10
8
43
I'm not angry, it was supposed to be a worried face. I've beem looking out for one of these for some time.
Just checked the Mellanox doc's and it can only be this model. Can you confirm?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Will have to check the logs on my flash systems, but iirc I verified 100GB as well as VPI capability (as per windows) so that should be the correct model.
In the end cross-flashing still seem to be possible so I wouldn't worry too much about it:)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Sorry for the delay in feedback, had some other issue to fix before I could get the old logs:

I flashed the cards with
flint -d /dev/mst flint/mt4115_pciconf0 -i fw-ConnectX4-rel-12_23_1020-MCX456A-ECA_Ax-UEFI-14.16.17-FlexBoot-3.5.504.bin -allow_psid_change burn

which made them MCX456A-ECATs;)
 

im2geek4you

Member
Feb 27, 2017
30
10
8
43
I've already got mines and I've been doing some connectivity tests with the FW that comes with them. They are detected in windows as ConnectX-4 with the standard drivers which is nice. They are also in Ethernet mode by default. This is what I get from flint (replaced MAC and GUID)

C:\Program Files\Mellanox\WinMFT>flint -d mt4115_pciconf0 q
Image type: FS3
FW Version: 12.17.2052
FW Release Date: 20.4.2017
Product Version: rel-12_17_2052
Rom Info: type=UEFI version=14.11.34 cpu=AMD64
type=PXE version=3.4.903
Description: UID GuidsNumber
Base GUID: xxxxxxxxxxxxxxx 8
Base MAC: xxxxxxxxxxxxx 8
Image VSD: N/A
Device VSD: N/A
PSID: DEL3240110033
Security Attributes: N/A

The firmware seems to be a custom version HW of some kind (not listed in the Dell OEM firmware list). They even have a sticker on the heatsink saying they are model CX456B ... :confused:
So far I could only manage to get them to 40Gbs on my SX6012. For some reason they don't appear to be able to do 56Gbs which the switch can do. Also tested with different cables with no luck. I think I read somewhere that the MCX456A-ECAT could do 56Gbs, but I might have mixed it up. Perhaps only in IB (haven't test that).

Anyway, I flashed one with the same firmware you indicated, and it seems to work OK with it (no luck making it go 56Gbs though). Leds seem to work, and both ports are detected and connect OK. By default the ports go to IB, and it is very annoying to change it in windows compared with the ConnectX-3 mcx354A where you have a GUI menu to do that. Just in case anyone needs to do that the command is the following:

mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Then a reboot of the server or the driver is needed. I haven't found other easier way of doing this :mad:. To change ports back to IB just replace number 2 with 1. To check status one can use:

mlxconfig -d mt4115_pciconf0 q

I'm waiting for a 100Gbs cable for some back to back connection and testing. Will post more info if someone is interested.
For what they can do and the price these cards are AWESOME, I'm surprised why they weren't all sold in a heartbeat. But from a practical point of view I guess the mcx354A are a good deal in the sort term, because it is very hard to find a 100Gb switch at a reasonable price.
Thanks for pointing this deal out ;)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Yes i had to run similar commands to convert back to ETH after flashing, was quite a struggle after being used to gui switching initially. Sorry I had no posted the commands
I never got them to go with 56GBs either, but I have not been able to do so with any of my 100GBs cards, and I only have never seen them supporting 56GBs officially.

And yes, those cards are a steal, totally perplexed they are not gone. I assume its the x16 requirement which is a rare commodity on older single cpu boards .
Now where to find a 100gbs switch;)
 

ecosse

Active Member
Jul 2, 2013
463
111
43
I think you've both answered why they aren't gone - x16 requirement, 100gbe switches cost a ton and although I probably fall out of sth membership for saying this what you gonna use them for that can't be done at 40 or 10 gbe?!??

Having said that its getting painful those still being there so I may buy 3 soon if they don't hurry up and get sold!
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
:)
Actually the main reason I got them was the fact that those are CX4 and not 3's which means they (should) support thinks like mixed IB/ETH setup in ESX and (hopefully) also SR-IOV (properly) - but have not gotten around testing it. Dabbled in RDMA but no good doing that on my PRD cluster;)
 
Last edited:

ecosse

Active Member
Jul 2, 2013
463
111
43
Applications that require extreme low latencies :D
Fair enough if that's a valid use case but how many people here run trading exchanges in their bedroom :) Never mind all the ultra low latency switches etc! Not knocking them mind you, would love to have 3 or 4 of them :)
 

schujj07

New Member
May 10, 2018
29
16
3
40
I've already got mines and I've been doing some connectivity tests with the FW that comes with them. They are detected in windows as ConnectX-4 with the standard drivers which is nice. They are also in Ethernet mode by default. This is what I get from flint (replaced MAC and GUID)

C:\Program Files\Mellanox\WinMFT>flint -d mt4115_pciconf0 q
Image type: FS3
FW Version: 12.17.2052
FW Release Date: 20.4.2017
Product Version: rel-12_17_2052
Rom Info: type=UEFI version=14.11.34 cpu=AMD64
type=PXE version=3.4.903
Description: UID GuidsNumber
Base GUID: xxxxxxxxxxxxxxx 8
Base MAC: xxxxxxxxxxxxx 8
Image VSD: N/A
Device VSD: N/A
PSID: DEL3240110033
Security Attributes: N/A

The firmware seems to be a custom version HW of some kind (not listed in the Dell OEM firmware list). They even have a sticker on the heatsink saying they are model CX456B ... :confused:
So far I could only manage to get them to 40Gbs on my SX6012. For some reason they don't appear to be able to do 56Gbs which the switch can do. Also tested with different cables with no luck. I think I read somewhere that the MCX456A-ECAT could do 56Gbs, but I might have mixed it up. Perhaps only in IB (haven't test that).

Anyway, I flashed one with the same firmware you indicated, and it seems to work OK with it (no luck making it go 56Gbs though). Leds seem to work, and both ports are detected and connect OK. By default the ports go to IB, and it is very annoying to change it in windows compared with the ConnectX-3 mcx354A where you have a GUI menu to do that. Just in case anyone needs to do that the command is the following:

mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Then a reboot of the server or the driver is needed. I haven't found other easier way of doing this :mad:. To change ports back to IB just replace number 2 with 1. To check status one can use:

mlxconfig -d mt4115_pciconf0 q

I'm waiting for a 100Gbs cable for some back to back connection and testing. Will post more info if someone is interested.
For what they can do and the price these cards are AWESOME, I'm surprised why they weren't all sold in a heartbeat. But from a practical point of view I guess the mcx354A are a good deal in the sort term, because it is very hard to find a 100Gb switch at a reasonable price.
Thanks for pointing this deal out ;)
http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf - This has the quick information on the cards. It does say that in Ethernet it should do 56Gbps, however, you will need 56Gbps DACs to be able to do it. If you only have regular 40s I doubt it will work. That being said the 56Gbps DACs aren't that expensive at Fibre Store 56G QSFP+ to QSFP+ DAC Twinax Cables | FS.COM

Had we not already gone to 25Gbps networking where I work I would have suggested getting these for our vSAN.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Not sure the ecat does FDR based on that document. The fcat will do 40/56 but it is not explicitly stated for the ecat.

Maybe I can test it tonight.
 

Laugh|nGMan

Member
Nov 27, 2012
36
7
8
PSID: DEL3240110033
Not sure if that is important, but according to my "investigation" this link (MSIP-REM-MLN-CX456B ConnectX-4 VPI adapter card) points to Sub-Model: 0HWTYK, 00272F and that combo with PSID number DEL3240110033 google search gives me link to this Mellanox pdf (http://www.mellanox.com/pdf/firmware/ConnectX4-FW-12_17_2052-Release_Notes.pdf)

Where..
1 Overview These are the release notes for the ConnectX®-4 adapters firmware Rev 12.17.2052. This firmware supports the following protocols: • InfiniBand - SDR, QDR, FDR10, FDR, EDR • Ethernet - 1GigE, 10GigE, 25GigE, 40GigE, 50GigE, 56GigE1 and 100GigE • PCI Express 3.0, supporting backwards compatibility for v2.0 and v1.1
1.1 Supported Devices This firmware supports the devices and protocols listed in Table 2
1. 56 GbE is a Mellanox propriety link speed and can be achieved while connecting a Mellanox adapter cards to Mellanox SX10XX switch series or connecting a Mellanox adapter card to another Mellanox adapter card.
Table 2 - Supported Devices
Device Part Number PSID Device Name
Compiled with FlexBoot
Compiled with UEFI
0068F2_0NNJ2M DEL2190110032 Mellanox ConnectX-4 Dual Port EDR PCIE Adapter LP
Yes No
06W1HY_0JJN39 DEL2180110032 Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
Yes No
0NHYP5_0XR0K2 DEL2150110033 ConnectX-4 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; ROHS R6
Yes No
00272F_0HWTYK DEL3240110033 Mellanox ConnectX-4 Dual Port 100 GbE QSFP Network Adapter
Yes Yes
DELL_C6320p_1P _EDR
DEL2010120032 Dell C6320p - ConnectX-4 adapter card; EDR IB (100Gb/s); single-port QSFP28; PCIe3.0 x16
Yes No
This firmware supports the devices and protocols listed in Table 2
Device Name Mellanox ConnectX-4 Dual Port 100 GbE QSFP Network Adapter 456B.JPG
 
  • Like
Reactions: Rand__

im2geek4you

Member
Feb 27, 2017
30
10
8
43
Got the cable for 100Gb and... it works! :)

This is actually connecting 2 cards back to back, one with original firmware in Linux CentOS and the other with the flashed version (12.23.1020) in Windows server 2016.
Cable from this tread. STH forum rules! :cool:
Now to find a 100Gb switch...:rolleyes:
 

Attachments

jerrytsao

MILAN X P5800X
Sep 11, 2016
39
31
18
Shanghai, CN
Great deal for European friends, as it's only shipped to selective European countries, forwarding to Asia or US will take a bit more effort.

Anyway, at 160€($185) this is the lowest ConnectX-4 VPI MCX456A-ECAT I have ever seen, albeit the unusual Dell OEM P/N.

Earlier this year I snatched 2*$270 (allegedly ES) from Isreal seller, at the time it's rarely seen cheap dual-port 100Gb CX4 cards flowing around on eBay, unfortunately, one card died not too long after I updated the firmware.

Until last month, a seller in China (could be more out there) is selling a large amount of Lenovo OEM 00MM962_Ax with a price of ¥1,500($217), these are not the typical "fake & self-made" Chinese NICs you would find on eBay. I got two more and now three cards are working flawlessly in 100GbE with $65 HPE OEM MCP1600-C003 QSFP28 3M ETH DAC previously bought on eBay. It's pretty easy to flash retail firmware with flint -allow_psid_change and then using mlxconfig to switch dual ports from IB to ETH, you could also do the "correct" OEM updating, but I found that was unnecessary (no compatibility issue after PSID change).

Other than Mellanox User Guides, I did not find much information when I was configuring CX4 cards, so I made a post on Chiphell back in April Mellanox ConnectX-4 VPI MCX456A-ECAT 2*100GbE测试(2018-09-14更新) - 电脑讨论 - Chiphell - 分享与交流用户体验, you can check out above link for guidance, Mellanox make the most comprehensive manuals so the process should be pretty smooth all by yourself.





 
Last edited: