MCB193A-FCAT possible crossflash?
- Thread starter efschu3
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
what if replace the cb193a flash chip to cx4 16Mb and flash cx4 firmware？bought one and trying to achieve 100GbE (not worth for 40/56GbE in my opinion when you can buy ConnectX-3 for about $40)
I failed - "bricked" it(possible to "unbrick" it via cables but need ~1h of time)
the chips:
mt4099 <- ConnectX-3 (super easy to flash from 10G to 40/56GbE)
mt4103 <- ConnectX-3 Pro (I think all of them are factory 40/56GbE so no need to flash)
mt4113 <- Mellanox CB193A/CB194A / Connect-IB <- ??! ?!??!??!?!?
mt4115 <- ConnectX-4 (super easy to flash from 40G/50G to 100GbE)
mt4119 <- ConnectX-5 (super easy to flash from 40G/50G/100G PCIE 3.0 to 100GbE PCIE 4.0)
ConnectX-6/ ConnectX-7 no experience yet - I don't have switch for 200/400GbE
Are you confirming it ?
I never need to flash ConnectX-3 Pro (and for me there is no difference between ConnectX-3 and ConnectX-3 Pro - just need simple 40GbE under windows). is ConnectX-3 Pro somehow... faster
My observations about CB193A are here:
FW from ConnectX-3 not working but if I remember correctly not bricked (was able to flash with ConnectX-4 and hoping there is nothing important over first 4MB - not true
FW from ConnectX-3 Pro <- not tested
FW from ConnectX-4 too big, after apply bricked
Flash memory inside ConnectX-3 is 1MB
Flash memory inside ConnectX-3 Pro is ???<- I don't know never need to flash them
Flash memory inside Connect-IB is 4MB
Flash memory inside ConnectX-3 is 16MB
Board and card looks like can support 100G (similar traces to CX4), but plug is QSFP+ not QSFP28, spec saying max 56Gbits not 100/112
Probably what happened here and why this model is PCIE x16:
2x 56Gbit IB is more than PCIE 3.0 x8 can deliver
This problem exist when using dual port ConnectX-3 with LACP, PCIE can do ~7.6GB/s card 8.84GB/s
I don't know. Still not worth I think (back then found lot of ConnectX-4 50GbE for 110EUR that you can flash to 100G)
Found it: quad SPI flash , under the radiator
The name of the chip is:
Winbond 25Q32BVSIG (~$1)
and similar 4 times bigger exist, the name is:
Winbond 25Q128BVSIG (~$4)
so.. yes you can swap 4Mbytes into 16Mbytes one it is small but just 8 pins <- possible to do it
$4 and 2h of work to solder and copy data from 32Mbit one to 128Mbit
now... the main chip
, under the radiatorThe name of the chip is:
Winbond 25Q32BVSIG (~$1)
and similar 4 times bigger exist, the name is:
Winbond 25Q128BVSIG (~$4)
so.. yes you can swap 4Mbytes into 16Mbytes one
it is small but just 8 pins <- possible to do it$4 and 2h of work to solder and copy data from 32Mbit one to 128Mbit
now... the main chip
ok found it on CX4 lol, CX4 is using Winbond 25Q128FV
I am not opening the radiator, because chips are different almost for sure
mt4099 <- ConnectX-3
mt4103 <- ConnectX-3 Pro
mt4113 <- Connect-IB
mt4115 <- ConnectX-4
so, now the big questions:
- if the FV is compatible between mt4113 and mt4115
- if the plug, traces and amplifier (transistors) are able to deliver data to the QSFP28 transceiver and run 100GbE
hm.... maybe should "we" try it first on mt4099 vs mt4103 ?
Somebody work at Mellanox, and can talk more about this "ancient but still amazing" technology (I hope it's not under NDA after so many years) ?
I am not opening the radiator, because chips are different almost for sure
mt4099 <- ConnectX-3
mt4103 <- ConnectX-3 Pro
mt4113 <- Connect-IB
mt4115 <- ConnectX-4
so, now the big questions:
- if the FV is compatible between mt4113 and mt4115
- if the plug, traces and amplifier (transistors) are able to deliver data to the QSFP28 transceiver and run 100GbE
hm.... maybe should "we" try it first on mt4099 vs mt4103 ?
Somebody work at Mellanox, and can talk more about this "ancient but still amazing" technology (I hope it's not under NDA after so many years) ?
So just 2 options left:
1. Unsolder chip from C-IB, solder 128Mbit one and flash with "random" CX-4 (for safety 40G/50G variant) Firmware
2. Unsolder chip from flashed to 40G/50G CX-4 and solder it for a while to C-IB
but if like you said mt4099 (CX-3) and mt4103 (CX-3 Pro) are not compatible, most likely it will be wasting of time.
by design board it's closer to CX-4 than CX-3 in my opinion, but I didn't spent too much time, in my brain this card is IB only (and I need only fast Ethernet, playing with IB maybe will be probably lot of fun but too much time to spent to learn new toys )
1. Unsolder chip from C-IB, solder 128Mbit one and flash with "random" CX-4 (for safety 40G/50G variant) Firmware
2. Unsolder chip from flashed to 40G/50G CX-4 and solder it for a while to C-IB
but if like you said mt4099 (CX-3) and mt4103 (CX-3 Pro) are not compatible, most likely it will be wasting of time.
by design board it's closer to CX-4 than CX-3 in my opinion, but I didn't spent too much time, in my brain this card is IB only
(and I need only fast Ethernet, playing with IB maybe will be probably lot of fun but too much time to spent to learn new toys )Alright! I know I am new but I have a habit of learning to use things incorrectly. I bought two em33's aka mt4113. The first one I put away just in case I screw something up and need a fresh start or a reference card. The second one had IBM firmward 10.14.1100. I wasnt able to get Windows 10 to find drivers after several ofeds and winofs installed. So I decided to try the 10.16.1200 desgned for cb194a-fcat. It flashed just fine. Then from there I tried a few other firmwares cx3,cx4,cx5 and all with no luck. No surprise really either. I then decided to swap to ubuntu which found it and says it is connected but I cant get traffic to pass on it. I tested a cx3 on the same install and couldnt get traffic to pass either so it must be something else. Once I figure out cx I'll swap it back out.
In the mean time I ordered a couple Winbond 128s listed above. They arrived today. I have few ideas. First was to swap chips and attempt to flash a cib firmware. My reasearch tells me that these are 5th gen not 4th. So I may have to modify a gen5 firmware and flash, but I have no idea how to modify the firmwares. I dont have a gen5 handy and so I cant swap chips and flash that way. If it comes to that it may be a long time.
Let me know what you guys think. It will probably be the weekend before I can dive back in. But I would be willing to try some things. Ill post more info here soon either way. I took pictures of the die and multiples of the front and backs of what cards I have.
In the mean time I ordered a couple Winbond 128s listed above. They arrived today. I have few ideas. First was to swap chips and attempt to flash a cib firmware. My reasearch tells me that these are 5th gen not 4th. So I may have to modify a gen5 firmware and flash, but I have no idea how to modify the firmwares. I dont have a gen5 handy and so I cant swap chips and flash that way. If it comes to that it may be a long time.
Let me know what you guys think. It will probably be the weekend before I can dive back in. But I would be willing to try some things. Ill post more info here soon either way. I took pictures of the die and multiples of the front and backs of what cards I have.
My cards are the IBM EC33's, which are an IBM variant of the MLX CB194a Connect-IB card. Which may be less compatible than the original Mellanox cards. As far as I can tell IBM only used them an AIX appliance. The drivers that are on the Mellanox website are only for AIX. In Linux I only tested the card I flashed to a CB194a which automatically detected and used the MLX_4 drivers. I will try the card with the original firmware at some point. Currently both cards are flashed with the IBM firmware, but one is a firmware from the Mellanox website and the other is the firmware I received it with.
for Linux (Ubuntu) even though it saw the card, it wouldn’t do anything else with it, most commands failed or returned unexpected results. “Mst status” seemed to be the only IB command that worked properly, I couldn’t even flash the card on Linux. After getting frustrated I decided to swap the EC33 out with a Connectx-3 card and still had issues. Maybe I just need a different version of Linux? So far I have had much better luck with Infiniband in Windows so I’ll probably go back to Windows. To see if I can at least get it to pass IB traffic at all. I have two known to be good workstations that I haven’t had any issues getting traffic to pass over IB with CX2 cards.
I was wrong about gen 5, I thought it was going to be the MLX_5 drivers not the MLX_4 drivers that ended up working, I attempted MLX_5 drivers in Linux, just because and it was not happy with me. For Windows I attempted several versions of both WinOF and WinOF2, but none of them found the CIB card. I didn’t try to force it to use an MLX_4 driver, but I will next time.
After my weekend of testing, I am pretty sure these aren’t going to work, they just seem too unique. I think if I had a real Mellanox CB194a I may be able to get it working for IB. Right now, I am trying to get my switches configured. Which is causing its own issues. Then once I have a more steady setup I’ll try again.
for Linux (Ubuntu) even though it saw the card, it wouldn’t do anything else with it, most commands failed or returned unexpected results. “Mst status” seemed to be the only IB command that worked properly, I couldn’t even flash the card on Linux. After getting frustrated I decided to swap the EC33 out with a Connectx-3 card and still had issues. Maybe I just need a different version of Linux? So far I have had much better luck with Infiniband in Windows so I’ll probably go back to Windows. To see if I can at least get it to pass IB traffic at all. I have two known to be good workstations that I haven’t had any issues getting traffic to pass over IB with CX2 cards.
I was wrong about gen 5, I thought it was going to be the MLX_5 drivers not the MLX_4 drivers that ended up working, I attempted MLX_5 drivers in Linux, just because and it was not happy with me. For Windows I attempted several versions of both WinOF and WinOF2, but none of them found the CIB card. I didn’t try to force it to use an MLX_4 driver, but I will next time.
After my weekend of testing, I am pretty sure these aren’t going to work, they just seem too unique. I think if I had a real Mellanox CB194a I may be able to get it working for IB. Right now, I am trying to get my switches configured. Which is causing its own issues. Then once I have a more steady setup I’ll try again.
Ok so probably no chances to use CB194 as a 40/56/100GbE cards (without internal Mellanox SDK for mt4113 chip).
What is maybe possible to make, is:
CB194a as a IB and do ~110GbE via EoIB (dual port and x16 PCIe 3.0)
so few CB194a cards for $17 + ~$200 IB switch
probably don't make any sense when this is possible:
100GbE Mellanox for ~$150 (or 40/56GbE for $35) + $400 switch
but using $17 cards as a ~100GbE NIC is a thing
What is maybe possible to make, is:
CB194a as a IB and do ~110GbE via EoIB (dual port and x16 PCIe 3.0)
so few CB194a cards for $17 + ~$200 IB switch
probably don't make any sense when this is possible:
100GbE Mellanox for ~$150 (or 40/56GbE for $35) + $400 switch
but using $17 cards as a ~100GbE NIC is a thing
I havnt had any time to look at this lately. I am probably just going to use mine as a bridge between my VM hosts. This year this was just a short venture into seeing if I could turn a $17 card into a $200 card. The connect-ib cards seem to just be ewaste. Mostly due to lack of driver support not really a fault of their own. If I could use mine in Windows I would gladly use them as is!