Flashing CX4131A-BCAT (40Gbe) to GCAT (50Gbe) what should I look out for?

[TonyA]

So, I think I've hit the limit of my understanding and have made a mistake somewhere.

I have the Mellanox cards in both machines working properly and at around 40c from using temporary fans to provide extra cooling till I come up with a better long term solution.

I have QSFP28 transceivers that support 100Gbe connections using an LC connection, using the fibre cabling I had to hand I could connect at 1Gbe and do pings at least. I had come to the conclusion my fibre was old/damaged/whatever so had ordered some 2m OM4 LC-LC cables, just to prove full functionality.

Those cables arrived today, but I can still only connect at 1Gb speeds?

Using ethtool -s xxxx speed option I get....

Settings for enpds0np0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: None     RS     BASER
    Advertised link modes:  10000baseKR/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
    Link detected: no (Cable issue, Unsupported cable)

my transceiver shows this....

Identifier                                : 0x11 (QSFP28)
    Extended identifier                       : 0xcc
    Extended identifier description           : 3.5W max. Power consumption
    Extended identifier description           : CDR present in TX, CDR present in RX
    Extended identifier description           : High Power Class (> 3.5 W) not enabled
    Power set                                 : Off
    Power override                            : On
    Connector                                 : 0x07 (LC)
    Transceiver codes                         : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 100G Ethernet: 100G CWDM4 MSA with FEC
    Encoding                                  : 0x07 ((256B/257B (transcoded FEC-enabled data))
    BR, Nominal                               : 25500Mbps
    Rate identifier                           : 0x02
    Length (SMF,km)                           : 2km
    Length (OM3 50um)                         : 0m
    Length (OM2 50um)                         : 0m
    Length (OM1 62.5um)                       : 0m
    Length (Copper or Active cable)           : 0m
    Transmitter technology                    : 0x40 (1310 nm DFB)
    Laser wavelength                          : 1301.000nm
    Laser wavelength tolerance                : 73.000nm
    Vendor name                               : INNOLIGHT
    Vendor OUI                                : 44:7c:7f
    Vendor PN                                 : TR-FC13T-NFB
    Vendor rev                                : 1A
    Vendor SN                                 : xxxxxxxxxxxx
    Date code                                 : 180226
    Revision Compliance                       : SFF-8636 Rev 2.5/2.6/2.7
    Rx loss of signal                         : None
    Tx loss of signal                         : [ Yes, Yes, Yes, Yes ]
    Rx loss of lock                           : [ Yes, Yes, Yes, Yes ]
    Tx loss of lock                           : [ Yes, Yes, Yes, Yes ]
    Tx fault                                  : None
    Module temperature                        : 40.37 degrees C / 104.67 degrees F
    Module voltage                            : 3.2912 V
    Alarm/warning flags implemented           : Yes
    Laser tx bias current (Channel 1)         : 40.736 mA
    Laser tx bias current (Channel 2)         : 40.546 mA
    Laser tx bias current (Channel 3)         : 40.436 mA
    Laser tx bias current (Channel 4)         : 40.356 mA
    Transmit avg optical power (Channel 1)    : 0.9971 mW / -0.01 dBm
    Transmit avg optical power (Channel 2)    : 1.1182 mW / 0.49 dBm
    Transmit avg optical power (Channel 3)    : 1.1581 mW / 0.64 dBm
    Transmit avg optical power (Channel 4)    : 1.1820 mW / 0.73 dBm
    Rcvr signal avg optical power(Channel 1)  : 0.3688 mW / -4.33 dBm
    Rcvr signal avg optical power(Channel 2)  : 0.2385 mW / -6.23 dBm
    Rcvr signal avg optical power(Channel 3)  : 0.1309 mW / -8.83 dBm
    Rcvr signal avg optical power(Channel 4)  : 0.2766 mW / -5.58 dBm
    Laser bias current high alarm   (Chan 1)  : Off
    Laser bias current low alarm    (Chan 1)  : Off
    Laser bias current high warning (Chan 1)  : Off
    Laser bias current low warning  (Chan 1)  : Off
    Laser bias current high alarm   (Chan 2)  : Off
    Laser bias current low alarm    (Chan 2)  : Off
    Laser bias current high warning (Chan 2)  : Off
    Laser bias current low warning  (Chan 2)  : Off
    Laser bias current high alarm   (Chan 3)  : Off
    Laser bias current low alarm    (Chan 3)  : Off
    Laser bias current high warning (Chan 3)  : Off
    Laser bias current low warning  (Chan 3)  : Off
    Laser bias current high alarm   (Chan 4)  : Off
    Laser bias current low alarm    (Chan 4)  : Off
    Laser bias current high warning (Chan 4)  : Off
    Laser bias current low warning  (Chan 4)  : Off
    Module temperature high alarm             : Off
    Module temperature low alarm              : Off
    Module temperature high warning           : Off
    Module temperature low warning            : Off
    Module voltage high alarm                 : Off
    Module voltage low alarm                  : Off
    Module voltage high warning               : Off
    Module voltage low warning                : Off
    Laser tx power high alarm   (Channel 1)   : Off
    Laser tx power low alarm    (Channel 1)   : Off
    Laser tx power high warning (Channel 1)   : Off
    Laser tx power low warning  (Channel 1)   : Off
    Laser tx power high alarm   (Channel 2)   : Off
    Laser tx power low alarm    (Channel 2)   : Off
    Laser tx power high warning (Channel 2)   : Off
    Laser tx power low warning  (Channel 2)   : Off
    Laser tx power high alarm   (Channel 3)   : Off
    Laser tx power low alarm    (Channel 3)   : Off
    Laser tx power high warning (Channel 3)   : Off
    Laser tx power low warning  (Channel 3)   : Off
    Laser tx power high alarm   (Channel 4)   : Off
    Laser tx power low alarm    (Channel 4)   : Off
    Laser tx power high warning (Channel 4)   : Off
    Laser tx power low warning  (Channel 4)   : Off
    Laser rx power high alarm   (Channel 1)   : Off
    Laser rx power low alarm    (Channel 1)   : Off
    Laser rx power high warning (Channel 1)   : Off
    Laser rx power low warning  (Channel 1)   : Off
    Laser rx power high alarm   (Channel 2)   : Off
    Laser rx power low alarm    (Channel 2)   : Off
    Laser rx power high warning (Channel 2)   : Off
    Laser rx power low warning  (Channel 2)   : Off
    Laser rx power high alarm   (Channel 3)   : Off
    Laser rx power low alarm    (Channel 3)   : Off
    Laser rx power high warning (Channel 3)   : Off
    Laser rx power low warning  (Channel 3)   : Off
    Laser rx power high alarm   (Channel 4)   : Off
    Laser rx power low alarm    (Channel 4)   : Off
    Laser rx power high warning (Channel 4)   : Off
    Laser rx power low warning  (Channel 4)   : Off
    Laser bias current high alarm threshold   : 75.000 mA
    Laser bias current low alarm threshold    : 10.000 mA
    Laser bias current high warning threshold : 70.000 mA
    Laser bias current low warning threshold  : 15.000 mA
    Laser output power high alarm threshold   : 2.8183 mW / 4.50 dBm
    Laser output power low alarm threshold    : 0.0708 mW / -11.50 dBm
    Laser output power high warning threshold : 2.5119 mW / 4.00 dBm
    Laser output power low warning threshold  : 0.1122 mW / -9.50 dBm
    Module temperature high alarm threshold   : 58.00 degrees C / 136.40 degrees F
    Module temperature low alarm threshold    : -6.00 degrees C / 21.20 degrees F
    Module temperature high warning threshold : 53.00 degrees C / 127.40 degrees F
    Module temperature low warning threshold  : -3.00 degrees C / 26.60 degrees F
    Module voltage high alarm threshold       : 3.7000 V
    Module voltage low alarm threshold        : 2.9000 V
    Module voltage high warning threshold     : 3.6000 V
    Module voltage low warning threshold      : 3.0000 V
    Laser rx power high alarm threshold       : 2.8183 mW / 4.50 dBm
    Laser rx power low alarm threshold        : 0.0177 mW / -17.52 dBm
    Laser rx power high warning threshold     : 2.2387 mW / 3.50 dBm
    Laser rx power low warning threshold      : 0.0354 mW / -14.51 dBm

The cable I'm using has LC - LC connectors and is described as

LC to LC MM Duplex 50/125um OM4

So, have I ordered the wrong cable? From all the reading I had done I was under the impression that OM3 and OM4 would both support 100Gbe speeds, with OM4 doing it for longer distances. Is that true but dependent on other factors than the cabling just being OM4?

Is any speed over 10Gb only supported by MPT/MTO connector cables?

Today, when I saw the issue I was confused so dived back into reading and read about

"......Wideband multimode fibre (WBMMF) is designated as OM5 in the standards...."

so to get 100Gbe should I actually have ordered an OM5 cable?

I realise the QSFP28 has four channels each of which that will support 25Gbe, but does that only work over a cable with 4 fibre cores or can WBMMF multiplex the four channels over a single fibre?

I realise many of these questions will read as quite simplistic to the more experienced readers, however, when you're coming at this for the first time you're bound to make mistakes and I'm not sure what I've done wrong here so some guidance would be appreciated.

 

[piranha32]

According to the information about transceiver, it uses a laser diode at 1310nm. This wavelength is used with single mode fiber. There is a good chance that the multimode fiber you try to use will not work at higher speeds.

 

[TonyA]

According to the information about transceiver, it uses a laser diode at 1310nm. This wavelength is used with single mode fiber. There is a good chance that the multimode fiber you try to use will not work at higher speeds.

Thanks, having had a chance to read further now it looks like I should be looking only at OS2 cabling rather than OM series stuff.

Are there other things I should be looking at specifically on the OS2 cabling description to ensure I'm getting the right thing? For example I notice a lot of stuff shows insertion loss rates and I've even noticed some OS2 cabling being described as "....supports 1Gb rates..." which gives me pause for thought as elsewhere I've read that OS2 should support 4 channels 'muxed' at 28Gb each, so is there a reason some cabling would only support 1Gb or is it more likely it's just a bad description?

 

[piranha32]

Are there other things I should be looking at specifically on the OS2 cabling description to ensure I'm getting the right thing? For example I notice a lot of stuff shows insertion loss rates and I've even noticed some OS2 cabling being described as "....supports 1Gb rates..." which gives me pause for thought as elsewhere I've read that OS2 should support 4 channels 'muxed' at 28Gb each, so is there a reason some cabling would only support 1Gb or is it more likely it's just a bad description?

Yes. Single mode fibers use two different and incompatible methods of polishing the ends of the fiber: perpendicular to the axis of the fiber, and at a slight angle. The problem with perpendicular cut (or so I have been told) is that at the connections it can create reflections, which will propagate back to the source, and may cause problems. This can be avoided by cutting the fiber at an angle, and directing the reflection off axis, where it can dissipate.
These two cuts are not compatible, however, this is where my knowledge ends.

 

[NablaSquaredG]

bad description?

yep, bad description

If you have a standard conforming OS2 fiber, it will support 400Gbit/s or even more.

Just make sure that you get UPC, which is what all transceivers use. There is APC (Angled Physical Contact), but this is more special purpose.

 

[Espada]

I've crossflashed a hpe 100gbe nic to vanilla mellanox firmware. For that I had to use the allow psid change parameter. I don't think that should be a problem.

If your card has qsfp28 connectors I would try to flash it with a 100gbe firmware (even when the cars has a pcie x8 host interface ). From my understanding mellanox only build one ic/asic per generation and then "configured" it by firmware what speeds it supports.
But like always make a backup of the existing firmware (and config/information about mac address in case you have to use the recovery mode) before trying to crossflash it to something different.

I think with the cx-5 family mellanox introduced signed firmware that could be a pita when trying to crossflash adapters, but I don't have any cards and therefore experience with the newer generations yet.

For someone who would like to flash a 100gbe firmware, do you have a suggestion for a particular model to do so?

 

[TonyA]

For someone who would like to flash a 100gbe firmware, do you have a suggestion for a particular model to do so?

I won't answer your question directly, will leave that for more experienced others.

However, if your machine doesn't support the required bus speeds you won't get 100Gbe speed throughput whatever you do. My Dell workstation would top out around 80Gbe throughput, so for that reason I decided to stick to 50Gbe firmware flashes as I didn't see the point of 'over stressing' the card if I wasn't going to get 100% of the benefit.

I do realise that the cards are supposedly identical, except they've been flashed with different firmwares that limit their speeds, however I'm not certain that the reason a specific card wasn't chosen to be at a specific speed wasn't pre flash 'binning', thus my inclination to be somewhat cautious.

 

[unphased]

I just read in another thread that 4131A cards can only go to 50Gbit and not 100Gbit. they'd probably be hampered by x8 lanes and 80Gbit anyway but 80 would be quite superior to 50... Not sure what the real answer on this is. @TonyA you were able to bring your BCAT's to support 50Gbit? How is it going? I just ordered a BCAT one of these, since I finally saw one at an acceptable price, and would prefer to have CX4 cards over CX3 cards. I worry though that if the 4131A cards are "half" their QSFP28 to get 50Gbit that it would mean I'd be locked to 20Gbit when running 40Gbit gear.

 

[TonyA]

I just read in another thread that 4131A cards can only go to 50Gbit and not 100Gbit. they'd probably be hampered by x8 lanes and 80Gbit anyway but 80 would be quite superior to 50... Not sure what the real answer on this is. @TonyA you were able to bring your BCAT's to support 50Gbit? How is it going? I just ordered a BCAT one of these, since I finally saw one at an acceptable price, and would prefer to have CX4 cards over CX3 cards. I worry though that if the 4131A cards are "half" their QSFP28 to get 50Gbit that it would mean I'd be locked to 20Gbit when running 40Gbit gear.

Yep, my BCATs are now reporting as 50Gbe supported, however I can't test throughput number till my newly ordered OS2 cabling arrives which is probably going to be early next week now.

The 40Gbe support is also enabled specifically, so I don't think you'd get 20Gbe speeds. If you look at my posting at 9:19pm on Tuesday you can see the output from ethtool for one of my cards and you'll see the supported speeds are....

Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full

I would guess that the multiplexing for 50Gbe is actually two channels at 25Gbe each, whereas 40Gbe is all four channels at 10Gbe each. In theory the card 'could' run at 100Gbe, except that of course it won't support 100Gbe throughput due to PCI limitations, so I suspect they've therefore disabled that in the firmware.

I have no desire to test whether 80Gbe (card flashed at 100Gbe) would or wouldn't work as, as I stated earlier, I wonder if the cards are 'pre binned' by tolerance tests to determine the most appropriate speed for each card before it's flashed, that's what they do with most CPUs after all.

Having come from 1Gbe recently the jump to 10Gbe (I used a couple of x540 cards for testing purposes) was pretty impressive, so the jump to 50Gbe would do me fine for now and my plan is to be on the lookout for some Dual Port 100Gbe cards and sellon my 50Gbe cards to offset the costs. Of course looking at 100Gbe switch costs they're crazy right now, so I'll be sticking to point to point connections for my systems that would benefit from the high speed connections right now.

If you don't mind me asking, how much did you pay for the CX4131A-BCAT you just ordered?

 

[unphased]

$40 plus a bit of tax. I figure it's worthwhile since the MCX455A-ECAT I got two weeks ago was $70. Able to use it with mlx5 driver over mlx4 (absent for macOS for example) is a factor. Also was motivated by finding a missing capacitor on my MCX455A card. It still works though. Mostly.