Beware of EMC switches sold as Mellanox SX6XXX on eBay

[maxermaxer]

It is great to see after so many years this post is still alive! I am a late joiner for SX6012. Just bought one from 2nd hand market. I will read this thread from the beginning but may I ask:
1) there seems to be two guides, one from mpogr and the other from dodgy route (a shorter version). (Thanks both first of all!). Should I go with dodgy's guide? Anything to be aware of?
2) any must-do or must-not before doing the conversion?
3) when I bought this SX6012 from the seller, the seller gave me the VPI licensing assistance. The seller said he had never done any conversion on this EMC SX6012. What should I do with this VPI licensing assistance? Is it still needed/will it still be working after the conversion?
4) what should I back up before the conversion?
5) in case anything goes wrong how do I go back to original state (EMC state)?

The part number is 100-886-236-04

Thanks!!

 

[figure7555]

It is great to see after so many years this post is still alive! I am a late joiner for SX6012. Just bought one from 2nd hand market. I will read this thread from the beginning but may I ask:
1) there seems to be two guides, one from mpogr and the other from dodgy route (a shorter version). (Thanks both first of all!). Should I go with dodgy's guide? Anything to be aware of?

I converted mine a couple weeks ago using dodgy route's method. It was all pretty straightforward. But he does spend a lot of time explaining how to set up things like TFTP and HTTP. I just did that all through Docker Compose. If you already have Docker installed, it's just a lot easier. Here's my docker-compose.yml file:

version: '3.4'
services:
  tfpd:
    image: stoxygen/tftpd-hpa:v2.0.0
    command: in.tftpd --ipv4 --foreground --listen --address 0.0.0.0:69 --create --permissive --secure -vvv /data
    volumes:
      - ./tfpd_share:/data
    ports:
      - 0.0.0.0:69:69/udp
  http:
    image: caddy:2.7.4-alpine
    volumes:
      - ./web_share:/usr/share/caddy
      - ./Caddyfile:/etc/caddy/Caddyfile
    ports:
      - 80:3000

And here's is the Caddyfile for the HTTP server:

{
  admin off
  auto_https off
}

:3000 {
  # set this path to your site's directory
  root * /usr/share/caddy

  route {
    header Cache-Control public,max-age=0,must-revalidate
    file_server
  }
}

Just put those two files in the same directory. Then add two more directories: "tfpd_share" and "web_share". Then you just copy the files mentioned in dodgy route's guide into the appropriate folder. When ready, just run "docker compose up" from the command line to start the http and tftp servers.

3) when I bought this SX6012 from the seller, the seller gave me the VPI licensing assistance. The seller said he had never done any conversion on this EMC SX6012. What should I do with this VPI licensing assistance? Is it still needed/will it still be working after the conversion?

dodgy route's guide covers licensing. You don't need any "assistance".

4) what should I back up before the conversion?

The guide covers this.

5) in case anything goes wrong how do I go back to original state (EMC state)?

IIRC, I don't think you're doing anything that can't be undone. But I'm not sure why anyone would want to go back. The old EMC firmware isn't much good for anything except a dumb IB switch.

 

[maxermaxer]

Thanks so much figure7555! I am in the middle of converting FRU.

 

[maxermaxer]

Hi, I have successfully converted my EMC SX6012 to Mellanox by following the guide from dodgy route. I am now on 3.4.0012 image. I have a couple of questions after reading all the threads here:

1) What is the safetest way to upgrade the image from 3.4.0012 to 3.6.8010 or 8012? Through the command lines or the web GUI?
2) Can I upgrade directly to 3.6.8010 or 8012 in web gui without following the uphgrade path (maximum 2 version # each time)
3) What is the benifit of using the latest image compared with the old one?
4) I saw some discussion about waiting

Any help is appreciated!

 

[maxermaxer]

Are you updating from the console or from the web ui?

I followed this process about a week ago up until the updating process. At that point, the guide mentioned updating both boot locations to the same version before moving on to the next. This made no sense to me because the entire purpose of having two boot locations is so you can revert in the event of a failed update. So having both boot locations on the same version would seem to serve no purpose. I went ahead and used the web ui to perform the updates. It would automatically install the new version to the unused boot location so in effect you would switch between location 1 and 2 for each update. I didn't install the same version to each boot location. I just installed one version to one boot location, then the next version on the other boot location.

This went smoothly until 3.6.8010. At that point boot was very slow and the web ui never came up. I did not see any errors in the console. I booted back to the previous version and tried the install again. On the second attempt I also did not see the web ui come up. So in the switch console I ran the command to enable the web ui. Something like "web enable". Try "web ?" to see what the commands are. After waiting a couple minutes the web ui was back up. I then proceeded to the 3.6.8012 update. However this one took forever. the web ui was showing 100% cpu usage so I just let it go. After around 3 hours (yes, really) it completed.

Hi figure7555, thanks for sharing! I am about to do the same to update the image. You said "... I booted back to the previous version and tried the install again". May I know how you "boot it back to theh previous version" since you said you didn't get into the GUI. Did you do that in the command line? What exactly did you type as the command? I am new to this so appreciate your help!

 

[figure7555]

Hi figure7555, thanks for sharing! I am about to do the same to update the image. You said "... I booted back to the previous version and tried the install again". May I know how you "boot it back to theh previous version" since you said you didn't get into the GUI. Did you do that in the command line? What exactly did you type as the command? I am new to this so appreciate your help!

You do it from the console. During initial boot it will tell you which boot partition it is going to boot from or to press some key to change it. Just press the key (I think it was "b") then select the boot partition you want.

 

[figure7555]

Hi, I have successfully converted my EMC SX6012 to Mellanox by following the guide from dodgy route. I am now on 3.4.0012 image. I have a couple of questions after reading all the threads here:

1) What is the safetest way to upgrade the image from 3.4.0012 to 3.6.8010 or 8012? Through the command lines or the web GUI?
2) Can I upgrade directly to 3.6.8010 or 8012 in web gui without following the uphgrade path (maximum 2 version # each time)
3) What is the benifit of using the latest image compared with the old one?
4) I saw some discussion about waiting

Any help is appreciated!

3. I think you need at least version 3.6 for the license keys to work

 

[maxermaxer]

3. I think you need at least version 3.6 for the license keys to work

Thanks for such a quick response! Appreciate!

Oh I didn't know that I need to upgrade to 3.6 for the license to work. Another dumb question if you don't mind. What does the license do? Anything I am losing if the license doesn't work? Thanks again!

 

[figure7555]

Thanks for such a quick response! Appreciate!

Oh I didn't know that I need to upgrade to 3.6 for the license to work. Another dumb question if you don't mind. What does the license do? Anything I am losing if the license doesn't work? Thanks again!

You need an Ethernet license if you want to use the switch for Ethernet. Being enterprise gear most of the features are locked behind extra licenses.

The dodgy route guide had a couple licenses and instructions on how to generate your own.

 

[maxermaxer]

You need an Ethernet license if you want to use the switch for Ethernet. Being enterprise gear most of the features are locked behind extra licenses.

The dodgy route guide had a couple licenses and instructions on how to generate your own.

I see thanks! I think I can use ETH mode even I am at this 3.4 image. Not sure if it is because I have got the VPI licensing assistance from the seller and already registered it.

Thanks again!

 

[maxermaxer]

Update: I followed the guide and successfully upgraded the image from 3.4.0012 to 3.6.3004 to 3.6.8012. All done in command line.

I need to change the admin webpage to begin with "https://"

After logging in I got the warning message:

2023/11/04 09:55:51 : Insufficient number of working fans in the system

2023/11/04 09:55:51 : MGMT\F1 speed is below minimal threshold

2023/11/04 09:55:51 : MGMT\F1 speed is below minimal threshold

2023/11/04 09:55:51 : MGMT\F1 speed is below minimal threshold

2023/11/04 09:55:51 : MGMT\F1 speed is below minimal threshold

I checked the fans status and the tempreture. It looks like the CPU monitor2 is very hot! Should I increase the fan speed?





I haven't been able to change the fan speed from command line. I have literally tried all kinds of possible commands. Some with error message like:

switch-a4e73e [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN1 1 65
Nov 04 20:50:34 INFO LOG: Initializing SX log with STDOUT as output file.
-E- mlxi2c: Set Fan power "/MGMT/FAN1" failed : MLXI2C_NOT_EXIST_FAN_TACH (52)

Some are without error message just like:

switch-a4e73e [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN4 1 90
Nov 04 20:52:01 INFO LOG: Initializing SX log with STDOUT as output file.

But I didn't notice any change of the fan speed. Have I missed anything?

 

[maxermaxer]

Can anyone help shed some light?

I was able to open the lid, use a 14cm fan to cool down the CPUBOARD. I did some investigation and found that my switch fans cannot be set the speed no matter what. Hence the RPMs are way below the treadsold.



In command line console the status are all "failed"


However interestingly in the WebUI control panel I can see they are all "OK" (see the bottom of the screen shot)


Does this mean I have four faulty fans? I have checked the fans they are all the original fans come with the EMC version of SX6012.

 

[klui]

There's a lower possibility all your fans have failed rather than I2C connection between the management plane and the fans. Unfortunately I have no idea how to resolve other than removing the pin headers for each fan and validate each fan is OK. Connect them to a motherboard to see if they work by playing around with its fan profile.

If they work then the problem is upstream from the fans.

Those commands work fine on my 6012.

Ambient 24C (75F)

sx6012b [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN1 1 30
Oct 22 04:34:28 INFO    LOG: Initializing SX log with STDOUT as output file.
sx6012b [standalone: master] (config) # show temperature
---------------------------------------------------------
Module      Component              Reg  CurTemp    Status
                                        (Celsius)
---------------------------------------------------------
MGMT        SX                     T1   35.00      OK
MGMT        QSFP_TEMP1             T1   32.00      OK
MGMT        QSFP_TEMP2             T1   34.50      OK
MGMT        QSFP_TEMP3             T1   35.00      OK
MGMT        BOARD_MONITOR          T1   34.00      OK
MGMT        CPU_BOARD_MONITOR      T1   35.00      OK
MGMT        CPU_BOARD_MONITOR      T2   51.00      OK

sx6012b [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN1 1 35
Oct 22 04:52:57 INFO    LOG: Initializing SX log with STDOUT as output file.
sx6012b [standalone: master] (config) # show temperature
---------------------------------------------------------
Module      Component              Reg  CurTemp    Status
                                        (Celsius)
---------------------------------------------------------
MGMT        SX                     T1   32.00      OK
MGMT        QSFP_TEMP1             T1   30.00      OK
MGMT        QSFP_TEMP2             T1   32.50      OK
MGMT        QSFP_TEMP3             T1   33.00      OK
MGMT        BOARD_MONITOR          T1   32.00      OK
MGMT        CPU_BOARD_MONITOR      T1   33.00      OK
MGMT        CPU_BOARD_MONITOR      T2   49.00      OK

sx6012b [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN1 1 40
Oct 22 04:44:11 INFO    LOG: Initializing SX log with STDOUT as output file.
sx6012b [standalone: master] (config) # show temperature
---------------------------------------------------------
Module      Component              Reg  CurTemp    Status
                                        (Celsius)
---------------------------------------------------------
MGMT        SX                     T1   31.00      OK
MGMT        QSFP_TEMP1             T1   30.00      OK
MGMT        QSFP_TEMP2             T1   32.00      OK
MGMT        QSFP_TEMP3             T1   32.00      OK
MGMT        BOARD_MONITOR          T1   31.00      OK
MGMT        CPU_BOARD_MONITOR      T1   32.00      OK
MGMT        CPU_BOARD_MONITOR      T2   48.00      OK

sx6012b [standalone: master] (config) # fae mlxi2c set_fan /MGMT/FAN1 1 45
Oct 22 04:56:48 INFO    LOG: Initializing SX log with STDOUT as output file.
sx6012b [standalone: master] (config) # show temperature
---------------------------------------------------------
Module      Component              Reg  CurTemp    Status
                                        (Celsius)
---------------------------------------------------------
MGMT        SX                     T1   30.00      OK
MGMT        QSFP_TEMP1             T1   29.00      OK
MGMT        QSFP_TEMP2             T1   31.00      OK
MGMT        QSFP_TEMP3             T1   31.50      OK
MGMT        BOARD_MONITOR          T1   31.00      OK
MGMT        CPU_BOARD_MONITOR      T1   31.00      OK
MGMT        CPU_BOARD_MONITOR      T2   47.00      OK

 

[maxermaxer]

Thanks! The fans actually spin. I have checked. I typed this “fae mlxi2c set_fan /MGMT/FAN1 1 XX” but the problem is it doesn’t show error message but the fan speed stay the same.

 

[klui]

Your fans spinning isn't in question. The fact they don't respond to PWM changes is.

My fans obviously spin but more importantly they respond to commands changing their RPM.

 

[bekax]

Hello everyone.
I'm starting to play with a original SX6012 to manage a small cluster.
Something I didn't yet understand is how to enable the full capabilities of the switch.

I used genlicense however it looks like it only gave me "RESTRICTED_CMDS_GEN2" which I'm pretty sure I won't use for now.
And "Generic SX license" which appears to be Ethernet, L2 and L3.

Shouldn't Gateway be enabled together with Eth L2 and L3 ? Or is this a different License ?

 

[longtongue]

Hello. I'm trying to convert SX6012 for MLNXOS according to the instructions Converting Mellanox EMC switch to SX60xx v1.12.docx. Can't boot via TFTP server. Has anyone encountered this?

Using ppc_4xx_eth0 device
TFTP from server 192.168.88.85; our IP address is 192.168.88.79
Filename 'mlnx460ex/vmlinuz'.
Load address: 0x400000
Loading: T T T T T T T T T T
Retry count exceeded; starting again
Waiting for PHY auto negotiation to complete...... TIMEOUT !
done
Using ppc_4xx_eth1 device
TFTP from server 192.168.88.85; our IP address is 192.168.88.79
Filename 'mlnx460ex/vmlinuz'.
Load address: 0x400000
Loading: T T T T T T T T T T
Retry count exceeded; starting again
Using ppc_4xx_eth0 device
TFTP from server 192.168.88.85; our IP address is 192.168.88.79
Filename 'mlnx460ex/vmlinuz'.
Load address: 0x400000
Loading: T T T T T

 

[Terry Wallace]

That means its not talking to your tftp server from the sounds of it.. on your tftp server side there is usually a log,, does it show connection from switch ? request for file ?

 

[Rand__]

>Waiting for PHY auto negotiation to complete...... TIMEOUT !

That does not sound too good - u sure its actually connected? Can you ping?

 

[longtongue]

That means its not talking to your tftp server from the sounds of it.. on your tftp server side there is usually a log,, does it show connection from switch ? request for file ?

>Waiting for PHY auto negotiation to complete...... TIMEOUT !

That does not sound too good - u sure its actually connected? Can you ping?

Thanks for answers. I figured it out. The problem was with the Microtik router. I had to connect directly to the PC. After updating the MLNXOS firmware, the connection with the router was established in normal mode.