Mellanox/EMC SX6012 - Revival Attempt

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
New project. I received an SX6012 for shipping costs only. Tried to get it running and to get a console today.

First of all, the switch starts. Both fans in the middle are running 100%. Both fans in front of the power supplies are running on minimum rpm.

Mgmt port shows straight green and amber light which both stay on straight, no blinking. The console port below of course shows nothing. No status lights are on and it won't change during the boot(?) process. I waited 5x a 10min to boot up the switch, but console connected to my laptop via DB9/RJ45 cable never comes up. I checked if there's an IP on the switch. No indication there's actually a valid setup running. Even with the mac address, the switch wasn't reachable. Fans rpm do not change while they should switch down after 2-3min at worst. So I'm unable to see if it's actually booting at all. Full reset was done twice as per manual.

I know there are some gurus here in the forum. Is there any other way to see if the switch is actually booting and to be able to configure the switch. Any tip is highly appreciated. Thank you in advance.
 
  • Like
Reactions: Patrick

necr

Active Member
Dec 27, 2017
151
47
28
124
No status lights, dead console? I’d start from power, check the boards for signs of damage, check PSU by PSU. Lots of tinkering probably with unpredictable results.
 
  • Like
Reactions: gb00s

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Thanks for the quick reply. Right side power supply works. Left side power supply goes on and off in 2-3sec sequence. I just noticed this morning that one amber LED on the top board is on. But i have no diagram or error code sheet. I also have no sheet what the PSU should deliver. Ok, this really sounds like a project.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Did you use the concole cable on another switch or device/do you know if the cable really works? Asking because I bought some cheap console usb cables from amazon and they all are dead/not working.

I'm not sure if there's another way to bring back the switch without the console or opening the switch...
Maybe @fohdeesha has some ideas? :D
 

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Thanks for the idea. Just tested the cable on a Brocade switch and its just working fine. What makes me thinking is that the two middle fans of the switch never ’calm’ down and the status lights never come on. So this thing might not even boot. I just can’t see it. Read somewhere else that even booting with errors, the fans are calming down after 2-3 minutes. Somebody else suggested to connect to interface 1 with QSFP+. But lights from MGmt port stay on. So ... Scanned for ip ranges that i might get somewhere. Nothing. Tried with mac address. I get nothing from there as well.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Btw don't forget this is an infinband switch in default configuration, troubleshooting it like a normal ethernet switch might not work.
 
  • Like
Reactions: gb00s

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Btw don't forget this is an infinband switch in default configuration, troubleshooting it like a normal ethernet switch might not work.
Yes and no. I don't have any experience with Mellanox IB switches. But I believe from what I read so far, if there's a valid setup with an IP, I should find it on the Mgmnt port the 'Ethernet' way.
I use TerraTerm for console display, you may need to scroll to the bottom to see any output.
I used TerraTerm, Putty of course on Win and tried to connect with Putty (Linux) and 'cu' and/or 'screen' from Linux. But, the scrolling idea could be the embarrassing part. Will have to test it if back home.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
Do you have an EMC SX6012 (black faceplate) or a genuine Mellanox SX6012 (black and blue faceplate)?
 
  • Like
Reactions: gb00s

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
if the fan speeds never change, there's no console output, and the LEDs never change state, it's not booting anything - sounds like the bootloader was wiped or something. I know these are powerPC (very similar CPU to the icx6610), you could JTAG a bootloader back on the thing if it has a header - can you take a picture of the inside? if they were nice enough to break out the JTAG pins, it'd be a ~16 pin header somewhere
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
Here are the relevant headers.

JP1: SX JTAG
JP2: PMBUS
JP3: CPLD JTAG
J38: CPU BDI

SX6012 headers.jpg
 
  • Like
Reactions: gb00s

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Do you have an EMC SX6012 (black faceplate) or a genuine Mellanox SX6012 (black and blue faceplate)?
I suppose it's an EMC as the serial starts with MT..... and it has a black faceplate. Thanks for making that clear. Didn't come to my mind until now.
if the fan speeds never change, there's no console output, and the LEDs never change state, it's not booting anything - sounds like the bootloader was wiped or something. I know these are powerPC (very similar CPU to the icx6610), you could JTAG a bootloader back on the thing if it has a header - can you take a picture of the inside? if they were nice enough to break out the JTAG pins, it'd be a ~16 pin header somewhere
That's what I thought that it doesn't even get into booting. Unfortunately, I have to read into flashing of these switches first.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
neat! not only is there a jtag header, they even named it after my unit! (BDI2000). I'd have to ship you mine to borrow...
Note I am not the individual who has the issue. But if there's anything you'd like me to do using your programmer I'm willing to help.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
I suppose it's an EMC as the serial starts with MT..... and it has a black faceplate. Thanks for making that clear. Didn't come to my mind until now.
The EMC SX6012s configure their management ports with the static IP 172.16.255.150/16. But their stack is flakey. Not sure why. Maybe it was meant to work in a cluster.

Switch-X(4)> ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
inet 127.0.0.1 netmask 0xff000000
eth0: flags=843<UP,BROADCAST,RUNNING,SIMPLEX> mtu 1500
inet 172.16.255.150 netmask 0xffff0000 broadcast 172.16.255.255
Switch-X(4)> ping -c 1 172.16.255.150
PING 172.16.255.150: 56 data bytes
64 bytes from 172.16.255.150: icmp_seq=0 ttl=255 time=98.930 ms
--- 172.16.255.150 ping statistics ---
1 packets transmitted, 1 packets received,
Switch-X(4)> ping -c 1 172.16.255.150
PING 172.16.255.150: 56 data bytes
64 bytes from 172.16.255.150: icmp_seq=0 ttl=255 time=8826.839 ms
--- 172.16.255.150 ping statistics ---
1 packets transmitted, 1 packets received,
Switch-X(4)> ping -c 1 172.16.255.150
PING 172.16.255.150: 56 data bytes
no answer from 172.16.255.150
Switch-X(4)> ping -c 1 172.16.255.150
PING 172.16.255.150: 56 data bytes
no answer from 172.16.255.150
Switch-X(4)> ping -c 1 172.16.255.150
PING 172.16.255.150: 56 data bytes
64 bytes from 172.16.255.150: icmp_seq=0 ttl=255 time=28348.375 ms
--- 172.16.255.150 ping statistics ---
1 packets transmitted, 1 packets received,
Switch-X(4)>
 
  • Like
Reactions: gb00s

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
What I noticed, after the start the amber LED on the main board is very weak, flickery ... like an HDD activity indicator LED. After ~30min it stays strong and straight up. Hmmm ...

Video 1 and Video 2

And yes I have the same headers like @klui

IMG_20210106_222507.jpg

Thanks guys.
 
Last edited:

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Adding to the previous post, I had never ever any indication from the green LEDs for VCORE, 3.3V AUX or 1.8V PG. Now I started the switch without all fans and then added them one by one and got all LEDs from above one-by-one on. Status lights remain dark and still no console.

Video >> You can hear the rpm in-/decreases with the flickering of the green LED and stabilizes then.

PS: ... and now the left power supply works :rolleyes::oops:
 
Last edited:

necr

Active Member
Dec 27, 2017
151
47
28
124
I’m still thinking the issue is before the boot loader. I would measure the outputs of both PSUs, if they’re okay I would unmount the PPC daughter board and check if the fans work okay and there’s activity on the base board (maybe try I2C). Only then I’d continue checking the daughter board. Seen similar behavior before, in my case it was a short circuit .
 

RageBone

Active Member
Jul 11, 2017
617
159
43
@gb00s
the green LEDs with the voltage labels that are visible in your last video with the fan ramping up and down, make me think that you have a fundamental board and power related issue on the switch.
Meaning an issue even before anything could output anything on any console, so i doubt that JTAG or any other software things you are planing will do the trick.

The varying amounts of RPM and light activity make me think of dead Caps shorting out a Voltage rail.
Have a look for bulged Caps and Zener-Diodes.
I'd also recommend grabbing a multi-meter and probing for the Power-supply-outputs and rails on the switch.
Pictures of everything and i mean everything, with details and such are appreciated so that we can have look at it too.
 
  • Like
Reactions: NablaSquaredG

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
First of all thanks for all the comments, ideas and thoughts. There is progress. Just to add something here, I received the information, that according to the serial number, this switch should be an engineering sample from EMC.

I checked both PSUs. At first, the left and 'faulty' PSU didn't provide the same voltage like the right and good PSU. Cleaned. Working. I took of the daughterboard off. Changed fans. Cleaned contacts everywhere. I still have the amber LED on the daughterboard on and I don't know what's it for. But it 'shines' light and bright. Then I started the switch only with the 'faulty' but now 'ok' PSU. Working but no console. Then I started the switch only with the right PSU. Voila, switch boots and I receive a console. Then booted with both and with the faulty PSU again, both give me a console. 'Also, after cleaning around a little bit, the 3 green LEDs with the voltage labels are also shining bright without flickering now, and I have a console now.

Video - LEDs / Video Console

But it's now around 15-20min past the console post and fans are still running 100% and there's no progress from the boot in console.

console_emc.png

Needs further digging .
 
Last edited:
  • Like
Reactions: fohdeesha