Beware of EMC switches sold as Mellanox SX6XXX on eBay

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

andvalb

Member
Feb 15, 2021
27
25
13
Ulyanovsk, Russian Federation
Thank you for all the tips @andvalb, highly appreciated. Been spending the better part of today on research about BDI 2000. Also been messaging people via e-mail, twitter, forums, and also forums I just registered for, about pp4 firmware and logic file. Found ppc version over on eevblog but that is for a different lineage of CPUs. Archive.org says former CEOs of Abatron AG in Switzerland are Max Vock and Rudolf Dummermuth. If desperate enough, might even try to call them up in their retirement and ask for help. Also already owning a sizeable collection of AMCC 460EX BDI configuration files.

Fortunately I have a dump from eetool which seems to be a serial EEPROM storing MAC addresses, serial numbers etc. so re-populating u-boot vars shall not be a problem. My view of the switch's complex is
  • 2 GB DDR2 ECC RAM
  • 16 MB NOR flash (U-Boot etc.) a JS28F128 on the backside of the CPU daughterboard
  • 1024 MB SLC NAND flash a Samsung K9WAG08U1D on topside CPU daughterboard
  • 24C02 256 byte EEPROM holding 0x52 "Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)" topside CPU daughterboard
  • 24C32 4KB EEPROM holding 0x50 "fru_cpu.bin" topside CPU daughterboard
  • 24C32 4KB EEPROM holding 0x51 "fru_backplate.bin" on motherboard U33 next to daughterboard connector
Will be interesting to see what manufacture.sh has damaged. I think I have a backup of every crucial bit of data in those chips.

NOR-Layout:
Creating 6 MTD partitions on "4ff000000.nor_flash":
0x00000000-0x001e0000 : "KERNEL_1"
0x001e0000-0x00200000 : "FDT_1"
0x00200000-0x003e0000 : "KERNEL_2"
0x003e0000-0x00400000 : "FDT_2"
0x00f80000-0x00fa0000 : "UBOOTENV"
0x00fa0000-0x01000000 : "UBOOT"

NAND-Layout:
Creating 4 MTD partitions on "4e0000000.ndfc.nand":
0x00000000-0x20000000 : "ROOT_1"
0x20000000-0x40000000 : "ROOT_2"
0x40000000-0x46400000 : "CONFIG"
0x46400000-0x7c000000 : "VAR"

Will be sidetracked for the next two weeks, switch will soon live again I think. This kind of hardware build quality deserves it.
This is a NOR content.
Rename extensions from zip to z01 ( mlx_sx6012_mtd_z01.zip must be named as mlx_sx6012_mtd.z01 and etc.)
 

Attachments

Last edited:
  • Like
Reactions: Stephan and klui

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Thank you for the files! They need to be renamed and opened in 7-Zip. Usually I just do copy /b a+b+c final.zip but did not work here.

Jury is still out on PP4 for BDI 2000. My first choice. Lucky me, on SX6018 the pins are still there, but no connector is populated. Also closeby suspiciously resistors and capacitors are missing. On SX6012 the connector is populated. Have not verified pinout so far, but 16-pin on BDI 2000 and 16-pin on the board, and BDI on silkscreen make me hopeful.

Another avenue I am exploring is a Teensy++ 2.0, "NORway" firmware + Python + Strata-Flash adaption (ugh), and then use a TSOP56 "360 clip" to in-system rewrite the flash. There is a jumper on the main board called J39 and if that is closed, switch appears to no longer get out of reset. Everything continues to be lit up like a christmas tree continuously. Seems perfect.

1645974834428.png

Yet another little known avenue would be to use the JTAG Boundary Scan Description Language (BDSL), if I had the description for the 460EX processor, and then use a generic JTAG USB interface to bitbang the NOR flash. But that is a hell I don't want to enter. Because not only do I not have the BDSL description, but also haven't looked at how the CPU interfaces with the NOR flash (could be identical to Canyonlands eval board, but ugh try to find documentation on that after 15 years of EOL), and finally figure out how to rewrite the flash through signals in the first place. If that is the only option, I'll pull a Rossmann and start desoldering with flux and heatgun.
 
  • Like
Reactions: andvalb

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Well what do you know, X-mas in February.

A very generous person sent me the correct PowerPC4xx BDI 2000 firmware for this platform. Even the very last release 1.25 from 2014. I am so thrilled. Like an archeologist who just discovered a new entrance to a pyramid. For posterity, reading this on archive.org around the year 2030, the files you need are b20pp4gd.125 (firmware) and pp4jed20.104 or pp4jed21.104 (CPLD) depending whether you have rev B oder rev C BDI 2000.

So I bought a BDI 2000 rev C from Texas on ebay yesterday. Just like the SX6012 itself a device built to last. Rev C will work down to 1.8V system Vcc instead of just 3.0V with Rev B. Taken together with some 13 QSFP nickel plugs by TE to complete my collection, this ran the same amount of what I paid for the switches. But just 10 years earlier it would have cost 20k for two switches with licenses and another (guessing) 10k for the BDI. Two pennies on the dollar.

Will write a BDI config for it and should I get it working, will post it here.

I've also taken a closer look at the 16-pin "CPU BDI" socket yesterday and at least GND and Vcc_target (3.3V) appear to be as specified in the BDI documentation. They talk about a "PowerPC 4xx JTAG connector specification" and by the looks of it in other ppc405 PDFs I found, this appears to have been proposed by IBM for their RISCWatch debugger.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
Hmm...

So I got what apparantly is a Mellanox Engineering Sample Switch (x86 CPU, probably an SB7700 ES)
It has... interesting things on it like

Code:
0: OPT-OS Testing and Diagnostic Environment X86_64 opt_os_20160921 2016-09-21
1: MLNX-OS Operating Environment X86_64 opt_os_20160921
So now I'd like to flash the standard MLNX-OS on it, but I don't really have much experience with flashing Mellanox Switches.

I tried running manufacture.sh and remanufacture.sh from the opt-os testing environment, but manufacture.sh does not run because *** Must specify -P if not run from the manufacturing environment

I can't run anything from the MLNX-OS Operating Environment because it does not initialise properly

I have backed up the SATA DOM already, just in case we brick the switch entirely :D

Any ideas on how to proceed?
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
FWIW, there are undocumented escape sequences in the motd banner

Code:
 banner login "LOGIN BANNER a \a b \b c \c d \d e \e f \f g \g h \h i \i j \j k \k l \l m \m n \n o \o p \p q \q r \r s \s t \t u \u v \v w \w x \x y \y z \z"
is transformed to

Code:
LOGIN BANNER a a b 9600 c c d Wed Mar 2  2022 e e f f g g h h i i j j k k l ttyS0 m ppc n switch-d446d4 o (none) p p q q r 3.10.94-MELLANOXuni-m460ex s Linux t 20:24:21 u 0  v PPC_M460EX jenkins #1 2019-02-13 12:36:41 w w x x y y z z
Code:
\b -> Baudrate
\d -> Date
\l -> Terminal
\m -> Arch
\n -> hostname
\o -> ?
\r -> Firmware Info?
\s -> OS?
\t -> Time
\u -> ?
\v -> Builddate?
 
  • Like
Reactions: Labs

andvalb

Member
Feb 15, 2021
27
25
13
Ulyanovsk, Russian Federation
Hmm...

So I got what apparantly is a Mellanox Engineering Sample Switch (x86 CPU, probably an SB7700 ES)
It has... interesting things on it like

Code:
0: OPT-OS Testing and Diagnostic Environment X86_64 opt_os_20160921 2016-09-21
1: MLNX-OS Operating Environment X86_64 opt_os_20160921
So now I'd like to flash the standard MLNX-OS on it, but I don't really have much experience with flashing Mellanox Switches.

I tried running manufacture.sh and remanufacture.sh from the opt-os testing environment, but manufacture.sh does not run because *** Must specify -P if not run from the manufacturing environment

I can't run anything from the MLNX-OS Operating Environment because it does not initialise properly

I have backed up the SATA DOM already, just in case we brick the switch entirely :D

Any ideas on how to proceed?
As the first step - check that such hardware is supported by the available MLNX-OS.
Then check what this -P is.
 

bentwire

New Member
Feb 19, 2022
9
3
3
@NablaSquaredG

I could try and image the SATA SSD in my SX6720 and post it somewhere if you want to try to just run it.

Not sure I have the hardware, may have to try to get it to boot off USB to image it that way.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
@bentwire

Yeah that would be amazing!
Best way would probably to image the entire disk with dd and upload it

I can provide you a folder in my OneDrive, that would probably be the easiest way

BTW: You have a SX6720? o_O I'd love to swap my SX6036 against those, but it's just too expensive to justify... And there's not a ton of those
 

bentwire

New Member
Feb 19, 2022
9
3
3
@bentwire

Yeah that would be amazing!
Best way would probably to image the entire disk with dd and upload it

I can provide you a folder in my OneDrive, that would probably be the easiest way

BTW: You have a SX6720? o_O I'd love to swap my SX6036 against those, but it's just too expensive to justify... And there's not a ton of those
Yeah I need to find a way to either find a way in to the BIOS so I can boot a normal linux off a USB stick to do the DD, or I need to get an adapter for the SSD, don't currently have a way to hook it to my Linux boxen...

Yeah the SX6720 is awesome, except for one thing.... The PSU fans won't go any slower than ~14500 RPM. The thing is deafening.

If I could get another one for around the $20 I payed for this one (Plus 90 to ship...) I would be more apt to try hacking the fans. As it is the only other ones I see on EBAY are 100X what I payed hah...
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
Do you have a free SATA 2.5" port in your machine?

I could order a 2.5" SATA to mSATA adapter on Amazon and deliver it to you in exchange for the drive image
 

bentwire

New Member
Feb 19, 2022
9
3
3
Do you have a free SATA 2.5" port in your machine?
I do.

I could order a 2.5" SATA to mSATA adapter on Amazon and deliver it to you in exchange for the drive image
Send me a link to the one you think is best, I can purchase it. I have a Celestica switch I may be able to use it with to try and get that thing working.

You don't happen to have an ONIE/ICOS image do you? :)
 

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
"Rescuing a switch made by a company that no longer exists, which uses a CPU that is no longer made, which was by a company that also no longer exists, using a JTAG debugger by yet another company, that also no longer exists."

aaa.jpg

The BDI 2000 was in the mail today, so I reflashed it right away to PPC400 on Linux using bdisetup V1.27. Then some serious hair pulling with a few of Abatron's quirks started. Also funny Intel, the NOR flash will come up completely locked, needs unlocking first before you can erase and program a new U-Boot.

I am leaving the required BDI config and a register definition file here for posterity. The switch lives... ;-) Could also work for other 460EX Mellanoxes. Files need to be put on TFTP where the BDI can grab them. Also you'll need the uboot.bin you want to flash. Then it's all just telnet into the BDI and
update ; reset ; erase ; prog ; md 0xfffa0000 ; md 0xffffff00
reset ; go 0xfffffffc
First line will rewrite U-Boot and dump a few bytes from start and end of NOR so you can see if it worked, second line will start U-Boot.

Even though AMCC is dead (sold to Macom, which immediately sold the CPU biz to an investor LLC, which did unknown stuff with it, maybe give or use it at Ampere), and even though the people implementing Linux for this platform are also gone, through reading the investor news bulletin of the AMCC/Macom merger I stumbled upon a German sub entity of AMCC. Which led me, after guessing Macom's email schema, to an AMCC PPC engineer from back in the day. Who was super kind to send me the 460EX User's Manual and also a spiffy Excel sheet to calculate the bootstrap EEPROM bits. Which all came in very handy because the BDI config contains mandatory TLB and 16-bit NOR flash setup, but without the manual you will never figure out what all the bits mean.

If you need any of the other mentioned files, send me a message.

Code:
; bdiGDB configuration file for Mellanox SwitchX-2 AMCC 460EX (Dingo)

[INIT]
WTLB        0xFF000075  0x4FF0003F      ; Boot space 16 MB (entire NOR flash)
WTLB        0x80000055  0x4000003F      ; SRAM and OCM space 1 MB (256 KB L2 + 64 KB OCM usable)

WDCR        0x12    0x10                ; Select EBC0_B0AP
WDCR        0x13    0x10055E00            ; B0AP: Flash (manual page 872: 1 00000000 0 10 10 10 11 11 000 0 0 0 0 00000)
WDCR        0x12    0x00                ; Select EBC0_B0CR
WDCR        0x13    0xFF09A000          ; B0CR: 16 MB, 16 Bit at 0xFF000000 (manual page 869: 111111110000 100 11 01 0000000000000)

WDCR        0x030   0x00000008          ; Disable L2 cache, enable SRAM R/W access
WDCR        0x020   0x00000984            ; Base address, size 64 KB, R/W access
WDCR        0x021   0x00010984            ; Base address, size 64 KB, R/W access
WDCR        0x022   0x00020984            ; Base address, size 64 KB, R/W access
WDCR        0x023   0x00030984            ; Base address, size 64 KB, R/W access
WDCR        0x02A   0x00000000            ; Disable parity checking
WDCR        0x0B0   0x00040984            ; Base address, size 64 KB, R/W access

WM32        0x80000000  0x48000000      ; Write opcode "b $pc" to SRAM
WREG        PC          0x80000000      ; Set PC to this infinite loop

WM16        0xFFFA0000  0x0060          ; Flash powers up with all sectors locked, unlock U-Boot sectors
WM16        0xFFFA0000  0x00D0
WM16        0xFFFC0000  0x0060
WM16        0xFFFC0000  0x00D0
WM16        0xFFFE0000  0x0060
WM16        0xFFFE0000  0x00D0
WM16        0xFFFE8000  0x0060
WM16        0xFFFE8000  0x00D0
WM16        0xFFFF0000  0x0060
WM16        0xFFFF0000  0x00D0
WM16        0xFFFF8000  0x0060
WM16        0xFFFF8000  0x00D0
WM16        0xFFFA0000  0xFFFF          ; Select read mode

[TARGET]
JTAGCLOCK    0                           ; BDI2000 16.6 MHz JTAG clock
CPUTYPE        440 FPU                     ; Target CPU type
WORKSPACE   0x80040000                  ; Workspace in OCM to dump FP registers
WAKEUP        500                         ; Wakeup time after reset
BREAKMODE    HARD                        ; Hardware breakpoint
STEPMODE    HWBP                        ; Hardware breakpoints

[HOST]
IP          192.168.128.100             ; TFTP server address
DUMP        /mlnx460ex/dump.bin
PROMPT      460EX>

[FLASH]
FILE        /mlnx460ex/uboot.bin        ; File on tftp server
CHIPTYPE    STRATAX16                   ; SX6012 uses a JS28F128P33TF70
CHIPSIZE    0x1000000                   ; Flash size 16 MB @ 0xFF000000
BUSWIDTH    16                          ; Flash memory bus width in bits
WORKSPACE   0x80040100                  ; Workspace in OCM for fast programming algorithm
FORMAT      BIN 0xFFFA0000              ; U-Boot file format is binary
ERASE       0xFFFA0000                  ; Erase top-most 128 + 128 + 32 + 32 + 32 + 32 KB
ERASE       0xFFFC0000
ERASE       0xFFFE0000
ERASE       0xFFFE8000
ERASE       0xFFFF0000
ERASE       0xFFFF8000

[REGS]
FILE        /mlnx460ex/reg460ex.def
IDCR1        0x010   0x011               ; SDRAM0_CFGADDR, SDRAM0_CFGDATA
IDCR2        0x012    0x013               ; EBC0_CFGADDR, EBC0_CFGDATA
IDCR3        0x00C    0x00D               ; CPR0_CFGADDR, CPR0_CFGDATA
IDCR4        0x00E    0x00F               ; SDR0_CFGADDR, SDR0_CFGDATA
Code:
;Register definition for PPC460EX
;================================
;
; name: user defined name of the register
; type: the type of the register
;   GPR general purpose register
;   SPR special purpose register
;   MM  memory mapped register
;   DMMx    direct memory mapped register with offset
;       x = 1..4
;       the base is defined in the configuration file
;       e.g. DMM1 0x02200000
;   IDCRx   indirect accessed DCR's
;       x = 1..4
;       the addr and data DCR is defined in the configuration file
;       e.g. IDCR1 0x010 0x011
; addr: the number, adddress or offset of the register
; size  the size of the register (8,16 or 32)
;
;name           type    addr            size
;-------------------------------------------
;
sp      GPR 1
;
;   Special Purpose Registers
;
xer             SPR     0x001
lr              SPR     0x008
ctr             SPR     0x009
dec             SPR     0x016
srr0            SPR     0x01a
srr1            SPR     0x01b
pid             SPR     0x030
decar           SPR     0x036
csrr0           SPR     0x03a
csrr1           SPR     0x03b
dear            SPR     0x03d
esr             SPR     0x03e
ivpr            SPR     0x03f
usprg0          SPR     0x100
sprg4r          SPR     0x104
sprg5r          SPR     0x105
sprg6r          SPR     0x106
sprg7r          SPR     0x107
tblr            SPR     0x10c
tbur            SPR     0x10d
sprg0           SPR     0x110
sprg1           SPR     0x111
sprg2           SPR     0x112
sprg3           SPR     0x113
sprg4w          SPR     0x114
sprg5w          SPR     0x115
sprg6w          SPR     0x116
sprg7w          SPR     0x117
tblw            SPR     0x11c
tbuw            SPR     0x11d
pir             SPR     0x11e
pvr             SPR     0x11f
dbsr            SPR     0x130
dbcr0           SPR     0x134
dbcr1           SPR     0x135
dbcr2           SPR     0x136
iac1            SPR     0x138
iac2            SPR     0x139
iac3            SPR     0x13a
iac4            SPR     0x13b
dac1            SPR     0x13c
dac2            SPR     0x13d
dvc1            SPR     0x13e
dvc2            SPR     0x13f
tsr             SPR     0x150
tcr             SPR     0x154
ivor0           SPR     0x190
ivor1           SPR     0x191
ivor2           SPR     0x192
ivor3           SPR     0x193
ivor4           SPR     0x194
ivor5           SPR     0x195
ivor6           SPR     0x196
ivor7           SPR     0x197
ivor8           SPR     0x198
ivor9           SPR     0x199
ivor10          SPR     0x19a
ivor11          SPR     0x19b
ivor12          SPR     0x19c
ivor13          SPR     0x19d
ivor14          SPR     0x19e
ivor15          SPR     0x19f
mcsrr0          SPR     0x23a
mcsrr1          SPR     0x23b
mcsr            SPR     0x23c
inv0            SPR     0x370
inv1            SPR     0x371
inv2            SPR     0x372
inv3            SPR     0x373
itv0            SPR     0x374
itv1            SPR     0x375
itv2            SPR     0x376
itv3            SPR     0x377
ccr1            SPR     0x378
dnv0            SPR     0x390
dnv1            SPR     0x391
dnv2            SPR     0x392
dnv3            SPR     0x393
dtv0            SPR     0x394
dtv1            SPR     0x395
dtv2            SPR     0x396
dtv3            SPR     0x397
dvlim           SPR     0x398
ivlim           SPR     0x399
rstcfg          SPR     0x39b
dcdbtrl         SPR     0x39c
dcdbtrh         SPR     0x39d
icdbtrl         SPR     0x39e
icdbtrh         SPR     0x39f
mmucr           SPR     0x3b2
ccr0            SPR     0x3b3
icdbdr          SPR     0x3d3
dbdr            SPR     0x3f3
;
;
;    Indirectly Accessed DCR's
;
;       IDCR1 must be set to SDRAM0_CFGADDR and SDRAM0_CFGDATA
;       IDCR2 must be set to EBC0_CFGADDR   and EBC0_CFGDATA
;       IDCR3 must be set to EBM0_CFGADDR   and EBM0_CFGDATA
;       IDCR4 must be set to PPM0_CFGADDR   and PPM0_CFGDATA
;       IDCR5 must be set to CPR0_CFGADDR   and CPR0_CFGDATA
;       IDCR6 must be set to SDR0_CFGADDR   and SDR0_CFGDATA
;
; External Bus Controller DCRs
;
ebc0_b0cr       IDCR2    0x00
ebc0_b1cr       IDCR2    0x01
ebc0_b2cr       IDCR2    0x02
ebc0_b3cr       IDCR2    0x03
ebc0_b4cr       IDCR2    0x04
ebc0_b5cr       IDCR2    0x05
ebc0_b6cr       IDCR2    0x06
ebc0_b7cr       IDCR2    0x07
ebc0_b0ap       IDCR2    0x10
ebc0_b1ap       IDCR2   0x11
ebc0_b2ap       IDCR2    0x12
ebc0_b3ap       IDCR2    0x13
ebc0_b4ap       IDCR2    0x14
ebc0_b5ap       IDCR2    0x15
ebc0_b6ap       IDCR2    0x16
ebc0_b7ap       IDCR2    0x17
ebc0_bear       IDCR2    0x20
ebc0_besr       IDCR2    0x21
ebc0_cfg        IDCR2    0x23
ebc0_cid        IDCR2    0x24
;
; Clocking and PowerOn Reset DCR
;
cpr0_clkupd     IDCR5   0x0020
cpr0_pllc       IDCR5    0x0040
cpr0_plld       IDCR5    0x0060
cpr0_plbed      IDCR5    0x0080
cpr0_plb2d      IDCR5    0x00A0
cpr0_opbd       IDCR5    0x00C0
cpr0_perd       IDCR5    0x00E0
cpr0_ahbd       IDCR5    0x0100
cpr0_icfg       IDCR5    0x0140
;
; System DCR
;
sdr0_sdstp0     IDCR6   0x0020
sdr0_sdstp1     IDCR6    0x0021
sdr0_pinstp     IDCR6    0x0040
sdr0_sdcs       IDCR6    0x0060
sdr0_ecid0      IDCR6    0x0080
sdr0_ecid1      IDCR6    0x0081
sdr0_ecid2      IDCR6    0x0082
sdr0_jtag       IDCR6    0x00C0
sdr0_ddrdl      IDCR6    0x00E0
sdr0_ebc        IDCR6    0x0100
sdr0_uart0      IDCR6    0x0120
sdr0_uart1      IDCR6    0x0121
sdr0_uart2      IDCR6    0x0122
sdr0_uart3      IDCR6    0x0123
sdr0_cp440      IDCR6    0x0180
sdr0_xcr        IDCR6    0x01C0
sdr0_xpllc      IDCR6    0x01C1
sdr0_xplld      IDCR6    0x01C2
sdr0_srst0      IDCR6    0x0200
sdr0_srst1      IDCR6    0x0201
sdr0_slpipe     IDCR6    0x0220
sdr0_amp        IDCR6    0x0240
sdr0_mirq0      IDCR6    0x0260
sdr0_mirq1      IDCR6    0x0261
sdr0_maltbl     IDCR6    0x0280
sdr0_malrbl     IDCR6    0x02A0
sdr0_maltbs     IDCR6    0x02C0
sdr0_malrbs     IDCR6    0x02E0
sdr0_pci0    IDCR6    0x0300
sdr0_usb2d0cr    IDCR6    0x0320
sdr0_usb2h0cr    IDCR6    0x0340
sdr0_cust0      IDCR6    0x4000
sdr0_sdstp2     IDCR6    0x4001
sdr0_cust1      IDCR6    0x4002
sdr0_sdstp3     IDCR6    0x4003
sdr0_pfc0       IDCR6    0x4100
sdr0_pfc1       IDCR6    0x4101
sdr0_usb2phy0cr IDCR6    0x4103
sdr0_plbtr      IDCR6    0x4200
sdr0_mfr        IDCR6    0x4300
sdr0_usb2host    IDCR6    0x4600
;
; Internal SRAM Controller
;
sram0_sb0cr     DCR     0x020
sram0_sb1cr     DCR     0x021
sram0_sb2cr     DCR     0x022
sram0_sb3cr     DCR     0x023
sram0_bear      DCR     0x024
sram0_besr0     DCR     0x025
sram0_besr1     DCR     0x026
sram0_pmeg      DCR     0x027
sram0_dpc       DCR     0x0a0
;
; Clock and Power Management
;
cpm0_er         DCR     0x160
cpm0_fr         DCR     0x161
cpm0_sr         DCR     0x162
;
; Double Data Rate (DDR) SDRAM Controller
;
mq0_b0bas       DCR     0x0040
mq0_b1bas       DCR     0x0041
mq0_b2bas       DCR     0x0042
mq0_b3bas       DCR     0x0043
mq0_cf1h        DCR     0x0045
mq0_esh         DCR     0x0047
mq0_eauh        DCR     0x0048
mq0_ealh        DCR     0x0049
mq0_baul    DCR     0x004A
mq0_cf1l    DCR     0x004B
mq0_esl        DCR     0x004C
mq0_eaul    DCR     0x004D
mq0_eall    DCR     0x004E
mq0_cfbhl    DCR     0x004F
mq0_bauh    DCR     0x0050
;
mcif0_mcstat    IDCR1   0x0014
mcif0_mcopt1    IDCR1    0x0020
mcif0_mcopt2    IDCR1    0x0021
mcif0_modt0    IDCR1    0x0022
mcif0_modt1    IDCR1    0x0023
mcif0_modt2    IDCR1    0x0024
mcif0_modt3    IDCR1    0x0025
mcif0_codt    IDCR1    0x0026
mcif0_vvpr    IDCR1    0x0027
mcif0_opar1    IDCR1    0x0028
;mcif0_opar2    IDCR1    0x0029
mcif0_rtr    IDCR1    0x0030
mcif0_mb0cf    IDCR1    0x0040
mcif0_mb1cf    IDCR1    0x0044
mcif0_mb2cf    IDCR1    0x0048
mcif0_mb3cf    IDCR1    0x004C
mcif0_initplr0    IDCR1    0x0050
mcif0_initplr1    IDCR1    0x0051
mcif0_initplr2    IDCR1    0x0052
mcif0_initplr3    IDCR1    0x0053
mcif0_initplr4    IDCR1    0x0054
mcif0_initplr5    IDCR1    0x0055
mcif0_initplr6    IDCR1    0x0056
mcif0_initplr7    IDCR1    0x0057
mcif0_initplr8    IDCR1    0x0058
mcif0_initplr9    IDCR1    0x0059
mcif0_initplr10    IDCR1    0x005A
mcif0_initplr11    IDCR1    0x005B
mcif0_initplr12    IDCR1    0x005C
mcif0_initplr13    IDCR1    0x005D
mcif0_initplr14    IDCR1    0x005E
mcif0_initplr15    IDCR1    0x005F
mcif0_rqdc     IDCR1    0x0070
mcif0_rfdc     IDCR1    0x0074
mcif0_rdcc      IDCR1    0x0078
mcif0_dlcr    IDCR1    0x007A
mcif0_clktr    IDCR1    0x0080
mcif0_wrdtr    IDCR1    0x0081
mcif0_sdtr1    IDCR1    0x0085
mcif0_sdtr2    IDCR1    0x0086
mcif0_sdtr3    IDCR1    0x0087
mcif0_mmode    IDCR1    0x0088
mcif0_memode    IDCR1    0x0089
mcif0_ecces    IDCR1    0x0098
mcif0_fcsr    IDCR1    0x00B0
mcif0_rtsr    IDCR1    0x00B1
;
;
; Memory Mapped Peripherals
; =========================
;
; PMM1 must point to Peripheral space (real 4_0000_0000)
;
; GPIO
;
gpio0_or        PMM1    0xEF600B00
gpio0_tcr       PMM1    0xEF600B04
gpio0_osrl      PMM1    0xEF600B08
gpio0_osrh      PMM1    0xEF600B0C
gpio0_tsrl      PMM1    0xEF600B10
gpio0_tsrh      PMM1    0xEF600B14
gpio0_odr       PMM1    0xEF600B18
gpio0_ir        PMM1    0xEF600B1C
gpio0_rr1       PMM1    0xEF600B20
gpio0_rr2       PMM1    0xEF600B24
gpio0_rr3       PMM1    0xEF600B28
gpio0_isr1l     PMM1    0xEF600B30
gpio0_isr1h     PMM1    0xEF600B34
gpio0_isr2l     PMM1    0xEF600B38
gpio0_isr2h     PMM1    0xEF600B3C
gpio0_isr3l     PMM1    0xEF600B40
gpio0_isr3h     PMM1    0xEF600B44
;
gpio1_or        PMM1    0xEF600C00
gpio1_tcr       PMM1    0xEF600C04
gpio1_osrl      PMM1    0xEF600C08
gpio1_osrh      PMM1    0xEF600C0C
gpio1_tsrl      PMM1    0xEF600C10
gpio1_tsrh      PMM1    0xEF600C14
gpio1_odr       PMM1    0xEF600C18
gpio1_ir        PMM1    0xEF600C1C
gpio1_rr1       PMM1    0xEF600C20
gpio1_rr2       PMM1    0xEF600C24
gpio1_rr3       PMM1    0xEF600C28
gpio1_isr1l     PMM1    0xEF600C30
gpio1_isr1h     PMM1    0xEF600C34
gpio1_isr2l     PMM1    0xEF600C38
gpio1_isr2h     PMM1    0xEF600C3C
gpio1_isr3l     PMM1    0xEF600C40
gpio1_isr3h     PMM1    0xEF600C44
;
 

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Brief note on breaking SX6012 56 Gbps ports down to 1 Gbps to connect slower gear: I bought four HP 655874-B21 (original Mellanox, just with a HP number) and four Cisco GLC-T V03 (version 03) 30-1410-03 somewhat cheap and these, plugged into each other, work on MLNX-OS 3.6.8012.

Code:
Port 1/7 state
        identifier             : QSFP to SFP adapter
        cable/module type      : Active optical cable
        ethernet speed and type: 1000BASE - T, 25G AOC, FEC based
        vendor                 : CISCO-AVAGO
        cable length           : 100m
        part number            : ABCU-5710RZ-CS4
        revision               : B2
        serial number          : DELETED...
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
@Stephan pretty awesome you're able to recover. Do you have a better idea why the conversion failed in that way?

I haven't had the time to do it on my 6012s which have been converted using @mpogr's guide, one to 1002 and another to 8012. I've read it's "very easy" to finish using the new guide after that but haven't looked into it.
 

andvalb

Member
Feb 15, 2021
27
25
13
Ulyanovsk, Russian Federation
Brief note on breaking SX6012 56 Gbps ports down to 1 Gbps to connect slower gear: I bought four HP 655874-B21 (original Mellanox, just with a HP number) and four Cisco GLC-T V03 (version 03) 30-1410-03 somewhat cheap and these, plugged into each other, work on MLNX-OS 3.6.8012.

Code:
Port 1/7 state
        identifier             : QSFP to SFP adapter
        cable/module type      : Active optical cable
        ethernet speed and type: 1000BASE - T, 25G AOC, FEC based
        vendor                 : CISCO-AVAGO
        cable length           : 100m
        part number            : ABCU-5710RZ-CS4
        revision               : B2
        serial number          : DELETED...
Also ONTi 10G RJ45 Copper SFP+ Module from Aliexpress works too.
 
  • Like
Reactions: Stephan

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
@klui Yeah happy myself it worked. Also very humbled by the generosity of several random complete strangers that helped me pull it off.

As for the post-mortem, I dumped the NOR in dead state, and while some things were still in there, the last 384 KB which should contain U-Boot were just blank 0xFF.

My working hypothesis is that manufacture.sh when used with most recent 3.6.8012 img file instead of 3.4.0012, together with me not using -v -v to see what is going on, together with me hitting Ctrl-C while the script was doing "nothing", together with using -B to update U-Boot, together with script authors causing a longish critical section of erasing the bootloader, then doing some wild bunch of sjit while the bootloader sits empty, instead of immediately writing a copy back into flash to minimize the chance of bricking, probably was the mix that blew up.

I checked bootstrap, CPU and board EEPROMs, no damage. SX chip also still had its EMC firmware in its own SPI EEPROMs. Haven't checked the Samsung NAND, because that gets redone during manufacture anyway.

PS: Bonus hardware pr0n. :)

bdi2000.jpg
 

Zombielinux

Member
Jun 14, 2019
71
21
8
I've just got my SX6036 in the mail, go to power it on, and nothing. No lights on anything.

I pull apart the power supply and start looking for missing magic smoke. Nothing immediately obvious except for L6 which appears to be missing/shorted.

Its got a P/N of 33574TLF-03 30/12. Given the nomenclature on the other large inductors and the lack of fruitful search results. I can only conclude this is some internal TDK Lambda part numbering scheme.

For good measure, I checked the onboard fuses, both tested good. Still no output from the Lambda board. Things to note, there is no "power on" signal to the TDK Lambda board, all the commanding/status comes from the small L shaped board I can only presume was made on contract.

This leads me to a followup question. On the back of the power supply, it notes 12V/25A and a TDK Lambda part number of YM-11-1845.

If that part number is put into everyone's favorite online marketplace, it interchanges to the following:

SX6015 Power supply
071-000-588
MSX60-PF
MSX60-PR
Lenovo 90Y3780

If that's the case, is there any other options for power supplies that I'm missing?

P.S. If anyone has internal pictures of their power supplies, that would go a long way to determine if the power supply is physically damaged or if TDK did something funky with their design.
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Thanks! That was fast.

Ok, ours look identical. L6 is some strange jumper wire in an inductor package. (Next to the three parallel blue square capacitors)

I can stop looking for a toroid in my lab lol.

But it does mean my power supply is extra dead and I’ll have to wait for a replacement or hotwire something in.
 

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Trying to wrap up the project... stuck with a few questions. I run two converted EMC SX6012 now with 3.6.8012. Only software modification is some fan taming added to /etc/rc.d/rc.local.
  1. What is the purpose of the "Host ID"? I set this to the 12 hex digits of mgmt0 ethernet MAC.
  2. Web interface is not showing information from within the power supplies, but only this:Capture.PNG
    I checked what power supplies are in there, they are CLP0212FPXX5Z01A open frame PSUs. Then I checked the pinout of HDR3 (see https://www.sager.com/_resources/pdfs/product/CLP0212.pdf) and it seems to me there is no I2C capability? So what the web interface shows is correct, there is just no information like input voltage etc. Is this perhaps only for bigger models with beefier hot-plug PSUs?
  3. CPU cooling of PPC could perform better at lower airflow. Original heatsink has ~57 mm distance between center of holes, height around 10mm. Is somebody running the switch with a suitable replacement?
 
  • Like
Reactions: andvalb