Beware of EMC switches sold as Mellanox SX6XXX on eBay

[klui]

It is supposedly a JS28F128P33TF70, which is SLC (100K erase cycles). But during verbose bootup from 3.4.0012 it shows it as either Samsung or Micron, and the chip has a FRU sticker covering it.

EDIT: Samsung and Micron made SLC and MLC versions of the same device: 0xd3 so one would need to remove the sticker to confirm. The MLC has 10K erase cycle endurance.

EDIT2: It's too bad the Linux kernel for the Mellanox doesn't print the 3rd byte (cycle) of Read ID so I didn't have to peel back the FRU sticker. My switch with the Micron is newer, with components with late 2017 date codes vs early 2015.

The Micron is an "MT29F16G08AJADA(WP) IT D" (16 Gbit, industrial grade -40C - 85C).
The Samsung is a "K9WAG08U1D-SCB0" (4 Gbit, commercial grade 0C - 70C).

Both are SLC.

NAND device: Manufacturer ID: 0xec, Chip ID: 0xd3 (Samsung NAND 1GiB 3,3V 8-bit), 1024MiB, page size: 2048, OOB size: 64
2 NAND chips detected

NAND device: Manufacturer ID: 0x2c, Chip ID: 0xd3 (Micron NAND 1GiB 3,3V 8-bit), 1024MiB, page size: 2048, OOB size: 64
2 NAND chips detected

 

[Stephan]

Hard to wear out SLC NAND. If you look at the RTC battery, these are BR2032 and not CR2032. Shelf life alone is more like 10-20 years, due to super low self-discharge. Mellanox built a switch here that will last 30 years and more. Which is why I like them so much. Even trouble like a dry capacitor or shot Mosfet in the PSUs should be relatively easy to troubleshoot and fix.

 

[klui]

The reasons why I'm a fan of these 40G switches are:

• 40G is good-enough for homelabbing
• shallow enough for my wall mount rack
• frugal power consumption
• acceptable noise levels in stock-Mellanox form, and changable

I don 't understand why Mellanox's full width 40G switches are so deep, even for their "short depth" models. The pictures I've seen clearly shows a lot of wasted space between the switch board and power/fan/management section in the rear connected by an umbilical. My guess is the air guides narrow the air channel to force more air across the ASICs.

Anyway I am glad I looked at the part number for my 6012s. I guess the Samsungs will stay in my lab while the Microns will be in production.

The reason I was concerned about NAND wear was when I used the original guide from @mpogr on my first 6012 with the Samsung part there were lot of NAND errors after initial conversion which were "normal?". They still showed up when I set loglevel to 6 in U-Boot after properly converting it using @dodgy route's enhanced notes. Initial observation with the Microns don't have that problem.

I didn't know about the BR2032s, although Panasonic and Energizer only rates their batteries to 10 years.

 

[Stephan]

Ask perplexity.ai about "nand factory bad blocks" ;-)

If you look at page 4 of https://www.rayovac.com/wp-content/uploads/2021/08/R6099_OEMReferenceGuide.pdf you can see less than half a percent self discharge for BR2032 per year. Considering the switch would be on all the time, this should be the major drain component. And if you squint really hard at page 3 with assumed 1μA RTC current then a BR2032 will hold up for 20 years. I appreciate hardware that needs a coin cell swap for hardware maintenance every 20-30 years and maybe a new cap or fan in the same time frame, and that is it.

In case you have an IR camera see if cooling of your QSFP modules is sufficient. Some run hot and want to be cooled or spread out on the slots to not sit right next to each other.

 

[klui]

Interesting. I don't know how to read that chart on pg 3 and convert the drain to obtain final nominal voltage at the end of 10/20 years.

My 6012s have Panasonic CR1620s in them, not BR2032. My oldest switch has 2013 44th week date codes printed on one chip and its battery still has 2.88V. My recent ones were manufactured ~2017 47th week. That battery has 2.84V. Panasonic states their shelf life is 8 years. Its capacity is less than the 2032s--75 mAh vs 220 mAh for BR (225 mAh for CR). The continuous drain metric is definitely better on the BRs though: 0.03 mA vs 0.1/0.2 mAh for CR1620/CR2032.

It would be nice to replace the 1620 with a BR1632 but I don't think it would fit. It was a little chore to remove the 1620 from the holder.

 

[clcorbin]

Folks,

I am working on configuring a SX6036 to work with my current VLAN setup on my ICX 6610 switches. So far, it has been a little bit different than the ICX world, but nothing too crazy.

Until I've tried to find out how to move the default vlan to a different vlan. It defaults to VLAN 1 and I have found NOTHING that will let me move it to my "normal" default VLAN. Probably one of the poorer choices I made when setting up the ICX switches is to use VLAN 1 as a non default VLAN (dedicated for switch to router traffic).

Can this be moved on the SX6036 (latest firmware, in ethernet mode, etc.)?

Thanks!

Clint

 

[klui]

See Native VLAN configuration for Mellanox switches

 

[clcorbin]

See Native VLAN configuration for Mellanox switches

klui,

I've see that thread several times. I still don't see where it says how to change the default vlan from vlan 1 to, say vlan 999 for example.

Perhaps I am getting confused because the ICX 6610 has the command "default-vlan-id" which allows you to change the DEFAULT vlan to any vlan (i.e.: 999 for example). And, I've used that command and then (in what now appears to be a mistake...) I used VLAN 1 for other type of traffic.

I'm starting to get the feeling that these switches do NOT support moving default from VLAN 1 and I may have to reconfigure my vlans on the ICX switches to play with the SX6036.

 

[klui]

There doesn't appear to be a command in MLNX-OS. Typically I would just change the PVID on a port. That's what the article and the MLNX-OS CLI documentation shows.

 

[cev]

in the GUI you would set the port to hybrid, then 999 next the that. you also need to remember to allow those vlans (or all) on that port.

from that link before for cmd line.

[standalone: master] (config interface ethernet 1/1) # switchport mode hybrid


[standalone: master] (config interface ethernet 1/1) # switchport access vlan 3

 

[clcorbin]

Thanks folks! It looks like it is possible with the commands above, I just can't do it in one hit like with the "default-vlan-id" command.

I appreciate the help with understanding this!

 

[lllllmmm]

Optimized Linux and BIOS. Now pings at 0,03 ms over the switch with 48GB/s in iperf

The journey with CX3 and now the SX6 has been just a delight.
So happy with these enterprise oldies for pennys
Wanted to upgrade to 10GBe and now all 40GBe for a fracture of 10GBe Hardware.

Thanks to everyone on building the guides and providing your time and help!

Could you write more about these optimizations?

 

[Marc_]

Afternoon all, I've aquired a SX1710 that doesn't boot. Turns out the storage was bad (showing 4MB total). Is there a way to reload the firmware possibly by modifiy the conversion process to only install a new image? This is all still very new to me but I did manage to convert an old EMC with the help of the guide. Any info or help is much appreciated.

Edit: It is a Mellanox not an EMC

 

[klui]

What are the symptoms? The datasheet for the 1710 says its control plane is x86 not PowerPC like the 1012/6012, 1018/6018, 1036/6036. The "guide"'s images won't work on your switch.

 

[Marc_]

What are the symptoms? The datasheet for the 1710 says its control plane is x86 not PowerPC like the 1012/6012, 1018/6018, 1036/6036. The "guide"'s images won't work on your switch.

It didn't see the msata to boot from. I wonder if I could restore it using the same method as the SN series and load onie?

 

[NablaSquaredG]

It didn't see the msata to boot from. I wonder if I could restore it using the same method as the SN series and load onie?

The available installers are too new for your switch. The maximum supported version, even with x86 control plane, is 3.6.8012. Do not try to use anything newer. It will brick your switch.

You would have to find image-X86_64-3.6.8012.img and build your own installer (it's possible, but a bit of work with bash magic etc).

 

[Marc_]

The available installers are too new for your switch. The maximum supported version, even with x86 control plane, is 3.6.8012. Do not try to use anything newer. It will brick your switch.

You would have to find image-X86_64-3.6.8012.img and build your own installer (it's possible, but a bit of work with bash magic etc).

I do have the image for this I think :-onyx-X86_64-3.6.8012.img. I have tried with an onie recovery but it it cannot read the eeprom data and it's in the wrong format so that was a bust. Bash magic is black magic to me.

 

[NablaSquaredG]

I do have the image for this I think :-onyx-X86_64-3.6.8012.img. I have tried with an onie recovery but it it cannot read the eeprom data and it's in the wrong format so that was a bust. Bash magic is black magic to me.

ONIE EEPROM on Mellanox x86 switches is weird.

What the OS (e.g. ONYX) considers the FRU EEPROM (from the mod guides for the PowerPC ones), what once was a completely separate chip, is now part of the ONIE EEPROM and stored in the Vendor Extension, or rather multiple vendor extensions (0xFD, 6 fields with 36, 164, 36, 36, 36, 20 bytes)

Recovering a botched ONIE EEPROM is not easy. You essentially need to erase it completely and hope that you have a copy of a known good version and then restore it.

 

[Marc_]

Anybody happen to know the u-boot command line password for these? Got a sx6036 that won't boot and want to reflash it but it's on the later u-boot with the password enabled.

 

[cy384]

Anybody happen to know the u-boot command line password for these? Got a sx6036 that won't boot and want to reflash it but it's on the later u-boot with the password enabled.

I believe the hash is $1$yCoib8pn$vSaWSssw2k17iOJRIdmcw/ but I don't think anyone has cracked it. The only other option is to use a SOIC clip to overwrite the hash in the flash chip.