Beware of EMC switches sold as Mellanox SX6XXX on eBay
- Thread starter mpogr
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
The aim was to get unmanaged Mellanox switch sx6005 (12 ports) (following @mpogr). After an erroneous flashing, the boot process from the console is interrupted (status = red).
And why would you want an unmanaged switch if you have a managed one and a conversion guide?
the console should provide a status message if you're actually connected to the console port?
The status led alone will not help you in fixing this
the console should provide a status message if you're actually connected to the console port?
The status led alone will not help you in fixing this
You can easily run IB on a managed switch too, and if you dont run the manager its basically unmanaged too.
If you connect to the console port and reboot the switch you should see it booting. Then you can interrupt uboot and reflash.
I assume you have followed the conversion guide so this should be familiar?
If you connect to the console port and reboot the switch you should see it booting. Then you can interrupt uboot and reflash.
I assume you have followed the conversion guide so this should be familiar?
Have you tried it with TNSR yet? Says it can handle 100Gbps in userspace instead of kernel (DPDK build). TNSR for the Home Lab
It was super expensive when it first came out, but they have a free homelab version now: TNSR High-Performance Software Router
Free TNSR for home lab IMO is the only option which makes sense.
Otherwise if you fancy a bit more effort without integrated CLI and have some linux experience, one can successfuly build DPDK + VPP + FRR and achieve exactly the same for free on Ubuntu. TNSR is build on DPDK+VPP itself plus of course other value add items one should value from their use case / support perspective.
I was forced to do so due TNSR not detecting ConnectX-3 which is mlx4 driver but it shouldn't be an issue with mlx5.
Otherwise if you fancy a bit more effort without integrated CLI and have some linux experience, one can successfuly build DPDK + VPP + FRR and achieve exactly the same for free on Ubuntu. TNSR is build on DPDK+VPP itself plus of course other value add items one should value from their use case / support perspective.
I was forced to do so due TNSR not detecting ConnectX-3 which is mlx4 driver but it shouldn't be an issue with mlx5.
Well I guess it's a no from me, as well - I was planning to use it with my ConnectX-3s too. WTAF.
If you want to build your own DPDK router with X3 I can dig my build config files for it. It wasn't that bad after all. You can also have startup config for VPP so on reboot you get back to what it should be. Any changes made directly to vpp via vppctl or TNSR (also vppctl) are not permanent. I was a little bit puzzled with no save config or anything like that in vpp itself but it is what it is, this is what TNSR does better.
Interesting TNSR is based on CentOS and we know how Redhat approached the subject...
I provided my feedback to Netgate when they asked about TNSR about mlx4 being excluded and it was basically impossible to build the support back in on my own. Don't expect any change though, these cards are good value for home lab but otherwise industry old.
Interesting TNSR is based on CentOS and we know how Redhat approached the subject...
I provided my feedback to Netgate when they asked about TNSR about mlx4 being excluded and it was basically impossible to build the support back in on my own. Don't expect any change though, these cards are good value for home lab but otherwise industry old.
Yeah, that'd be great. I'm still a little confused about how the stuff fits together. I built dpdk a few times years ago, but then discovered there was nothing to really use it with at the time, given that basically all linux network software was using kernel based TCP routing - either that, or I was just very confused.
Have you seen there's a dpdk OpenvSwitch package? I wonder if that might be useful. I heard it has some overhead, but if you're using dpdk probably wouldn't matter, huh?
Code:
% apt show openvswitch-switch-dpdk
Package: openvswitch-switch-dpdk
Version: 2.15.0+ds1-2
Priority: optional
Section: net
Source: openvswitch
Maintainer: Debian OpenStack <team+openstack@tracker.debian.org>
Installed-Size: 3,203 kB
Depends: dpdk, openvswitch-switch (= 2.15.0+ds1-2), libc6 (>= 2.29), libcap-ng0 (>= 0.7.9), librte-eal21 (>= 20.11), librte-ethdev21 (>= 20.11), librte-mbuf21 (>= 20.11), librte-mempool21 (>= 20.11), librte-meter21 (>= 20.11), librte-vhost21 (>= 20.11), libssl1.1 (>= 1.1.0), libunbound8 (>= 1.8.0), openvswitch-common (>= 2.15.0+ds1)
Enhances: openvswitch-switch
Homepage: http://openvswitch.org/
Download-Size: 979 kB
APT-Sources: http://debian.osuosl.org/debian bullseye/main amd64 Packages
Description: DPDK enabled Open vSwitch switch implementation
Open vSwitch is a production quality, multilayer, software-based,
Ethernet virtual switch. It is designed to enable massive network
automation through programmatic extension, while still supporting
standard management interfaces and protocols (e.g. NetFlow, IPFIX,
sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed
to support distribution across multiple physical servers similar to
VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V.
.
openvswitch-switch provides the userspace components and utilities for
the Open vSwitch kernel-based switch.
.
DPDK is a set of libraries for fast packet processing. Applications run
in user-space and communicate directly with dedicated network interfaces.
.
This package provides a DPDK enabled implementation of the ovs-vswitchd
binary.But then, there's this, which is kernel based: XDP - IO Visor Project
Ever tried the xdp stuff? I don't really see it as often, but it sounds like it'd be more compatible with legacy network software, given it leverages the kernel...
Last edited:
True 
To maybe get back from off-top, I was unable to resolve ACLs on VLAN interfaces on SX6012. The only other option was to separate filtered and non-filtered VLANs and have SX6012 do L3 routing for non-filtered ones and have service VLAN to pfsense which is handling filtering and L3 routing of everything else. Compromise yet working quite well.
M
To maybe get back from off-top, I was unable to resolve ACLs on VLAN interfaces on SX6012. The only other option was to separate filtered and non-filtered VLANs and have SX6012 do L3 routing for non-filtered ones and have service VLAN to pfsense which is handling filtering and L3 routing of everything else. Compromise yet working quite well.
M
I stuck our thread hijack over here: https://forums.servethehome.com/ind...-g-xdp-vs-dpdk-what-do-you-use-and-why.32101/True
To maybe get back from off-top, I was unable to resolve ACLs on VLAN interfaces on SX6012. The only other option was to separate filtered and non-filtered VLANs and have SX6012 do L3 routing for non-filtered ones and have service VLAN to pfsense which is handling filtering and L3 routing of everything else. Compromise yet working quite well.
M
This thread has had many useful hints and information that has let me get my EMC OS SX6018 cross-flashed. I also have a native MLNX-OS SX6036 and have come up with a bit of a hybrid approach to dealing with these EMC OS units.
Instead of flashing mlnxbase into the first OS area, /dev/mtdblock6, TFTPing into the manufacturing OS from https://forums.servethehome.com/ind...anox-sx6xxx-on-ebay.10786/page-42#post-288220 has all the tools necessary to cross-flash the switch. You just need to chroot into the OS area to run
Instead of modifying customer_rootflop.sh I am applying a license key after messing around with
Using my guide from https://forums.servethehome.com/index.php?threads/mellanox-sx6036-36ports-qsfp-€150.24158/post-296976 you can get bash CLI access without the need to mess with passwd and shadow symlinks. Using this method you can still add, delete, and/or modify user accounts and passwords.
And finally, while not related to the cross-flash procedure, you can IB SM HA cluster with a native MLNX-OS unit if you review my details in https://forums.servethehome.com/index.php?threads/mellanox-sx6036-36ports-qsfp-€150.24158/post-296976.
Instead of flashing mlnxbase into the first OS area, /dev/mtdblock6, TFTPing into the manufacturing OS from https://forums.servethehome.com/ind...anox-sx6xxx-on-ebay.10786/page-42#post-288220 has all the tools necessary to cross-flash the switch. You just need to chroot into the OS area to run
mddbreq.Instead of modifying customer_rootflop.sh I am applying a license key after messing around with
ltrace and the dumplicense / genlicense binaries. This has the added benefit of being able to enable the fabric inspector without messing with mddbreq and also gets rid of the [web.WARNING]: util_ui_api_is_ib_eth_vpi_enabled: HW supports: IB ETH VPI profiles, but they are not enabled by HW VPD or license. log messages without the need to hexedit /opt/tms/lib/web/handlers/rh per https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-292322, at least in my experience.Using my guide from https://forums.servethehome.com/index.php?threads/mellanox-sx6036-36ports-qsfp-€150.24158/post-296976 you can get bash CLI access without the need to mess with passwd and shadow symlinks. Using this method you can still add, delete, and/or modify user accounts and passwords.
And finally, while not related to the cross-flash procedure, you can IB SM HA cluster with a native MLNX-OS unit if you review my details in https://forums.servethehome.com/index.php?threads/mellanox-sx6036-36ports-qsfp-€150.24158/post-296976.
Last edited: