Looking for >=25G network equipment for homelab

[benni.zone]

Hello,

I was reading and searching here a few days. I first wanna say that this Forum impressed me. Very helpfull and friendly people. Second: I for sure wanna use the searchfunction. Sometimes I only need some hints or a special keyword to search for. So: I don't need the full solution.

Currently I have a very mixed up network (a hassivo 5 x 2.5G PoE + 2 x 10G SFP+ Switch. TrueNAS and my PC connected to 10G. A Zyxel 24 x 1G Switch. A D-Link 24 x 1G + 4 SFP (1G) Switch and a 8 x 1G Netgear (my first managed one). All of them are managed - couple of VLANs, LACP between the two 24 Port Switches. One Lenovo Tiny has a 2.5G NIC instead of the WIFI Adapter - works like a charm for pfSense. Another Tiny for Homeassistant and one for Jellyfin.
I have a couple of Lenovo Tinys and sprend them RAM, SSDs and spare NICs had laying around. Just to try out Proxmox with ceph. Needles to say performance is very limited. So I decided to upgrade my Network. I already ordered the PCIe 8x Riser for the Tinys and wanna create a Proxmox Cluster.

Looking for a 10G managed Switch with a few ports shocked me. I've seen QSFP+ same price or cheaper. Unfortunately I'm new to this. Without having practiice it is hard to understand.

What I wanna build up:
- 5 Node Proxmox Cluster
- 2 TrueNAS Server
- 2 Baremetal opnSense
- connect my PC as fast as Possible
My whish (with my current knowledge):
- An Mellanox SX6012, 6018 or 6036 but the are very rare or to expensive for meh (at least here in germany)
- Connect all of the above listed devices via QSFP+
- Connect my current switches via SFP and SFP+ uplinks for Accesspoints, TV, Nvidia Shield, ...

A few questions:
1. How hard is it to convert an EMC SX6012 to a Ethernet licenced managed SX 6012? This is the only managed Mellanox I found on ebay under 500€ (I'm in germany here) Using a serial console is not new to me... if needed.
2. I found another one offering a SX6036 on ebay and asked him for a product Number (currently waiting for it) - is it possible to get all the Firmware files (HP seems to be freely available) and the Ethernet licence (heard looking for ltrack and a special folder in firmware) ?
3. Would it theoretically be possible to use an Mellanox IS5030, install OpenSM on one or more Hosts and use the switch unmanaged - is this VLAN aware? Can these configured for e.g. QSFP+ to 4 x SFP+ to uplink to another Switch?
4. Are there similar and afordable (its just homelab and not bussines) alternatives to those switches? 25G or so?
5. Are I'm right that SFP+ is SFP compatible? So if I use an QSFP+ to 4 x SFP+ Cable to connect one to an 10G Switch and one (or wo with LACP) to an SFP Switch?

I would be happy with any advice. As mentioned above. I wanna do the work myself - a hint what to look/ search for woud be nice.

Greetings from germany

Benni

 

[amlai]

The easiest to follow instructions are written by dodgy route.

The most obvious alternative would be the CRS504-4XQ-IN and use some breakout cables. I have an SX6012 and a CRS504-4XQ-IN. As cost effective as Mikrotik is, I like the way the mlnx-os works much better than RouterOS.

I've been able to mix and match SFP28 with SFP+ and it seems to work in most cases. I haven't tried putting an SFP+ into SFP, but it should work.

 

[nexox]

As far as I am aware, unmanaged Infiniband switches don't do Ethernet at all, even with OpenSM, and IPoIB is not particularly fast. You may want to check the gigantic Brocade thread for a selection of 10/40G switches, 25G is still pretty new and expensive.

 

[benni.zone]

Thank you very much.
CRS504-4XQ-IN looks nice. but very expensive.

I found an SX6012 EMC on ebay for 280€
I also found an HPE branded SX6036 for 420€
I would prefer the HPE one because I read that those are stock MellanoxOS without the need of flashing an EMC device. Is this correct?

I also got an PM from amlai with helpfull hints.

Thank you all

 

[benni.zone]

Hello everyone, I wanted to let you know that I was able to get my hands on an HPE SX6036. I have now even been able to update it to the latest firmware. I have now received the Mellanox ConnecX 2 cards and flashed them to the latest version and set them to Ethernet mode.
Now I'm struggling with the license.
Did I understand correctly that I have to set up a PowerPC32 VM using QEMU to run the keygen there? I'm not that savvy with Linux. When I try to use QEMU using various instructions (some of which are very old), I always come across error messages such as gtk not available or usbdevice parameters no longer available. Before I bother and run in the wrong direction, I wanted to ask if the path is the right one. Somehow I'll get it done. The Internet is big and wise. :-D

 

[i386]

Did I understand correctly that I have to set up a PowerPC32 VM using QEMU to run the keygen there?

No.
There are x86 images of mlnx-os and they contain binaries that you can unzip to a ubuntu vm (I didn't try other distos) and run the magical stuff (ltrace) there.

 

[benni.zone]

Oh Thank you... a Ubuntu VM (or Bare metal) isn't a problem. I'll try it after work.