US MELLANOX SX6036 - $200

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

solon

Member
Apr 1, 2021
61
5
8
Never mind that last, some a bit of duckduckgo with the original fan number FFB0412UHN eventually turned up that it's a product from Delta in Thailand. For future reference:

Red: +12V
Black: GND
Blue: RPM
Yellow: PWM

That allowed me to translate it to industry-standard 4 pin on the Noctua.
 

solon

Member
Apr 1, 2021
61
5
8
I had the same problem. I plugged the unit straight into a linux pc from the mgmt1 port and ran:

Code:
sudo tcpdump --interface eth1
Then I could see the IP address that was broadcasting packets was 10.22.13.3 - and I reset my ip address to be 1 up from that in the same class D and got into the web pages with admin/admin to update the firmware.
I've just finally sat down with this. I'm no expert on tcpdump, but I'm getting:

Code:
IP 0.0.0.0.bootpc >
It seems to be making mdns requests, probably looking for a network boot? Is anyone familiar with this behavious by any chance?
 

aboron

New Member
Jul 19, 2020
4
2
3
I've just finally sat down with this. I'm no expert on tcpdump, but I'm getting:

Code:
IP 0.0.0.0.bootpc >
It seems to be making mdns requests, probably looking for a network boot? Is anyone familiar with this behavious by any chance?
Have you set a temporary static ip on your laptop/pc (anything is fine)? If not, that may just be your own machine looking for an IP address. If this is the mellanox, then it is already looking for dhcp on it's own.

When tcpdump prints "bootpc" it just means port 68 - which is the client side of dhcp (which is how you can ask for a network boot server as well as ip address).
 

solon

Member
Apr 1, 2021
61
5
8
Thanks for the quick reply.

It was doing this both with and without an address set on my ethernet port that's directly connected to the mellanox. MAC address is also the Mellanox's.

Not fully understanding why it isn't receiving an adress when I connect it on my network. There must be some restriction to what acceptable dhcp settings are. I suppose I'll just have to get a serial cable to find out what's going on.
 

saratoga

New Member
Nov 5, 2021
8
5
3
I'm interested in getting a QSFP+ ethernet switch with at least 5 ports to replace my SFP+ unit, so I was looking at these:


Just to make sure I understand, I would need to update this with the HP firmware from the March 2012 to a newer build. Then I would need to get an ethernet license. Is that right? For basic testing and ethernet use how new of a firmware do I need to update to from HP?
 

bytebuilder

New Member
Mar 19, 2018
9
1
3
I'm interested in getting a QSFP+ ethernet switch with at least 5 ports to replace my SFP+ unit, so I was looking at these:


Just to make sure I understand, I would need to update this with the HP firmware from the March 2012 to a newer build. Then I would need to get an ethernet license. Is that right? For basic testing and ethernet use how new of a firmware do I need to update to from HP?
As a quick side note about this specific listing, I purchased one from him last week and it's in fine working order, as much as I can tell so far without testing all of the ports. Awesome seller too. I too found this thread and landed on getting an SX6036 since they well priced. From your question I believe that is the process. The firmware that came with mine is SX_3.3.5006 2014-05-20 12:19:44 ppc and based on this thread am expecting a bit of an upgrade grind. But that's part of the fun of home labbing :)
 

thetoad

Active Member
Feb 10, 2021
236
102
28
I think i started doing the grind, then just skipped to the end, as got tired of it and thing seemed to work fine. :)
 

saratoga

New Member
Nov 5, 2021
8
5
3
The cover on mine seems fairly stuck. Looking at the instructions I'm supposed to take out the screws (done), remove the power supplies and fan modules (done), and then pull it back (won't budge). I got a couple people over to pull on it and it seems really stuck. Is there some trick to opening them that I'm missing?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
You tried prying /twisting /turning with a screwdriver?
That usually works for me;)
 

RedX1

Active Member
Aug 11, 2017
132
144
43
Hello



I had the same difficulty when first using these switches.

I would make sure that you have not missed the screw that is sometimes hidden under a small round security sticker, placed on the top cover, above the power connector in the LH side.



I hope that helps.



Good luck.



RedX1
 

saratoga

New Member
Nov 5, 2021
8
5
3
Dumb question, but is there some way to route from the 10/40G side through the MGMT RJ45 interfaces? I've got a laser printer I want on the network and I am looking to eventually get rid of my old switch (currently serving only the printer).
 

TrumanHW

Active Member
Sep 16, 2018
253
34
28
This project looks pretty interesting (and difficult) ... Do you think (in retrospect) IB was worth the effort ..?

I'm pretty much a networking ignoramus ... but I was also a total clue re: FN / TN before I jumped in and thus, learned a ton!
Am I correct in thinking the IB upside is:

It's an ultra low-latency, proprietary protocol which is great...
Unless Macs use that network, in which case it'd be limited to IB over Ethernet..?

Furthermore, MacOS lacks support for ROCE nor RDMA.

Workarounds..?
- Use MacOS as a HyperVisor to run a VM which supports IB and has Mellanox drivers like Windows 11 (I think) or Linux.
- If a VM can't work, add an OS "on metal" on a small partition & save files to an ExFat formatted SSD (so both OS can R/W to the dev)..?
Another benefit to Linux (AFAIK) it allows modification of Thunderbolt 3's reserved bandwidth & thus might exceed the 22Gb MacOS limit.
 

solon

Member
Apr 1, 2021
61
5
8
Not sure whether you were reacting to my project, and this is a little bit of a late reply, but I'm happy with the little infiniband network I'm running. I'm only using it as access to my network drives at the moment, but I am running games off of m.2's in my server which deliver data at around 2Gb/sec, which is more than adequate and 20x as fast as the 1gbe can do. I have a iscsi drive that looks local to my windows VM which I have for gaming.

The main upside of infiniband is the rdma I think. CPU load on the iscsi provider is negligible, which is mainly really nice when you want to move storage away from the client machine.

I'm vaguely considering moving the a whole windows install to a iscsi setup, but that would only really start to be interesting if I wanted to boot the whole VM on multiple machines. A solution that's looking for a problem at the moment, other than that it'd scratch my "it can be done so it must be done" itch. I will probably do it that way when I move to new hardware. At the moment the limiting factor is that there isn't enough pcie space for sufficient m.2 storage in my server for all my needs if I want full bandwidth for each drive.

I'm finally getting around to trying to get my 40gbe set up. Also not terribly neccesary or anything it keeps me off the street.
 

solon

Member
Apr 1, 2021
61
5
8
So, unfortunately, despite specifically looking for and finding a SX6012 with no trace of EMC markings anywhere on the box, it appears that this does in fact have a EMC uboot image... guess I'm off to find that thread about switching from EMC to regular MLNX-OS.

Code:
* Chasis Type        : DINGO
* Number of Ports    : 12
* U-Boot Revision    :U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)
* Firmware Revision  : 9.9.1260
* INI file Revision  : 0x31010016
Strangely though, nothing I do seems to be making it possible to connect to the unit over SSH or making anything like a webinterface accessible.
 

unphased

Active Member
Jun 9, 2022
148
26
28
So... I'd love to have the smaller footprint and power usage of the smaller switches but if it is easier to configure the 36 port ones, they're only $50 more so I should just get one of those. It sounds like. I'm trying to evaluate all the 40GbE options available to me (looking to play around with RoCEv1 with all these CX3 cards I got) and SX6036/6015 etc. look to be the ticket right now.
 

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
Hi guys looks like NVIDIA stopped licensing this SX6036 Devices right? i have managed to upgrade the latest version MLXN_OS 3.6.8012 available on NVIDIA site.. is there any info available on how to swap infiniband to Ethernet mode? as i have 2 servers with 40gbps MCX354 ECAT nics.. running upgraded with latest NVIDIA firmware available and both set on ethernet mode.. i would love to get them working with this switch to get more nics and hook another servers in my LAN.
 

cw823

Active Member
Jan 14, 2014
414
189
43
Hi guys looks like NVIDIA stopped licensing this SX6036 Devices right? i have managed to upgrade the latest version MLXN_OS 3.6.8012 available on NVIDIA site.. is there any info available on how to swap infiniband to Ethernet mode? as i have 2 servers with 40gbps MCX354 ECAT nics.. running upgraded with latest NVIDIA firmware available and both set on ethernet mode.. i would love to get them working with this switch to get more nics and hook another servers in my LAN.
drop me a PM
 

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
Ok so on NVIDIA developers site i read out some information about The SX6036 requires Gateway license to run Ethernet. is this a physical device or just a key? cause i found someone selling it online fairly cheap.. aldo its charging me pretty much the same price on postal service.. so i am now wondering if its a file license key? or a physical usb device or something to attach to the switch?
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
ause i found someone selling it online fairly cheap..
don't support/engage these people.
You can generate the required key(s) (depending on what features you want) pretty easily yourself with a vm running a linux distro with ltrace and files extracted from a mlnx-os x86 firmware file.