US MELLANOX SX6036 - $200

[solon]

Never mind that last, some a bit of duckduckgo with the original fan number FFB0412UHN eventually turned up that it's a product from Delta in Thailand. For future reference:

Red: +12V
Black: GND
Blue: RPM
Yellow: PWM

That allowed me to translate it to industry-standard 4 pin on the Noctua.

 

[solon]

I had the same problem. I plugged the unit straight into a linux pc from the mgmt1 port and ran:

sudo tcpdump --interface eth1

Then I could see the IP address that was broadcasting packets was 10.22.13.3 - and I reset my ip address to be 1 up from that in the same class D and got into the web pages with admin/admin to update the firmware.

I've just finally sat down with this. I'm no expert on tcpdump, but I'm getting:

IP 0.0.0.0.bootpc >

It seems to be making mdns requests, probably looking for a network boot? Is anyone familiar with this behavious by any chance?

 

[aboron]

I've just finally sat down with this. I'm no expert on tcpdump, but I'm getting:

IP 0.0.0.0.bootpc >

It seems to be making mdns requests, probably looking for a network boot? Is anyone familiar with this behavious by any chance?

Have you set a temporary static ip on your laptop/pc (anything is fine)? If not, that may just be your own machine looking for an IP address. If this is the mellanox, then it is already looking for dhcp on it's own.

When tcpdump prints "bootpc" it just means port 68 - which is the client side of dhcp (which is how you can ask for a network boot server as well as ip address).

 

[solon]

Thanks for the quick reply.

It was doing this both with and without an address set on my ethernet port that's directly connected to the mellanox. MAC address is also the Mellanox's.

Not fully understanding why it isn't receiving an adress when I connect it on my network. There must be some restriction to what acceptable dhcp settings are. I suppose I'll just have to get a serial cable to find out what's going on.

 

[saratoga]

I'm interested in getting a QSFP+ ethernet switch with at least 5 ports to replace my SFP+ unit, so I was looking at these:

Mellanox SX6036 Model SXX0 36 Port Infiniband Switch Single P.S. | eBay

So, this picture is of an older unit that I no longer have - none of these currently for sale have 'rackmount ears'.

www.ebay.com

Just to make sure I understand, I would need to update this with the HP firmware from the March 2012 to a newer build. Then I would need to get an ethernet license. Is that right? For basic testing and ethernet use how new of a firmware do I need to update to from HP?

 

[bytebuilder]

I'm interested in getting a QSFP+ ethernet switch with at least 5 ports to replace my SFP+ unit, so I was looking at these:

Mellanox SX6036 Model SXX0 36 Port Infiniband Switch Single P.S. | eBay

So, this picture is of an older unit that I no longer have - none of these currently for sale have 'rackmount ears'.

www.ebay.com

Just to make sure I understand, I would need to update this with the HP firmware from the March 2012 to a newer build. Then I would need to get an ethernet license. Is that right? For basic testing and ethernet use how new of a firmware do I need to update to from HP?

As a quick side note about this specific listing, I purchased one from him last week and it's in fine working order, as much as I can tell so far without testing all of the ports. Awesome seller too. I too found this thread and landed on getting an SX6036 since they well priced. From your question I believe that is the process. The firmware that came with mine is SX_3.3.5006 2014-05-20 12:19:44 ppc and based on this thread am expecting a bit of an upgrade grind. But that's part of the fun of home labbing

 

[thetoad]

I think i started doing the grind, then just skipped to the end, as got tired of it and thing seemed to work fine.

 

[saratoga]

The cover on mine seems fairly stuck. Looking at the instructions I'm supposed to take out the screws (done), remove the power supplies and fan modules (done), and then pull it back (won't budge). I got a couple people over to pull on it and it seems really stuck. Is there some trick to opening them that I'm missing?

 

[Rand__]

You tried prying /twisting /turning with a screwdriver?
That usually works for me

 

[RedX1]

Hello



I had the same difficulty when first using these switches.

I would make sure that you have not missed the screw that is sometimes hidden under a small round security sticker, placed on the top cover, above the power connector in the LH side.



I hope that helps.



Good luck.



RedX1

 

[saratoga]

Dumb question, but is there some way to route from the 10/40G side through the MGMT RJ45 interfaces? I've got a laser printer I want on the network and I am looking to eventually get rid of my old switch (currently serving only the printer).

 

[TrumanHW]

This project looks pretty interesting (and difficult) ... Do you think (in retrospect) IB was worth the effort ..?

I'm pretty much a networking ignoramus ... but I was also a total clue re: FN / TN before I jumped in and thus, learned a ton!
Am I correct in thinking the IB upside is:

It's an ultra low-latency, proprietary protocol which is great...
Unless Macs use that network, in which case it'd be limited to IB over Ethernet..?

Furthermore, MacOS lacks support for ROCE nor RDMA.

Workarounds..?
- Use MacOS as a HyperVisor to run a VM which supports IB and has Mellanox drivers like Windows 11 (I think) or Linux.
- If a VM can't work, add an OS "on metal" on a small partition & save files to an ExFat formatted SSD (so both OS can R/W to the dev)..?
Another benefit to Linux (AFAIK) it allows modification of Thunderbolt 3's reserved bandwidth & thus might exceed the 22Gb MacOS limit.

 

[solon]

Not sure whether you were reacting to my project, and this is a little bit of a late reply, but I'm happy with the little infiniband network I'm running. I'm only using it as access to my network drives at the moment, but I am running games off of m.2's in my server which deliver data at around 2Gb/sec, which is more than adequate and 20x as fast as the 1gbe can do. I have a iscsi drive that looks local to my windows VM which I have for gaming.

The main upside of infiniband is the rdma I think. CPU load on the iscsi provider is negligible, which is mainly really nice when you want to move storage away from the client machine.

I'm vaguely considering moving the a whole windows install to a iscsi setup, but that would only really start to be interesting if I wanted to boot the whole VM on multiple machines. A solution that's looking for a problem at the moment, other than that it'd scratch my "it can be done so it must be done" itch. I will probably do it that way when I move to new hardware. At the moment the limiting factor is that there isn't enough pcie space for sufficient m.2 storage in my server for all my needs if I want full bandwidth for each drive.

I'm finally getting around to trying to get my 40gbe set up. Also not terribly neccesary or anything it keeps me off the street.

 

[i386]

It's an ultra low-latency, proprietary protocol which is great...

Ifiniband is an open standard by the ibta (infiniband trading association)
(Ib was first designed as the "succesor" of pci in the late 90s, some of the ideas later became pcie and now cxl with pcie 5.0, https://network.nvidia.com/pdf/whitepapers/PCI_3GIO_IB_WP_120.pdf)

 

[solon]

So, unfortunately, despite specifically looking for and finding a SX6012 with no trace of EMC markings anywhere on the box, it appears that this does in fact have a EMC uboot image... guess I'm off to find that thread about switching from EMC to regular MLNX-OS.

* Chasis Type        : DINGO
* Number of Ports    : 12
* U-Boot Revision    :U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)
* Firmware Revision  : 9.9.1260
* INI file Revision  : 0x31010016

Strangely though, nothing I do seems to be making it possible to connect to the unit over SSH or making anything like a webinterface accessible.