Help with connecting serial port on Supermicro MOBO (H11DSi) to Mellanox switch (SX6036)

[erock]

I need some guidance on connecting a Mellanox SX6036 switch to the serial port of a Supermicro H11Dsi. My SX6036 is used and did not come with a RJ45-DB9 serial console cable. I bought this one on eBay. However, I have not been able to access the switch.

OS: PopOS (Ubuntu 22.04)

I tried the command dmesg | grep ttyS, which showed two serial ports ttyS0 and ttyS1. I then tried to login to device /dev/ttyS0 using minicom but the switch was not accessed and the port shows up as inactive. I also check the bios to make sure the serial port option is enabled.

This my first go with managed switches and serial ports so I have a lot to learn. I am not sure if I have the wrong cable or lack understanding of the correct workflow (or both). Any guidance you can give to point me in the right direction would be appreciated.

 

[erock]

Update: I made it into the switch! I had to add my username to the tty and dialout groups:

sudo usermod -a -G tty <username>
sudo user mod -a -G dial out <username>

I also used putty which made things a bit easier to follow for noobs. We will see how far I get from this point.

 

[unphased]

Good luck! I found the hints scattered around this forum to be very helpful unlocking all of the desired functionality for me. After I fiddled around (swapped transceivers around) even my connectx4 card on Thunderbolt enclosure is working reliably to hotplug to my apple silicon macbook. I even discovered this CX4 card has a capacitor knocked off it. I also get a second CX4 card as backup.

A bit confused you mention some linux commands and then say you used putty which is windows (although I'm aware a linux version of putty exists... but why...)

I was definitely super stuck until I got my RJ45 to serial to usb cable! You got past the first step.

 

[klui]

What was the transceiver that allowed your CX4 to work after long disconnect/standby?

 

[unphased]

What was the transceiver that allowed your CX4 to work after long disconnect/standby?

I am using these: https://www.amazon.com/gp/product/B00U7BJAU8/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

There are two of them. I ordered them both at the same time. So after seeing the issue for a while both with point to point and to my SX6036 I swapped around the transceivers on a whim and so far so good, knock on wood.

I don't know if the capacitor on the CX4 card got knocked off by me or not. I feel like it could also be related.

SR4 transceivers seem preferable to LR4 ones due to being non-high power which is relevant for operation in this switch (it only allows up to 4 high power transceivers).

But SR4's are still more expensive for now.

Also getting good use out of my DACs however I did notice some large numbers of retries on some link directions so there are some issues I should still try to iron out...

 

[klui]

You're saying you have 2 of the same SKU but things worked when you swapped them between your switch and Mac? SRs (short range) require multi mode while LRs require single mode so the cable type would be different. I recall for short run you can interchange them (duplex patch cords) but I don't recall which (replace SM with MM or vice versa). You shouldn't be able to use a SM MPO in an SR transceiver or MM MPO into LR transceiver because SM MPO/transceivers are APC while the MM variety are UPC.

 

[erock]

Good luck! I found the hints scattered around this forum to be very helpful unlocking all of the desired functionality for me. After I fiddled around (swapped transceivers around) even my connectx4 card on Thunderbolt enclosure is working reliably to hotplug to my apple silicon macbook. I even discovered this CX4 card has a capacitor knocked off it. I also get a second CX4 card as backup.

A bit confused you mention some linux commands and then say you used putty which is windows (although I'm aware a linux version of putty exists... but why...)

I was definitely super stuck until I got my RJ45 to serial to usb cable! You got past the first step.

Putty on Ubuntu is easy to install, and the default inputs worked without modification (dev/ttyS0, baud rate = 9600 etc.) so the tool showed me how to make this work. I can get minicom and screen to work now but Putty just made sense to me.

I am playing with ltrace now and trying to understand what I am seeing. Let’s see if I can make sense of this.

 

[unphased]

You're saying you have 2 of the same SKU but things worked when you swapped them between your switch and Mac? SRs (short range) require multi mode while LRs require single mode so the cable type would be different. I recall for short run you can interchange them (duplex patch cords) but I don't recall which (replace SM with MM or vice versa). You shouldn't be able to use a SM MPO in an SR transceiver or MM MPO into LR transceiver because SM MPO/transceivers are APC while the MM variety are UPC.

Yeah and this is the fiber used with those: https://www.amazon.com/gp/product/B08Q327KBN/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Both products state OM3 so it should all be compatible and multimode.

Yes I'm aware that the LR4 (long range? version 4?) should be single mode. And that there exist LC fiber in both single and multi mode. So definitely not interchangeable there.

So far this MPO is the only actual fiber run i've run anywhere in my house. Aside from things being a bit fiddly at times it's all been really satisfying.

 

[klui]

The 4 means number of lanes used by the transceiver. See 100 Gigabit Ethernet - Wikipedia where it describes 100G, then 40G.

40GBase-LR4 means 40G, long range, 4 lanes [4 lambda (light wavelength), and 2 physical fibers]. According to the table, 40G and 100G LR4 specifies QSFP28 modules with LC connectors. In fact most single mode use LC connectors. Only multi mode high speed QSFP28 use a mixture of LC/MPO for short range.