EU Mellanox SX6036 36ports QSFP €150

joddlarn

Member
Jul 14, 2018
43
15
8
Sweden
I don't think there is a general problem with the DACs from FS. I just got mine from a "bad batch" (they had green pullers on them, compared to the working ones that had black pullers). They said that they had a bad batch from the upstream supplier of the 40G DACs and that all batches after the one i got was working as intended.

Afaik there is no 56G optical modules at all, on any formfactor. Optical modules are one speed only, so you either have to go with 40G between the switches or use breakout cables and do 10G.
Breakout optics: FS for Mellanox MC2210511-PLR4 Compatible 4x10GBASE-LR QSFP+ 1310nm 10km MTP/MPO DOM Transceiver Module
You also need a breakout cable, such as this one: 5m (16ft) MTP-8 LC OS2 9/125 MTP Breakout Cable, Polarity B, OFNP
I suppose that in your case you rather go with 40G between the switches tho. The breakout solution is used to connect to switches that only has 10G interfaces.
 
  • Like
Reactions: kh78

kh78

New Member
Mar 31, 2020
27
6
3
If you got genuine ones then OS shouldnt be an issue, fw links above.

Have not used transceivers on mine (DAC only) but I believe MLX are not locked, so most should work fine.

The goto card for these would be a (branded) CX356, flashed to native MLX for ~$20 ea - various threads about these here too.
I believe they're Mellanox genuine ones, but time will tell with eBay

You mention the CX356, but I'm finding a lot less info and mentions of those than the MCX354A-FCBT, which seems like it should do the job. Any idea what the full model number of the CX356's is?

Sorry for all the questions, just on a steep learning curve having never played with more than 10G ethernet. I'm liking what I have been learning about Infiniband though, look forward to playing with it!
 

kh78

New Member
Mar 31, 2020
27
6
3
If its not urgent I can test a pair of 100G modules on my 6036 to see if they do 56 or 40 only
Not urgent, have a few days until I need to put in the order, so if you get the chance to test, that would be appreciated mate, thanks!
 

i386

Well-Known Member
Mar 18, 2016
2,047
542
113
31
Germany
What I know, there are no 56Gbe optical modules. If you want to use optical modules you have to go with 40Gbe. (if I'm wrong here, please correct me).
There are qsfp14 transceivers for fdr infiniband, not sure if they work with mellanox 56GBE
 

kh78

New Member
Mar 31, 2020
27
6
3
Well, my switches arrived, just enjoying their harmonics whilst bringing them up to 3.6.8010.

I'm going to have a look into the ethernet side of things....here's hoping i can work it out :)
 

Freebsd1976

Active Member
Feb 23, 2018
220
33
28
Well, my switches arrived, just enjoying their harmonics whilst bringing them up to 3.6.8010.

I'm going to have a look into the ethernet side of things....here's hoping i can work it out :)
the last version is 8012 , just search 8012 in the forum you will find it
 
  • Like
Reactions: kh78

kh78

New Member
Mar 31, 2020
27
6
3
I'm playing around with strings on some interesting files in the x86_64 .tar.gz from the lenovo site, but I'm clearly missing something...If anyone feels like PM'ing to help out with learning this, it would be appreciated!
 

kh78

New Member
Mar 31, 2020
27
6
3
Well, thanks to the help of a friendly forum member, I have learnt a bit and have the switches updated and operational with 56G ethernet and IB FDR capabilities.

On to hunting down some cables and HCA's to put these bad boys to work :)
 
  • Like
Reactions: Tyrant82

kh78

New Member
Mar 31, 2020
27
6
3
If its not urgent I can test a pair of 100G modules on my 6036 to see if they do 56 or 40 only
Any chance that you got around to testing this @Rand__ ?

I need to get the fibre side of things ordered pretty soon, so I'll just take a guess if you haven't had a chance to look :)
 

Rand__

Well-Known Member
Mar 6, 2014
4,590
912
113
Good that you remind me, but encountered an issue - one of the adapters complains that "High power transceiver is not supported." (actually one of two identical ones, so maybe its port depended). That's a known issue but have not had time to look into it, sorry

Edit: Mellanox Interconnect Community

Will try on port 1 & 3.
Dont have a MLX transceiver though, lets see if that works.
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,590
912
113
Ok, so the ports only link while being set to 40G, not 56 if being used with a Dell 100GBASE-SR (identified as Finisair).
 
  • Like
Reactions: kh78

kh78

New Member
Mar 31, 2020
27
6
3
Tested 5m fiber Mellanox 100G cable from EDR switch, unbelievable, but that 100G xceiver supported multirate of both 40/56/100G.
*********************

EDIT: Nevermind, as ever, just a case of RTFM:
https://www.mellanox.com/related-docs/prod_cables/PB_MFS1S00-HxxxE_200Gbps_QSFP56_AOC.pdf
200Gbps AOCs: " It contains four multi-mode fibers (MMF) optic transceivers per end; each operating at data rates of up to 50Gb/s."
56Gbps VPI AOC's: "MC220731V AOCs contain four multi-mode fibers (MMF) optical transceivers per end, each operating at data rates of up to 14.0625Gb/s."
Based on that, it's 4 core pairs of multi-mode, and I"m guessing probably OM3 or higher (being OM4).
The only other question is whether or not they tinker with the inbuilt transceivers to do different transmission powers or have better optics for improved receive sensitivities on their different length cables. I'm guessing it isn't worth their while to do so, and that for everything less than 100m they are using the same transceiver configuration. Perhaps if they got really excited then ~1-30m is the same and ~30+m are all another variant. I'm guessing 100+m will be single mode. I might buy a short 2m and a longer one if I can find it, and see what I can learn just buy splicing in extra cable in the middle (not keen enough to try dismantling the optics :eek:)

EDIT 2:
Yeah, longer cables are single mode ribbon fiber, 4 core pairs as well obviously:
56Gbps, VPI, 200m AOC:
https://www.mellanox.com/related-docs/prod_cables/PB_MC2207312-xxx_56Gbps_QSFP_AOC.pdf
"Standard single-mode ribbon fiber"

********************

Would anyone happen to know what type of fiber is used in the AOC cables such as these?
Since I have access to a fiber splicer, I might try to source some of the same type of fiber, pull that through and then just graft the original ends back on...could get a bit owned depending on the range of the optics in the cable though....

Other option is a buy a long original AOC cable, lop the end off, pull that through and then fuse the end back on. That's the most likely form to succeed. Annoying though as I have already pulled some single mode through the roof, and redoing that is not trivial as the metal roofing sheets have to come off...

I'm hoping someone will tell me that it's normal single mode fiber (OS2?) in the AOC cables...but I'm doubting it is :)
 
Last edited:

kh78

New Member
Mar 31, 2020
27
6
3
I'm looking into fan control options on these, my starting point being to try and create some pigtail intercept cables to monitor what sort of signalling is going on between the switch Mobo <> Fan Control Module PCB (Will see on this, lots of pins), and then the Fan Control Module PCB <> Fans. Similarly for the power supplies and their respective fan.

Step one was working out what the hell the connector was, after literally hours of searching (never knew how many 4 pin connectors there could be....), I've concluded that in my SX6036's, it uses Molex PicoBlades, being either 3 pin or 4 pin varieties:
PicoBlade Connector System - Molex
The 3 pin connector just goes to the indicator panel on the fan module for diagnostic light.

I'm going to hunt some down, make up some cables and then have a look see on the o'scope. Not expecting anything magic, presuming pretty standard freq PWM and 2 pulse per rev tacho on the fans, but just want to get some baseline numbers from the switch at a known ambient temp and idle load, before I start stuffing around with it all....I'm also allergic to cutting cables in things that otherwise working correctly....
 
Last edited: