EXPIRED Mellanox SX6036 36ports QSFP €150
- Thread starter Rand__
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I don't think there is a general problem with the DACs from FS. I just got mine from a "bad batch" (they had green pullers on them, compared to the working ones that had black pullers). They said that they had a bad batch from the upstream supplier of the 40G DACs and that all batches after the one i got was working as intended.
Afaik there is no 56G optical modules at all, on any formfactor. Optical modules are one speed only, so you either have to go with 40G between the switches or use breakout cables and do 10G.
Breakout optics: FS for Mellanox MC2210511-PLR4 Compatible 4x10GBASE-LR QSFP+ 1310nm 10km MTP/MPO DOM Transceiver Module
You also need a breakout cable, such as this one: 5m (16ft) MTP-8 LC OS2 9/125 MTP Breakout Cable, Polarity B, OFNP
I suppose that in your case you rather go with 40G between the switches tho. The breakout solution is used to connect to switches that only has 10G interfaces.
Afaik there is no 56G optical modules at all, on any formfactor. Optical modules are one speed only, so you either have to go with 40G between the switches or use breakout cables and do 10G.
Breakout optics: FS for Mellanox MC2210511-PLR4 Compatible 4x10GBASE-LR QSFP+ 1310nm 10km MTP/MPO DOM Transceiver Module
You also need a breakout cable, such as this one: 5m (16ft) MTP-8 LC OS2 9/125 MTP Breakout Cable, Polarity B, OFNP
I suppose that in your case you rather go with 40G between the switches tho. The breakout solution is used to connect to switches that only has 10G interfaces.
True. But a bit of a pain to cable tho, as the modules are stuck to the cable
If that isn't a problem, here is a 50m version: FS for 50m Mellanox MC220731V-050 56G QSFP+ AOC Cable
I believe they're Mellanox genuine ones, but time will tell with eBay
You mention the CX356, but I'm finding a lot less info and mentions of those than the MCX354A-FCBT, which seems like it should do the job. Any idea what the full model number of the CX356's is?
Sorry for all the questions, just on a steep learning curve having never played with more than 10G ethernet. I'm liking what I have been learning about Infiniband though, look forward to playing with it!
If its not urgent I can test a pair of 100G modules on my 6036 to see if they do 56 or 40 only
Not urgent, have a few days until I need to put in the order, so if you get the chance to test, that would be appreciated mate, thanks!
This is a handy quick reference on the HCA's if anyone else is looking into it:
https://community.mellanox.com/s/article/mellanox-adapters---comparison-table
https://community.mellanox.com/s/article/mellanox-adapters---comparison-table
There are qsfp14 transceivers for fdr infiniband, not sure if they work with mellanox 56GBE
the last version is 8012 ， just search 8012 in the forum you will find itWell, my switches arrived, just enjoying their harmonics whilst bringing them up to 3.6.8010.
I'm going to have a look into the ethernet side of things....here's hoping i can work it out
Any chance that you got around to testing this @Rand__ ?
I need to get the fibre side of things ordered pretty soon, so I'll just take a guess if you haven't had a chance to look
Good that you remind me, but encountered an issue - one of the adapters complains that "High power transceiver is not supported." (actually one of two identical ones, so maybe its port depended). That's a known issue but have not had time to look into it, sorry
Edit: Mellanox Interconnect Community
Will try on port 1 & 3.
Dont have a MLX transceiver though, lets see if that works.
Edit: Mellanox Interconnect Community
Will try on port 1 & 3.
Dont have a MLX transceiver though, lets see if that works.
Last edited:
Tested 5m fiber Mellanox 100G cable from EDR switch, unbelievable, but that 100G xceiver supported multirate of both 40/56/100G.
https://www.mellanox.com/related-docs/prod_cables/PB_MFA1A00-Exxx_100Gbps_QSFP28_MMF_AOC.pdf
https://www.mellanox.com/related-docs/prod_cables/PB_MFA1A00-Exxx_100Gbps_QSFP28_MMF_AOC.pdf
Last edited:
*********************
EDIT: Nevermind, as ever, just a case of RTFM:
https://www.mellanox.com/related-docs/prod_cables/PB_MFS1S00-HxxxE_200Gbps_QSFP56_AOC.pdf
200Gbps AOCs: " It contains four multi-mode fibers (MMF) optic transceivers per end; each operating at data rates of up to 50Gb/s."
56Gbps VPI AOC's: "MC220731V AOCs contain four multi-mode fibers (MMF) optical transceivers per end, each operating at data rates of up to 14.0625Gb/s."
Based on that, it's 4 core pairs of multi-mode, and I"m guessing probably OM3 or higher (being OM4).
The only other question is whether or not they tinker with the inbuilt transceivers to do different transmission powers or have better optics for improved receive sensitivities on their different length cables. I'm guessing it isn't worth their while to do so, and that for everything less than 100m they are using the same transceiver configuration. Perhaps if they got really excited then ~1-30m is the same and ~30+m are all another variant. I'm guessing 100+m will be single mode. I might buy a short 2m and a longer one if I can find it, and see what I can learn just buy splicing in extra cable in the middle (not keen enough to try dismantling the optics
)EDIT 2:
Yeah, longer cables are single mode ribbon fiber, 4 core pairs as well obviously:
56Gbps, VPI, 200m AOC:
https://www.mellanox.com/related-docs/prod_cables/PB_MC2207312-xxx_56Gbps_QSFP_AOC.pdf
"Standard single-mode ribbon fiber"
********************
Would anyone happen to know what type of fiber is used in the AOC cables such as these?
Since I have access to a fiber splicer, I might try to source some of the same type of fiber, pull that through and then just graft the original ends back on...could get a bit owned depending on the range of the optics in the cable though....
Other option is a buy a long original AOC cable, lop the end off, pull that through and then fuse the end back on. That's the most likely form to succeed. Annoying though as I have already pulled some single mode through the roof, and redoing that is not trivial as the metal roofing sheets have to come off...
I'm hoping someone will tell me that it's normal single mode fiber (OS2?) in the AOC cables...but I'm doubting it is
Last edited:
I'm looking into fan control options on these, my starting point being to try and create some pigtail intercept cables to monitor what sort of signalling is going on between the switch Mobo <> Fan Control Module PCB (Will see on this, lots of pins), and then the Fan Control Module PCB <> Fans. Similarly for the power supplies and their respective fan.
Step one was working out what the hell the connector was, after literally hours of searching (never knew how many 4 pin connectors there could be....), I've concluded that in my SX6036's, it uses Molex PicoBlades, being either 3 pin or 4 pin varieties:
PicoBlade Connector System - Molex
The 3 pin connector just goes to the indicator panel on the fan module for diagnostic light.
I'm going to hunt some down, make up some cables and then have a look see on the o'scope. Not expecting anything magic, presuming pretty standard freq PWM and 2 pulse per rev tacho on the fans, but just want to get some baseline numbers from the switch at a known ambient temp and idle load, before I start stuffing around with it all....I'm also allergic to cutting cables in things that otherwise working correctly....
Step one was working out what the hell the connector was, after literally hours of searching (never knew how many 4 pin connectors there could be....), I've concluded that in my SX6036's, it uses Molex PicoBlades, being either 3 pin or 4 pin varieties:
PicoBlade Connector System - Molex
The 3 pin connector just goes to the indicator panel on the fan module for diagnostic light.
I'm going to hunt some down, make up some cables and then have a look see on the o'scope. Not expecting anything magic, presuming pretty standard freq PWM and 2 pulse per rev tacho on the fans, but just want to get some baseline numbers from the switch at a known ambient temp and idle load, before I start stuffing around with it all....I'm also allergic to cutting cables in things that otherwise working correctly....
Last edited: