If I read your post correctly, you were trying to do a 4x10 split with a 100GBASE-PSM4 or a 100GBASE-SR4 transceiver? It won't work that way. You need a compatible breakout cable. And you can read this Mellanox document for the cables P/N and configuration details:
Mellanox Switches - Tips & Tricks
- Thread starter NablaSquaredG
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Based on this document, it seems like it should work:
enterprise-support.nvidia.com
Maybe I should try to get a 40GbE optical transceiver and try the breakout? It’s odd as in the CLI and in Onyx it lists 10GbE (25, 10, and 1) as an available speed for each breakout.
EnterpriseSupport
Maybe I should try to get a 40GbE optical transceiver and try the breakout? It’s odd as in the CLI and in Onyx it lists 10GbE (25, 10, and 1) as an available speed for each breakout.
The transceivers themselves support breakout. I'm just not sure whether they support 10G link speed, it is possible that those transceivers can only do 25G link speed per lane.
Yeah, maybe that's the way to go. 40G-SR4 or 40G-PLR4 / 40G-PLRL4 are quite affordable.
For upto 100G Meallnox switch, only passive copper breakout cables for 2x50G, 4x25G or 4x10G have been listed as compatible by Nvidia/Mellanox, though there are other non-compatible AOC breakout cables available.Based on this document, it seems like it should work:
EnterpriseSupport
Maybe I should try to get a 40GbE optical transceiver and try the breakout? It’s odd as in the CLI and in Onyx it lists 10GbE (25, 10, and 1) as an available speed for each breakout.
Doesn't matter, I've used breakout transceiver on my own Mellanox switches (both SwitchX and Spectrum 1) a lot and I can assure that it works.
It would be quite interesting if it will actually work by adding a 40G transceiver to 100G switch port and then split.
Of course it will work... I am running this exact same setup with both 40G and 100G transceivers
I guess it dpends what cables and the attached transceivers you were using. Currently, I have a working 2x50G split setup in my SN2010 switch setup, but with a compatible passive copper cable. Had tried the non-compatiable AOC cable split (with pre-attached transceivers) before, and it didn't work. It complained about the incompatible transceiver (pre-attached) voltage with the switch
Last edited:
If you could give the exact transceiver P/N and cable you use, that will make the difference
I am running 100G -> 4x 25G with an AOC on SN2700. QSFP-100G-4SAO15
The only thing worth mentioning about this setup is that I had to set FEC to Reed Solomon on the other end as the other switch defaulted to Fire Code, while the spectrum switch uses RS by default.
My 40G-PLR4 adapter finally showed up and I was able to get it to link up at 10G. I bought the cheapest transceiver I could find online (Approved Networks JNP-QSFP-4X10GE-LR-A 40GBASE-PLR4 SMF 1310nm 10km Transceiver), and it worked. The only downside is that the instant I plugged it in, the fans in the switch spun up a lot higher - maybe 100%? Looking at the logs, it seems like the temperature readings weren't coming through from the transceiver.
I forced the fans to a lower speed by running:
fae mlxi2c set_fan /MGMT/FAN1 1 30
fae mlxi2c set_fan /MGMT/FAN2 1 30
fae mlxi2c set_fan /MGMT/FAN3 1 30
fae mlxi2c set_fan /MGMT/FAN4 1 30
Some brands of optical transceivers generate more heat than other even idle. In addition, the type of transceiver modules also matters. LR SMF 1310nm (10km) transceiver requires more power than SR MMF 850nm. In my opinion, it is a wasteful of power resource to use LR SMF 1310nm 10km Transceiver in a homelab setup or shorter distance. So, I prefer to use SR MMF 850nm in my setup.
I disagree with your statement, sure its not efficient but I would not say wasteful. There is 5? MMF versions and one SMF - you can standardize on one and use the same optics, cables and all working all the time. There is no need to worry. Yes I will not have to worry about shooting light 1km+ down the road but I do have light shooting 300m which gets into the maybe MMF maybe not.
My opinion about wasteful of power resource was from homelab setup perspective and not from DC deployment. I would prefer to have gears which are energy efficient. And I'm wondering how many of us actually need to have optical cable run more than 100m in a homlab setup?
I started with MMF. However, after I decided I wanted to run more fiber, I went with SMF because I could get 100G optics for $5 a piece. In addition, when I ran fiber from the basement to the attic, I could only get 1 MTP/MPO terminated cable and a cat 6A cable through the hole I drilled down from the attic. (And drilling that single run was a royal pain.) If I went with MMF, that would mean that I could only get 1 fiber trunk through. With SMF, I could run 1 24 fiber trunk assembly, go into a break out cassette and have more capacity than I could ever know what to do with. Given all of that, it just made sense to standardize on SMF and only use MMF for specific scenarios or when I already had the optics.
You can get MTP cables for MMF just like SMF, but the costs of the MMF cables start to grow astronomically after 3-5M. I needed to run 300M to my shed when I was building it back 4 years ago, SMF 12 MTP was ~1k armored and such, MMF would of been 3.5k for the exact same cable. Optics are cheaper, but the cables get expensive fast. So I moved to SMF everywhere and get optics used.