Can I attach a 10GBase-LR to QSFP-40G-LR4 (CWDM)?

[ky56]

The 10GBase-LR is 1310nm and the QSFP-40G-LR4 CWDM channels are 1271, 1291, 1311 and 1331 nm. Are the 1310nm and 3rd channel 1311nm technically compatible?

I have a 10Gb and (will eventually have) a 40Gb switch both fitted with basic LR (Q)SFP+ transceivers and want to know if I can directly connect them with a SMF cable?

I’m looking at using the KAIAM XQX2502 QSFP+. Also what is the difference between the XQX2502, XQX2503 and other QSFP+? Are different brands also intercompatible?

 

[Prophes0r]

I think 1310 and 1311 are compatible.

Technically the distinction comes from the fiber specification, not the transceivers.
As long as you are using G652C/G652D and up it should be fine? It's been 20+ years since the change so all fiber you can actually get your hands on should work.

Also, this is just the center of the 20nm wide channels.
So this is really a "close enough" kinda situation.

NOTE: I just read up about CWDM/DWDM this weekend and I was blown away at the idea of a completely passive frequency-based fiber splitter.
I understand it at an intellectual level, of-course we can separate light by splitting it in a prism, but I still get amazed at the little micro/macro level intricacies of the physical world we live in. Cool stuff.

 

[voided]

Use QSA adapters.

 

[Prophes0r]

Use QSA adapters.

That has nothing to do with what they are asking.

OP wants to know if they can use 1 of the 4 channels on their 40G CWDM optic(which is 4x10g channels) with a single channel 10G optic of the same frequency. 1310nm.

The answer is yeah. Probably.

 

[voided]

He's saying he's getting a switch with QSFP 40g and wants to hook it up to another device with a 10g LR SFP+ based optic. How would he break the individual channels out? Fiber optic transceivers don't do autonegotiation so a 40g LR QSFP optic can't just modulate down to 10g. He would need to use a QSFP-4x10G-LR optic with an MTP/MPO cable that pigtails to 4x 10g LC duplex connections to make that work. Or, he can use a QSA sled inside of the QSFP switch and put a single 10g-LR in there to pair it with his other 10g LR optic.

By design, CWDM4 sends and receives on four different wavelengths simultaneously. These aren't channels in the sense that you can just "choose" to use only one. On the software side of the switch or router, there might be ways to configure the port to operate at a reduced rate, effectively "turning off" some lanes. This is highly dependent on the specific equipment and software you're using, and even then, it's not a standard feature.

It's basically contraindicated to do what you're implying. The QSFP-4x10G-LR optic will do the breakout in hardware.

 

[ky56]

He's saying he's getting a switch with QSFP 40g and wants to hook it up to another device with a 10g LR SFP+ based optic. How would he break the individual channels out? Fiber optic transceivers don't do autonegotiation so a 40g LR QSFP optic can't just modulate down to 10g. He would need to use a QSFP-4x10G-LR optic with an MTP/MPO cable that pigtails to 4x 10g LC duplex connections to make that work. Or, he can use a QSA sled inside of the QSFP switch and put a single 10g-LR in there to pair it with his other 10g LR optic.

By design, CWDM4 sends and receives on four different wavelengths simultaneously. These aren't channels in the sense that you can just "choose" to use only one. On the software side of the switch or router, there might be ways to configure the port to operate at a reduced rate, effectively "turning off" some lanes. This is highly dependent on the specific equipment and software you're using, and even then, it's not a standard feature.

It's basically contraindicated to do what you're implying. The QSFP-4x10G-LR optic will do the breakout in hardware.

Prophes0r had the right idea.

I don't need to breakout the QSFP CWDM channels as SFPs only need one. I will be using XQX2502 QSFP+ LR Lite transceivers using CWDM. Whatever the lite part means. They use lc-lc connectors. So that's 4 different wavelengths (1270nm, 1290nm, 1310nm, and 1330nm) over a single fiber.

Through the magic of google I found out that auto-channelisation is a thing. Which is separate from auto-negotiation. Link. Though i have no idea if a converted (using the STH thread) Mellanox SX6012 will support this.

Also regarding the the wavelengths, it's my understanding that the "prism" doing the CWDM is only for splitting and combining light in general. It's the individual rx photodiodes themselves that are wavelength specific. So in theory the QSFP switch would auto-channelize and the SFP being 1310nm would only use the third channel of the QSFP. The third channel of the QSFP being 1310nm.

This is all just theory at the moment anyway. I don't know if it's technically possible or what document/spec I should look up to try and find out. I have ordered the hardware anyway and will find out. This will be an interesting home-lab journey.

 

[Prophes0r]

By design, CWDM4 sends and receives on four different wavelengths simultaneously. These aren't channels in the sense that you can just "choose" to use only one.

That is EXACTLY what they actually do. =)

Think of it like those [40G QSFP+] to [4x 10G SFP+] breakout cables.
You go into the switch and configure the 4x channels into a single link, or you break it into 4x links.

CWDM4 is actually 4x 10G links, each in a frequency band, over a single fiber.

And this part blew my mind when I read up on it. No sarcasm here.

There are boxes that will mux and demux the light going down the fiber into different wavelengths PASSIVELY!

1 fiber with 4 colors goes in.
4 fibers with 1 color come out.

Or the other way around.

So...
If they can go into the switch and turn on the single channel they want, they should be able to hook up a single 10G optic on the other end.
It should just work.

It might even work if the 40G optic is trying to send all 4 channels.

----------

You can read up a bit on it in my first post.
This account is new enough that it takes several hours to get a message approved if I put a link in it, so I'll just mention it here.

cheers.

EDIT:
Using a QSA adapter is certainly the easy option.
But it also adds cost.

In my new setup, I'm only paying $3.50 shipped for a 40G CWDM4 optic.
Add another $4 for a 10G 1311nm optic.

If those 'just work' on their own, or by flipping a software switch, I save $15 on a QSA adapter.
(and I need less 10G optics too.)

 

[Prophes0r]

Whatever the lite part means.

It usually means they are lower power optics designed for 500m-2km instead of the 10km norm for an LR optic.

Also:
You should see if the optic you are using can actually disable the other 3 channels using commands on the switch.
It might be smart enough to just stop sending if it get's no light on those wavelengths, but maybe not.
If you can, it should lower the power usage/heat some.

 

[ky56]

That is EXACTLY what they actually do. =)

Think of it like those [40G QSFP+] to [4x 10G SFP+] breakout cables.
You go into the switch and configure the 4x channels into a single link, or you break it into 4x links.

CWDM4 is actually 4x 10G links, each in a frequency band, over a single fiber.

And this part blew my mind when I read up on it. No sarcasm here.

There are boxes that will mux and demux the light going down the fiber into different wavelengths PASSIVELY!

1 fiber with 4 colors goes in.
4 fibers with 1 color come out.

Or the other way around.

So...
If they can go into the switch and turn on the single channel they want, they should be able to hook up a single 10G optic on the other end.
It should just work.

It might even work if the 40G optic is trying to send all 4 channels.

----------

You can read up a bit on it in my first post.
This account is new enough that it takes several hours to get a message approved if I put a link in it, so I'll just mention it here.

cheers.

Yea I just realized how wrong I was about that. I mixed up a passive splitter and CWDM Mux. I was browsing CWDM boxes again and noticed they all have wavelength labels on each port. I also remembered that they use prisms which should have been the clue. Oh well. It's a shame that those CWDM modules are so expensive at the required wavelengths. I found one for 70AUD but it was for 15XXnm wavelengths.

Another interesting way of splitting links is to convert a QSFP port into 4 SFP ports. I have only seen one box that can do that on Aliexpress and it's way too expensive for what's inside. Supposedly there is no ASIC. You can just take each of the QSFP lanes and connect them to the SFP lanes with decoupling caps and it should just work. I suspect you'll need a microcontroller to setup the I2C management interface of each (Q)SFP. Might be an interesting custom PCB project.

 

[Prophes0r]

I also remembered that they use prisms...

They actually use fancy optic coatings with constructive/destructive interference.
Even cooler than prisms IMO.

Another interesting way of splitting links is to convert a QSFP port into 4 SFP ports. I have only seen one box that can do that on Aliexpress and it's way too expensive for what's inside. Supposedly there is no ASIC. You can just take each of the QSFP lanes and connect them to the SFP lanes with decoupling caps and it should just work. I suspect you'll need a microcontroller to setup the I2C management interface of each (Q)SFP. Might be an interesting custom PCB project.

If your stuff is close, you can get a "breakout" DAC cable.
They are usually $20-$30 for 1-3m.

If not...you still have 12 ports to work with.
If this works like I think it will (I have parts on order and a "new" SX6036 to play with) then you could 'waste' 8 of your 12 ports with these 10G connections without having to resort to breakouts.

You could also go with part of what @voided was saying and get an MPO(8) optic which breaks out to 4x SFP+ ones.
There are options ranging from...

• 1 part
  • 40G to 4x 10G Active Optic breakout
• 2 parts
  • 40G which breaks out to 4x LC connections
  • 4x 10G optics
• 3 parts
  • 40G MPO
  • MPO to 4x LC fiber
  • 4x 10G optics

There is a blurb on the FS community blog about it showing the different options.
(I usually link every reference I post about and it's killing me to not link things...)

 

[ky56]

It usually means they are lower power optics designed for 500m-2km instead of the 10km norm for an LR optic.

Also:
You should see if the optic you are using can actually disable the other 3 channels using commands on the switch.
It might be smart enough to just stop sending if it get's no light on those wavelengths, but maybe not.
If you can, it should lower the power usage/heat some.

Oops missed you second comment.

I had though similarly of that but I figured the "pings" it sends out to detect a new hotplug on the other channels would interfere. But outright disabling the other channels sounds interesting. They are essentially 4 SFPs merged into one. The SFF specs I think testing with real hardware is where it's at now so I'll have to wait and find out.

 

[ky56]

They actually use fancy optic coatings with constructive/destructive interference.
Even cooler than prisms IMO.


If your stuff is close, you can get a "breakout" DAC cable.
They are usually $20-$30 for 1-3m.

If not...you still have 12 ports to work with.
If this works like I think it will (I have parts on order and a "new" SX6036 to play with) then you could 'waste' 8 of your 12 ports with these 10G connections without having to resort to breakouts.

You could also go with part of what @voided was saying and get an MPO(8) optic which breaks out to 4x SFP+ ones.
There are options ranging from...

• 1 part
  • 40G to 4x 10G Active Optic breakout
• 2 parts
  • 40G which breaks out to 4x LC connections
  • 4x 10G optics
• 3 parts
  • 40G MPO
  • MPO to 4x LC fiber
  • 4x 10G optics

There is a blurb on the FS community blog about it showing the different options.
(I usually link every reference I post about and it's killing me to not link things...)

The idea was to have the SX6012 feed points around the house and both 40G and 10G equipment can just plug into it. All these other solutions are great or better for a rack situation though. Guess I'll need different plans for 40/10G connections. I think at this point I'm completely over complicating the solution though. Should the disabling channel idea not pan out I'll just have a separate SMF run for 40G and 10G and be done with it.

Did you use the STH guide for converting your Mellanox switch? How did it go and was it very difficult?

BTW the CWDM passive mux/demux stuff sounds like an interesting read.

 

[Prophes0r]

Did you use the STH guide for converting your Mellanox switch?

HA!
I literally got the switch in my door about 8 hours ago.
I checked it for major dings, plugged it in and let it sort itself out until the red lights went away, then made some food.

...and then lost power for 6 hours...

I haven't started it back up yet because it's late and I had to spin the whole network back up again.
(I have 90 minutes of battery power. I don't need to run services. Just enough to shut down gracefully at the moment.)

What conversion are you talking about? Link?

I was planning on spending tomorrow getting the SX6036 and a Brocade 6450 set up, licensed, and talking with each other.
I also need to actually start in with the network closet setup...
(I forgot that the Mellanox switch is 24" depth... I have to redesign my entire idea for the 20" deep closet... Probably vertical mount.)

 

[Jason Antes]

The idea was to have the SX6012 feed points around the house and both 40G and 10G equipment can just plug into it.

You can. Use option #3 from [B]Prophes0r[/B]. The break out box goes to LC which you can then run anywhere you want it to go within the limitations of the QSFP/SFP+. Did this with an Arista 40GbE QSFP switch to a 10GbE Mellanox SFP+ switch that was about 50 meters away in a secured "EMP" box within a datacenter while the Arista switch was in a side datacenter. Broke out the QSFP MPO with a little splitter box and ran LC all the way to the switch in the secured box. 40Gb will negotiate to 10Gb. There are specialty QSFP's out there that run on LC too so you can just run LC FC cable everywhere. To get 40GbE you just have to have 40GbE transceivers on both ends.

 

[klui]

Hmm, an interesting exercise. I tried it on my SX6012.

I don't have any LR 10G optics so I used a 40G-UNIV module using OM4 MMF. Both 40G and 10G transceivers are Arista. Linked between the SX6012 and Supermicro X24S. The UNIV module links at 40G on the 6012.

Splitting the 6012's port doesn't work.

Setting the 6012's port to 10G doesn't work.

Using a QSA works.

 

[ky56]

HA!
I literally got the switch in my door about 8 hours ago.
I checked it for major dings, plugged it in and let it sort itself out until the red lights went away, then made some food.

...and then lost power for 6 hours...

I haven't started it back up yet because it's late and I had to spin the whole network back up again.
(I have 90 minutes of battery power. I don't need to run services. Just enough to shut down gracefully at the moment.)

What conversion are you talking about? Link?

I was planning on spending tomorrow getting the SX6036 and a Brocade 6450 set up, licensed, and talking with each other.
I also need to actually start in with the network closet setup...
(I forgot that the Mellanox switch is 24" depth... I have to redesign my entire idea for the 20" deep closet... Probably vertical mount.)

It's a guide for converting Dell EMC SX6012 from just InfiniBand to a full 40/56GbE switch. But it applies to some other Mellanox switches as well.

https://forums.servethehome.com/index.php?threads/beware-of-emc-switches-sold-as-mellanox-sx6xxx-on-ebay.10786

https://ctvbs.com/2021/09/30/mellanox-switch-conversion-guide-v1-1/

Mellanox Switch Conversion Guide v1.1 | Winners Industrial (HK) Co. Ltd

web.archive.org

If you are unable to find the zip file specified in the guide I can make a link somewhere.

 

[Jason Antes]

In a home, not sure why you'd need LR optics in the first place unless you were given them. I suppose I should have read a little closer that they were SMF LR optics. The do make SMF breakout cables and cassettes though, but the switch and optics would need to support it for it to work.

 

[Prophes0r]

@Jason Antes The breakout cable is assuming the direct connection doesn't just work.
(I wasn't talking about a box)

I have similar hardware and 40G CWDM4 optics.
I ordered a few regular 10G 1311nm optics to see if I can get a direct link.

The whole question here has been whether or not CWDM4 actually acts like 4x10G channels like a breakout does.
And if it does, will it work with a single channel connection directly, without any conversions or breakouts.

             1271nm ━━━┓                  ┏━━━ 1271nm  ignored
[40G CWDM4]  1291nm ━┓ ┃                  ┃ ┏━ 1291nm  ignored
             1311nm ━╋━┻━━ Single Fiber ━━┻━╋━ 1311nm  [10G SFP+]
             1331nm ━┛                      ┗━ 1331nm  ignored

 

[Prophes0r]

In a home, not sure why you'd need LR optics in the first place unless you were given them...

The LR optics are for a few reasons.

1. Single mode fiber. If I am going to run fiber all over my house, I don't want to have to do it again in the next 15-20 years. Most of the REALLY high-speed stuff all needs SM fiber. Lots of datacenters also followed this idea so they can simply upgrade their 100G optics without new fiber. Which is what happens a lot, hence...
2. They tend to be the ones that are SUPER cheap. I got 20x 40G ones for $70 shipped.

 

[Prophes0r]

@ky56 Ahh ok.
I THINK I have a Mellanox switch not an EMC one.
I haven't actually connected anything to it to see what it is running. I may not need to do anything other than "License" it.

I'll need to check now actually...
It boots with MLNX-OS.
It's an OLD version though.
I'm having the stupid issue on Nvidia's site again too.
The first page I went to works fine. Now ever single page gives me "Access Denied".

Archive.org to the rescue... At least the Mellanox download links still work.