10G fiber noob questions

[lunadesign]

I'm relatively new to 10G Ethernet. I've got Intel X710-DA4 cards in three servers and have been directly connecting them to each other (no switch so far) with 3 meter passive SFP+ DAC cables and everything has been working great. (I started with an Intel DAC cable but then found the ones from 10Gtek work just as well.)

I'd like to connect a workstation to two of these servers but it's a bit further away, about 8.5 meters from the closest server and 10.5 meters from the farthest server.

My understanding is that 7 meters is far as passive DAC can go. So, I was initially thinking of going with 10 or 12 meter active DACs. However those are pricey enough that it seems like fiber might be worth looking at but I'm completely new to fiber and the amount of options is a bit overwhelming.

Questions:

1) What are the pro's/con's between active DACs and fiber at the distances I'm looking at (ie, cost, performance, power consumption, etc)?

2) Does it make sense to (a) buy separate transceivers and cables or (b) buy active optical cables? It seems like (a) might be cheaper and give me more flexibility if I get an SFP+ switch or other non-Intel SFP+ device in the future that requires a different transceiver on one end (I'm assuming you can mix and match?).

3) It seems there are a lot of different types of cables and connectors. However, from what I can tell, my distance is clearly in the SR range, those tend to be multi-mode fiber and since the Intel transceiver takes Duplex LC connectors, I'm guessing Duplex LC/LC patch cables are the way to go?

4) OM3 or OM4? Or does it not matter at my distance?

5) Are the FS.com and 10Gtek products as good as the name-brand components?

Many thanks in advance!

 

[Lost-Benji]

Single Mode is my preference.

If you can see the red light, it is MM 850 nm.
If you can't see the light then it is likely SM 1310 nm.

SC = Standard Cock
LC = Little Cock

I use Ubiquity gear, cheap and works. Many options in BiDi gear as well.

 

[PithyChats]

1. The major difference is that with SFP+ and fiber, you have to worry about coding of the lamps (SFP+ modules) in general, while with DACs, most switches and adapters will allow them. One exception is HP, where you must have HP coded DACs, but most other manufacturers will allow most/all DACs. Generally DACs are still cheaper, but with lamps, if you need a different length in the future, you can just replace a cheap piece of fiber. Power should be about the same.

2. As in 1, you definitely can mix and match. A lot of cards (Intel for example) can have the transceiver coding turned off in most OSes. Again this is a quagmire, but I've had good luck with Intel cards and non-intel lamps though you sometimes need to add a line to a config file. Most switches let you override transceiver coding, but again watch out for HP which generally does not. Edit - I was going off the X520 cards. For the X710, it seems a bit more involved. See here:

Intel Ethernet Drivers and Utilities / Mailing Lists

If this seems a bridge too far, I would recommend either getting DACs or intel coded lamps.

3. You probably should go with duplex LC/LC OM3 for short distances, that going to be cheapest and easy to work with. OS2 single mode has some advantages (for example you can run 40Gbe over it) and I agree other things equal single mode is nicer than multi-mode, but the lamps tend to be more money, which probably isn't worth it at this stage for you.

4. OM3 is fine.

5. FS.com has solid products. I've used them extensively, as have others with no issues. For lamps however, you can get pretty good generics like finisar or avago for very cheap; I would go that route over FS.com for lamps unless you need specifically coded ones. I cannot speak to 10Gtek.

 

[kaih1984]

1. The major difference is that with SFP+ and fiber, you have to worry about coding of the lamps (SFP+ modules) in general, while with DACs, most switches and adapters will allow them. One exception is HP, where you must have HP coded DACs, but most other manufacturers will allow most/all DACs. Generally DACs are still cheaper, but with lamps, if you need a different length in the future, you can just replace a cheap piece of fiber. Power should be about the same.

2. As in 1, you definitely can mix and match. A lot of cards (Intel for example) can have the transceiver coding turned off in most OSes. Again this is a quagmire, but I've had good luck with Intel cards and non-intel lamps though you sometimes need to add a line to a config file. Most switches let you override transceiver coding, but again watch out for HP which generally does not. Edit - I was going off the X520 cards. For the X710, it seems a bit more involved. See here:

Intel Ethernet Drivers and Utilities / Mailing Lists

If this seems a bridge too far, I would recommend either getting DACs or intel coded lamps.

3. You probably should go with duplex LC/LC OM3 for short distances, that going to be cheapest and easy to work with. OS2 single mode has some advantages (for example you can run 40Gbe over it) and I agree other things equal single mode is nicer than multi-mode, but the lamps tend to be more money, which probably isn't worth it at this stage for you.

4. OM3 is fine.

5. FS.com has solid products. I've used them extensively, as have others with no issues. For lamps however, you can get pretty good generics like finisar or avago for very cheap; I would go that route over FS.com for lamps unless you need specifically coded ones. I cannot speak to 10Gtek.

Would you mind elaborate a little bit on coding of lamps? I thought sfp+ DAC cable should just work fine with any sfp+ port on any switch (like RJ45) and apparently I was mistaken. I purchased D-Link DGS-1510-28X and DGS-1510-28P and two cheap "Lot Of 2 Mellanox Connectx-2 PCI-Epress x 8 10GBe Ethernet Network Server Adapter Interface Card MNPA19-XTR In Bulk Package" from amazon and want to use DAC cable to comnect them. What kind of DAC cable should I get, as the switch and the nic are from different vendors? I also noticed there are "generic compatible sfp+ dac cable" on fs.com and probably I should just buy them?

 

[i386]

There are magic numbers coded in any sfp/sfp+ connector. Some vendors only allow their magic numbers to be used, others just log a warning and the transceiver work without problems.

For your mellanox cards and the d-link switch I would try cisco branded dac cables.

 

[PithyChats]

Would you mind elaborate a little bit on coding of lamps? I thought sfp+ DAC cable should just work fine with any sfp+ port on any switch (like RJ45) and apparently I was mistaken. I purchased D-Link DGS-1510-28X and DGS-1510-28P and two cheap "Lot Of 2 Mellanox Connectx-2 PCI-Epress x 8 10GBe Ethernet Network Server Adapter Interface Card MNPA19-XTR In Bulk Package" from amazon and want to use DAC cable to comnect them. What kind of DAC cable should I get, as the switch and the nic are from different vendors? I also noticed there are "generic compatible sfp+ dac cable" on fs.com and probably I should just buy them?

Most vendors recognize that DAC cables are generally not coded differently on each end, so they allow any DAC cable to work. HP is a notable exception. Dual coded DAC cables do exist, but are generally not needed. See:

Customized Compatibility 10G SFP+ Passive Direct Attach Copper Twinax Cable | FS.COM

For SFP+ transceivers, many vendors such as Cisco, Arista, some HP, Intel etc allow you to override exclusions. Sometimes this may mean voiding warranty or getting a special code as in the case of Arista. In general, most lower tier players are not as stringent as the big boys, but I cannot comment on D-Link directly, having never used them. Cisco coded DACs generally do work pretty well, though I've had good luck with a variety of DACs. I'd probably try some cheap cisco DACs off Ebay. You can often get them under $10. If you want to be extra safe, you can just get a D-Link coded DAC either an original OEM or from a place like FS.com

For cards, Mellanox are notorious for accepting just about anything. I've used many different DACs with Mellanox and never had an issue.

Hope this helps you out. Good luck!

 

[i386]

getting a special code as in the case of Arista

On my 7050q-16-r (eos v4.17.0 and newer) it was enough to create an empty file to make third party transceivers work, no special code needed.

 

[lunadesign]

1. The major difference is that with SFP+ and fiber, you have to worry about coding of the lamps (SFP+ modules) in general, while with DACs, most switches and adapters will allow them. One exception is HP, where you must have HP coded DACs, but most other manufacturers will allow most/all DACs. Generally DACs are still cheaper, but with lamps, if you need a different length in the future, you can just replace a cheap piece of fiber. Power should be about the same

When you say "coding of the lamps", the coding is actually in the SFP+ connectors, right? I'm assuming there isn't anything vendor-specific about the light signals or else it wouldn't be possible to say, connect one end to an Intel card and the other end to an HP switch.

Also, any difference in performance? I'm curious if the conversion from electrical-to-light and then back to electrical again has a performance impact.

5. FS.com has solid products. I've used them extensively, as have others with no issues. For lamps however, you can get pretty good generics like finisar or avago for very cheap; I would go that route over FS.com for lamps unless you need specifically coded ones. I cannot speak to 10Gtek.

If not FS.com, where do you recommend shopping for the generic lamps?

Thanks for your helpful response!

 

[PithyChats]

On my 7050q-16-r (eos v4.17.0 and newer) it was enough to create an empty file to make third party transceivers work, no special code needed.

Interesting, was not aware of this. I'll try it on a 7050q-32, but I suspect it should be the same. Thanks!

 

[PithyChats]

When you say "coding of the lamps", the coding is actually in the SFP+ connectors, right? I'm assuming there isn't anything vendor-specific about the light signals or else it wouldn't be possible to say, connect one end to an Intel card and the other end to an HP switch.

Also, any difference in performance? I'm curious if the conversion from electrical-to-light and then back to electrical again has a performance impact.


If not FS.com, where do you recommend shopping for the generic lamps?

Thanks for your helpful response!

Yeah, basically it's just a code in firmware. Many manufactures make what is essentially the same product and just burn a different coding in. An example of this is Finisar.

FTLX8571D3BCL is the generic SFP+ SR lamp,
FTLX8571D3BCVIT1 is the intel SFP+ SR lamp.

You will notice the model numbers are almost the same. That's because AFAIK they are the same hardware, just coded differently.

For generics, I've used FS.com which I've never had a problem with, but I've also used avago and finisar to great success. They are often cheaper on Ebay than FS.com generics. I would only go FS.com if you want new lamps, or need specific coding.

As for the lamps themselves, they follow the SPF+ spec, so should all perform basically the same. I've used lots of different lamps and have never found any performance differences. One small note is a lot of avago lamps have a plastic piece which I found more flimsy than the all metal finisar, but that's about it. Maybe someone else can chime in, but I've never had any issues related to any SFP+ connectors whether lamps or DACs other than coding issues, and those have mostly been with HP and intel.

Here's a list of one's I've used (from memory so not necessarily exhaustive)

Intel (finisar)
Intel (unknown oem)
Cisco
HP
Avago
Arista
Finisar - generic
JDSU
IBM(Avago)
IBM (unknown oem)
Nebula
Monoprice

I cannot say that I've noticed a functional difference between any of these. Most of these include both DACs and lamps.

I hope this gives some assurance that as long as the coding works, you are probably ok with just about anything.

There are some small differences with DACs vs lamps in latency and heat, but for a couple of ports, this is not worth worrying about.

 

[PithyChats]

I used to think HPE would kick out "non-HPE" SFPs but after I tried some from FS.com I realized it's not. BTW, I used the 850nm SR version, with OM3 LC duplex jumper.

Interesting. Which switch was this on? I haven't had much luck with generic SFP+'s and HP/HPE switches. I've even had people send me DACs and SFP+'s guaranteeing they work with HPE, they didn't. Maybe newer firmware has changed this somewhat? Admittedly even though I love HPE switches in general, I've moved away from them a bit largely due to the difficulty with DACs and SFP+'s

 

[aero]

Yeah, basically it's just a code in firmware. Many manufactures make what is essentially the same product and just burn a different coding in. An example of this is Finisar.

FTLX8571D3BCL is the generic SFP+ SR lamp,
FTLX8571D3BCVIT1 is the intel SFP+ SR lamp.

You will notice the model numbers are almost the same. That's because AFAIK they are the same hardware, just coded differently.

For generics, I've used FS.com which I've never had a problem with, but I've also used avago and finisar to great success. They are often cheaper on Ebay than FS.com generics. I would only go FS.com if you want new lamps, or need specific coding.

As for the lamps themselves, they follow the SPF+ spec, so should all perform basically the same. I've used lots of different lamps and have never found any performance differences. One small note is a lot of avago lamps have a plastic piece which I found more flimsy than the all metal finisar, but that's about it. Maybe someone else can chime in, but I've never had any issues related to any SFP+ connectors whether lamps or DACs other than coding issues, and those have mostly been with HP and intel.

Here's a list of one's I've used (from memory so not necessarily exhaustive)

Intel (finisar)
Intel (unknown oem)
Cisco
HP
Avago
Arista
Finisar - generic
JDSU
IBM(Avago)
IBM (unknown oem)
Nebula
Monoprice

I cannot say that I've noticed a functional difference between any of these. Most of these include both DACs and lamps.

I hope this gives some assurance that as long as the coding works, you are probably ok with just about anything.

There are some small differences with DACs vs lamps in latency and heat, but for a couple of ports, this is not worth worrying about.

While generally not applicable to home use cases....
One thing I always look at are the detailed specs of transceivers to understand their operational parameters. They are different between models.

Receiver sensitivity
Tx power
Support for DOM
Operating temperature range

Other things worth mentioning...

DACs generally lower latency than optical transceivers. (If nanoseconds matter to you)

I've seen issues with performance of some DACs... Specifically cdw house brand, proline. They would "work", but constantly cause crc errors on ports. This happened enough that I stopped purchasing proline.

 

[PithyChats]

While generally not applicable to home use cases....
One thing I always look at are the detailed specs of transceivers to understand their operational parameters. They are different between models.

Receiver sensitivity
Tx power
Support for DOM
Operating temperature range

Other things worth mentioning...

DACs generally lower latency than optical transceivers. (If nanoseconds matter to you)

I've seen issues with performance of some DACs... Specifically cdw house brand, proline. They would "work", but constantly cause crc errors on ports. This happened enough that I stopped purchasing proline.

Perhaps I should have added this proviso. I totally agree. I've never dealt with proline, but for most major brands, finisar, avago, etc, they all have pretty solid specs. Having a handful of SFP+ modules in single switch is very different from having multiple switches worth in dozens of racks. Where small variance in power and sensitivity matters for a DC environment, most home users rarely run anything longer than 100' and probably won't ever note a difference. I'll keep on eye out for proline in the future. (Personally I've dealt with Cisco the most in professional settings and have seen virtually no issues)

For latency, this is true, but likely trumped by switch latency, unless you have a pretty solid switch, which even used is going to be $$$.

For example, based on these numbers:

Copper is faster than fibre!

you are looking at less at 8 ns latency difference at around 7m between fiber and DACs. Compared to the latency of say an Arista 7050QX which runs around 550ns, unless you are ultra sensitive to latency, it doesn't seem to really matter much, and you are getting at best around a 2% improvement in latency.

 

[PithyChats]

I used this SFP HPE J4858C 1000BASE-SX SFP (mini-GBIC) Transceiver | FS.COM (SX not SR, my mistake) in HPE 1820-24G switch. The switch recognized the SFP without issue.

Ah, that makes more sense. I meant non-HP coded generics such as finisar. I've never had problems with 3rd party HP coded lamps. But they still tend to be a bit expensive...

 

[nikey22]

I'm using LC to LC OM3 Fiber Patch Cable Multimode Duplex - 10m, with Finisar FTLX8571D3BCL transeivers (850nm), with 2 Mellanox ConnectX-2 single-Port MNPA19-XTR in a peer-to-peer.

I haven't chosen a good switch yet (thinking about the Quanta)

 

[fohdeesha]

I haven't chosen a good switch yet (thinking about the Quanta)

do eeeet