10G fiber noob questions

Discussion in 'Networking' started by lunadesign, Feb 24, 2018.

  1. lunadesign

    lunadesign Member

    Joined:
    Aug 7, 2013
    Messages:
    93
    Likes Received:
    7
    I'm relatively new to 10G Ethernet. I've got Intel X710-DA4 cards in three servers and have been directly connecting them to each other (no switch so far) with 3 meter passive SFP+ DAC cables and everything has been working great. (I started with an Intel DAC cable but then found the ones from 10Gtek work just as well.)

    I'd like to connect a workstation to two of these servers but it's a bit further away, about 8.5 meters from the closest server and 10.5 meters from the farthest server.

    My understanding is that 7 meters is far as passive DAC can go. So, I was initially thinking of going with 10 or 12 meter active DACs. However those are pricey enough that it seems like fiber might be worth looking at but I'm completely new to fiber and the amount of options is a bit overwhelming.

    Questions:

    1) What are the pro's/con's between active DACs and fiber at the distances I'm looking at (ie, cost, performance, power consumption, etc)?

    2) Does it make sense to (a) buy separate transceivers and cables or (b) buy active optical cables? It seems like (a) might be cheaper and give me more flexibility if I get an SFP+ switch or other non-Intel SFP+ device in the future that requires a different transceiver on one end (I'm assuming you can mix and match?).

    3) It seems there are a lot of different types of cables and connectors. However, from what I can tell, my distance is clearly in the SR range, those tend to be multi-mode fiber and since the Intel transceiver takes Duplex LC connectors, I'm guessing Duplex LC/LC patch cables are the way to go?

    4) OM3 or OM4? Or does it not matter at my distance?

    5) Are the FS.com and 10Gtek products as good as the name-brand components?

    Many thanks in advance!
     
    #1
  2. Lost-Benji

    Lost-Benji Member

    Joined:
    Jan 21, 2013
    Messages:
    368
    Likes Received:
    18
    Single Mode is my preference.

    If you can see the red light, it is MM 850 nm.
    If you can't see the light then it is likely SM 1310 nm.

    SC = Standard Cock
    LC = Little Cock

    I use Ubiquity gear, cheap and works. Many options in BiDi gear as well.
     
    #2
  3. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63
    1. The major difference is that with SFP+ and fiber, you have to worry about coding of the lamps (SFP+ modules) in general, while with DACs, most switches and adapters will allow them. One exception is HP, where you must have HP coded DACs, but most other manufacturers will allow most/all DACs. Generally DACs are still cheaper, but with lamps, if you need a different length in the future, you can just replace a cheap piece of fiber. Power should be about the same.

    2. As in 1, you definitely can mix and match. A lot of cards (Intel for example) can have the transceiver coding turned off in most OSes. Again this is a quagmire, but I've had good luck with Intel cards and non-intel lamps though you sometimes need to add a line to a config file. Most switches let you override transceiver coding, but again watch out for HP which generally does not. Edit - I was going off the X520 cards. For the X710, it seems a bit more involved. See here:

    Intel Ethernet Drivers and Utilities / Mailing Lists

    If this seems a bridge too far, I would recommend either getting DACs or intel coded lamps.

    3. You probably should go with duplex LC/LC OM3 for short distances, that going to be cheapest and easy to work with. OS2 single mode has some advantages (for example you can run 40Gbe over it) and I agree other things equal single mode is nicer than multi-mode, but the lamps tend to be more money, which probably isn't worth it at this stage for you.

    4. OM3 is fine.

    5. FS.com has solid products. I've used them extensively, as have others with no issues. For lamps however, you can get pretty good generics like finisar or avago for very cheap; I would go that route over FS.com for lamps unless you need specifically coded ones. I cannot speak to 10Gtek.
     
    #3
  4. kaih1984

    kaih1984 New Member

    Joined:
    May 8, 2017
    Messages:
    2
    Likes Received:
    0
    Would you mind elaborate a little bit on coding of lamps? I thought sfp+ DAC cable should just work fine with any sfp+ port on any switch (like RJ45) and apparently I was mistaken. I purchased D-Link DGS-1510-28X and DGS-1510-28P and two cheap "Lot Of 2 Mellanox Connectx-2 PCI-Epress x 8 10GBe Ethernet Network Server Adapter Interface Card MNPA19-XTR In Bulk Package" from amazon and want to use DAC cable to comnect them. What kind of DAC cable should I get, as the switch and the nic are from different vendors? I also noticed there are "generic compatible sfp+ dac cable" on fs.com and probably I should just buy them?
     
    #4
  5. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,721
    Likes Received:
    426
    There are magic numbers coded in any sfp/sfp+ connector. Some vendors only allow their magic numbers to be used, others just log a warning and the transceiver work without problems.

    For your mellanox cards and the d-link switch I would try cisco branded dac cables.
     
    #5
  6. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63

    Most vendors recognize that DAC cables are generally not coded differently on each end, so they allow any DAC cable to work. HP is a notable exception. Dual coded DAC cables do exist, but are generally not needed. See:

    Customized Compatibility 10G SFP+ Passive Direct Attach Copper Twinax Cable | FS.COM

    For SFP+ transceivers, many vendors such as Cisco, Arista, some HP, Intel etc allow you to override exclusions. Sometimes this may mean voiding warranty or getting a special code as in the case of Arista. In general, most lower tier players are not as stringent as the big boys, but I cannot comment on D-Link directly, having never used them. Cisco coded DACs generally do work pretty well, though I've had good luck with a variety of DACs. I'd probably try some cheap cisco DACs off Ebay. You can often get them under $10. If you want to be extra safe, you can just get a D-Link coded DAC either an original OEM or from a place like FS.com

    For cards, Mellanox are notorious for accepting just about anything. I've used many different DACs with Mellanox and never had an issue.

    Hope this helps you out. Good luck!
     
    #6
  7. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,721
    Likes Received:
    426
    On my 7050q-16-r (eos v4.17.0 and newer) it was enough to create an empty file to make third party transceivers work, no special code needed.
     
    #7
    PithyChats likes this.
  8. lunadesign

    lunadesign Member

    Joined:
    Aug 7, 2013
    Messages:
    93
    Likes Received:
    7
    When you say "coding of the lamps", the coding is actually in the SFP+ connectors, right? I'm assuming there isn't anything vendor-specific about the light signals or else it wouldn't be possible to say, connect one end to an Intel card and the other end to an HP switch.

    Also, any difference in performance? I'm curious if the conversion from electrical-to-light and then back to electrical again has a performance impact.

    If not FS.com, where do you recommend shopping for the generic lamps?

    Thanks for your helpful response!
     
    #8
  9. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63
    Interesting, was not aware of this. I'll try it on a 7050q-32, but I suspect it should be the same. Thanks!
     
    #9
  10. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63

    Yeah, basically it's just a code in firmware. Many manufactures make what is essentially the same product and just burn a different coding in. An example of this is Finisar.

    FTLX8571D3BCL is the generic SFP+ SR lamp,
    FTLX8571D3BCVIT1 is the intel SFP+ SR lamp.

    You will notice the model numbers are almost the same. That's because AFAIK they are the same hardware, just coded differently.

    For generics, I've used FS.com which I've never had a problem with, but I've also used avago and finisar to great success. They are often cheaper on Ebay than FS.com generics. I would only go FS.com if you want new lamps, or need specific coding.

    As for the lamps themselves, they follow the SPF+ spec, so should all perform basically the same. I've used lots of different lamps and have never found any performance differences. One small note is a lot of avago lamps have a plastic piece which I found more flimsy than the all metal finisar, but that's about it. Maybe someone else can chime in, but I've never had any issues related to any SFP+ connectors whether lamps or DACs other than coding issues, and those have mostly been with HP and intel.

    Here's a list of one's I've used (from memory so not necessarily exhaustive)

    Intel (finisar)
    Intel (unknown oem)
    Cisco
    HP
    Avago
    Arista
    Finisar - generic
    JDSU
    IBM(Avago)
    IBM (unknown oem)
    Nebula
    Monoprice

    I cannot say that I've noticed a functional difference between any of these. Most of these include both DACs and lamps.

    I hope this gives some assurance that as long as the coding works, you are probably ok with just about anything.

    There are some small differences with DACs vs lamps in latency and heat, but for a couple of ports, this is not worth worrying about.
     
    #10
    migolmusk likes this.
  11. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63

    Interesting. Which switch was this on? I haven't had much luck with generic SFP+'s and HP/HPE switches. I've even had people send me DACs and SFP+'s guaranteeing they work with HPE, they didn't. Maybe newer firmware has changed this somewhat? Admittedly even though I love HPE switches in general, I've moved away from them a bit largely due to the difficulty with DACs and SFP+'s
     
    #11
  12. aero

    aero Active Member

    Joined:
    Apr 27, 2016
    Messages:
    309
    Likes Received:
    51
    While generally not applicable to home use cases....
    One thing I always look at are the detailed specs of transceivers to understand their operational parameters. They are different between models.

    Receiver sensitivity
    Tx power
    Support for DOM
    Operating temperature range

    Other things worth mentioning...

    DACs generally lower latency than optical transceivers. (If nanoseconds matter to you)

    I've seen issues with performance of some DACs... Specifically cdw house brand, proline. They would "work", but constantly cause crc errors on ports. This happened enough that I stopped purchasing proline.
     
    #12
    PithyChats likes this.
  13. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63

    Perhaps I should have added this proviso. I totally agree. I've never dealt with proline, but for most major brands, finisar, avago, etc, they all have pretty solid specs. Having a handful of SFP+ modules in single switch is very different from having multiple switches worth in dozens of racks. Where small variance in power and sensitivity matters for a DC environment, most home users rarely run anything longer than 100' and probably won't ever note a difference. I'll keep on eye out for proline in the future. (Personally I've dealt with Cisco the most in professional settings and have seen virtually no issues)

    For latency, this is true, but likely trumped by switch latency, unless you have a pretty solid switch, which even used is going to be $$$.

    For example, based on these numbers:

    Copper is faster than fibre!

    you are looking at less at 8 ns latency difference at around 7m between fiber and DACs. Compared to the latency of say an Arista 7050QX which runs around 550ns, unless you are ultra sensitive to latency, it doesn't seem to really matter much, and you are getting at best around a 2% improvement in latency.
     
    #13
    Last edited: Feb 28, 2018
    gustav and migolmusk like this.
  14. PithyChats

    PithyChats Active Member

    Joined:
    Feb 3, 2015
    Messages:
    145
    Likes Received:
    63

    Ah, that makes more sense. I meant non-HP coded generics such as finisar. I've never had problems with 3rd party HP coded lamps. But they still tend to be a bit expensive...
     
    #14
    gustav likes this.
  15. nikey22

    nikey22 New Member

    Joined:
    Feb 19, 2018
    Messages:
    14
    Likes Received:
    0
    #15
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,254
    do eeeet
     
    #16
Similar Threads: fiber noob
Forum Title Date
Networking Looking for router advice before moving to AT&T fiber Jun 17, 2019
Networking New fiber run - parts compatability help please Jun 16, 2019
Networking Fiber vs. Copper For Short Lengths and Transceiver Reliability Mar 17, 2019
Networking 400ft new fiber run thoughts? Nov 1, 2018
Networking Has anyone else bought Fiber store Cat8 ethernet cable Sep 5, 2018

Share This Page