Likely going to start playing with fiber - what to look out for?

[TheUnnamedNewbie]

This summer I'm going to play around with upgrading to fiber for some networking. I want to do it so I have some experience with it - I have CAT6A cables through the house so (for now) it's not like I need an upgrade.

The questions I have are related to how to select my hardware. With Copper, it's quite easy. Pick a cable. Pick a connector. Put the two together, plug it in, and done.

But I'm very unfamiliar with fiber, so it's all kinda intimidating. Apart from the theory behind them (and thus knowing the difference between MM and SM) I have no practical experience.
Are all SFP/SFP+ modules compatible (given they are both the same speed and fiber mode, I understand combining a single mode and a multi mode won't work)? Is all fiber cable alike? How do I know what fiber I can use? Do they all have the same terminations, and can I just get any fiber and put it in my module?

These might seem stupid questions to someone who has been doing it for years now, but I don't really know where to start.

 

[Pete L.]

This summer I'm going to play around with upgrading to fiber for some networking. I want to do it so I have some experience with it - I have CAT6A cables through the house so (for now) it's not like I need an upgrade.

The questions I have are related to how to select my hardware. With Copper, it's quite easy. Pick a cable. Pick a connector. Put the two together, plug it in, and done.

But I'm very unfamiliar with fiber, so it's all kinda intimidating. Apart from the theory behind them (and thus knowing the difference between MM and SM) I have no practical experience.
Are all SFP/SFP+ modules compatible (given they are both the same speed and fiber mode, I understand combining a single mode and a multi mode won't work)? Is all fiber cable alike? How do I know what fiber I can use? Do they all have the same terminations, and can I just get any fiber and put it in my module?

These might seem stupid questions to someone who has been doing it for years now, but I don't really know where to start.

Unfortunately it isn't as simple as copper is but it isn't that complicated either once you understand the basics. There are several different types of connectors for fiber but you will more than likely be using LC Connectors they are the most commonly used these days.

With that in mind SFP / SFP+ Modules are NOT all a like, they have different types like Single Mode, Multi-Mode, Long Range and Short Range they also come in different speed like 100mb/1gb, 10gb and beyond.

Fiber is also not the same, just like with copper you have different types within the families. For example Multi-Mode Fiber comes in OM1 through OM4 you can read about them here.

The Difference Between OM Multimode Fibers - OM1, OM2, OM3 & OM4

You will want / need to have the same Type of SFP / SFP+ Module on each end of your connection HOWEVER some devices are "Vendor Locked" which means you won't be able to use the same exact module you will need to get one that is compatible with their card / switch / whatever device. An example is a Cisco Switch will want Cisco SFP Modules but some others might work, an Intel 10G Card will want an Intel SFP Module. These can get to be quite the pain in the butt as they are the same module speed wise say 10G but they might not be compatible.

You can also use DAC Cables which are SFP / SFP+ Modules with a Copper Cable between them. These are also vendor specific but seem to be more compatible for some odd reason. These will offer a quick / easy connection between devices within relatively short distances.

I would suggest checking out a company like Fiber Store, they are very, very reasonably priced, they offer compatible SFP Modules and they stand behind their products.

Oh and you can also get normal fiber or armored fiber, only you can decide which way you want to go. In my house I ran several Armored Fibers from my 2nd floor in to my basement which is where i have my Servers / Networking equipment. In the basement I have a Cisco 10G Switch and my Servers have Mellanox and Intel 10G Cards in them. The Mellanox Cards seem to be very flexible as to which vendors SFP / DAC Cables where the Cisco Switch and Intel Cards are much more picky.

Feel free to ask away any questions, we will be glad to help.

 

[TheUnnamedNewbie]

Unfortunately it isn't as simple as copper is but it isn't that complicated either once you understand the basics. There are several different types of connectors for fiber but you will more than likely be using LC Connectors they are the most commonly used these days.

With that in mind SFP / SFP+ Modules are NOT all a like, they have different types like Single Mode, Multi-Mode, Long Range and Short Range they also come in different speed like 100mb/1gb, 10gb and beyond.

Fiber is also not the same, just like with copper you have different types within the families. For example Multi-Mode Fiber comes in OM1 through OM4 you can read about them here.

The Difference Between OM Multimode Fibers - OM1, OM2, OM3 & OM4

You will want / need to have the same Type of SFP / SFP+ Module on each end of your connection HOWEVER some devices are "Vendor Locked" which means you won't be able to use the same exact module you will need to get one that is compatible with their card / switch / whatever device. An example is a Cisco Switch will want Cisco SFP Modules but some others might work, an Intel 10G Card will want an Intel SFP Module. These can get to be quite the pain in the butt as they are the same module speed wise say 10G but they might not be compatible.

You can also use DAC Cables which are SFP / SFP+ Modules with a Copper Cable between them. These are also vendor specific but seem to be more compatible for some odd reason. These will offer a quick / easy connection between devices within relatively short distances.

I would suggest checking out a company like Fiber Store, they are very, very reasonably priced, they offer compatible SFP Modules and they stand behind their products.

Oh and you can also get normal fiber or armored fiber, only you can decide which way you want to go. In my house I ran several Armored Fibers from my 2nd floor in to my basement which is where i have my Servers / Networking equipment. In the basement I have a Cisco 10G Switch and my Servers have Mellanox and Intel 10G Cards in them. The Mellanox Cards seem to be very flexible as to which vendors SFP / DAC Cables where the Cisco Switch and Intel Cards are much more picky.

Feel free to ask away any questions, we will be glad to help.

Thanks for the response. I didn't know there was so much locking of vendors between modules. I was looking at getting one of those Quanta switches, heard they pretty much support every brand of module. Good to know intel NICs are picky about their modules tho.

What about using different brands of modules but of the same type (Say, a cisco and another brand, both 10G, MM short range) on a single connection? Will that work, or do both ends of the fiber need exactly the same module?

And I've read somewhere that SFP/SFP+/.. only support one network speed. Is this fixed to the speed of the switch, or can I use say a 1G SFP+ module in a switch that supports 10G SFP+ modules?

I've also seen the DAC cables, and might be getting some of those to connect the servers in my rack to the switch, but I am likely to be using a few longer runs (10-15m minimum, maybe upwards of 35m, which from what I've seen doesn't really seem to exist in DAC cables) so I thought I might as well look into and learn to work with fiber.

Will read through that link later today.

 

[Pete L.]

Thanks for the response. I didn't know there was so much locking of vendors between modules. I was looking at getting one of those Quanta switches, heard they pretty much support every brand of module. Good to know intel NICs are picky about their modules tho.

What about using different brands of modules but of the same type (Say, a cisco and another brand, both 10G, MM short range) on a single connection? Will that work, or do both ends of the fiber need exactly the same module?

And I've read somewhere that SFP/SFP+/.. only support one network speed. Is this fixed to the speed of the switch, or can I use say a 1G SFP+ module in a switch that supports 10G SFP+ modules?

I've also seen the DAC cables, and might be getting some of those to connect the servers in my rack to the switch, but I am likely to be using a few longer runs (10-15m minimum, maybe upwards of 35m, which from what I've seen doesn't really seem to exist in DAC cables) so I thought I might as well look into and learn to work with fiber.

Will read through that link later today.

So lets say you have a Cisco 10G Switch and an Intel 10G Card and they both take SFP+ Modules, you can try using the same module (same vendor) of say a 10G SR (Short Range) SFP+ Module (Multi-Mode) it "May" work. The only question is about the vendor locking on the devices. The Ideal configuration according to the manufacturer (of course) would be a Cisco SFP+ in the Cisco and an Intel in the Intel. Of course different cards / switches allow different modules but only "Their" modules are supported will get you support if you have any issues.

a 10G Switch with SFP+ Slots can take 1G Modules as well as 1G Copper-SPF Modules they will need to go to a compatible connection on the other end of course. But they are backwards compatible. In Theory you can connect a 10G SFP+ Module to a 1G SFP and it should work, I've yet to see this actually work though so YMMV.

DAC Cables do introduce some latency probably nothing you will ever notice but it is there. I was using DAC Cables but ended up getting all the modules / fiber jumpers from Fiber Store to go all fiber.

Also worth mentioning is that Fiber is NOT Auto X-Over Detecting when it comes to Transmit / Receive. Some Fiber Jumpers are Straight through and some are crossed over you may have to "Flip" the transmit / receive. it is easy to tell with MM as the light is visible so you can just make sure you have light coming out of one side and going in to the other side of the module.

 

[TheUnnamedNewbie]

So lets say you have a Cisco 10G Switch and an Intel 10G Card and they both take SFP+ Modules, you can try using the same module (same vendor) of say a 10G SR (Short Range) SFP+ Module (Multi-Mode) it "May" work. The only question is about the vendor locking on the devices. The Ideal configuration according to the manufacturer (of course) would be a Cisco SFP+ in the Cisco and an Intel in the Intel. Of course different cards / switches allow different modules but only "Their" modules are supported will get you support if you have any issues.

a 10G Switch with SFP+ Slots can take 1G Modules as well as 1G Copper-SPF Modules they will need to go to a compatible connection on the other end of course. But they are backwards compatible. In Theory you can connect a 10G SFP+ Module to a 1G SFP and it should work, I've yet to see this actually work though so YMMV.

DAC Cables do introduce some latency probably nothing you will ever notice but it is there. I was using DAC Cables but ended up getting all the modules / fiber jumpers from Fiber Store to go all fiber.

Also worth mentioning is that Fiber is NOT Auto X-Over Detecting when it comes to Transmit / Receive. Some Fiber Jumpers are Straight through and some are crossed over you may have to "Flip" the transmit / receive. it is easy to tell with MM as the light is visible so you can just make sure you have light coming out of one side and going in to the other side of the module.

Great! I think I have enought to start ordering stuff this weekend. I just have one last question that I can't seem to find any Yes/no awnser too: If I get a pre-terminated cables like the one in the link below, will it fit any SFP+ 10G multimode SR module? Or do I have to pay attention to what kind of "connector" my SFP+ module has (next to ofcourse making sure thet the fiber is MM when using a MM module and viceversa ofcourse).
30M LC UPC to LC UPC 10G 50/125 OM4 Duplex Multimode Armored Fiber Optic Patch Cable - 3.0mm PVC Jacket | FS.COM

I'm quite confident it will work from what I've seen in "guides" and so on, but I would rather be sure before I buy hardware/fiber only to find out it's not compatible.

 

[Pete L.]

Great! I think I have enought to start ordering stuff this weekend. I just have one last question that I can't seem to find any Yes/no awnser too: If I get a pre-terminated cables like the one in the link below, will it fit any SFP+ 10G multimode SR module? Or do I have to pay attention to what kind of "connector" my SFP+ module has (next to ofcourse making sure thet the fiber is MM when using a MM module and viceversa ofcourse).
30M LC UPC to LC UPC 10G 50/125 OM4 Duplex Multimode Armored Fiber Optic Patch Cable - 3.0mm PVC Jacket | FS.COM

I'm quite confident it will work from what I've seen in "guides" and so on, but I would rather be sure before I buy hardware/fiber only to find out it's not compatible.

Yes it will work, SFP / SFP+ Modules use LC Connectors and all LC Connectors are the same.

 

[TheUnnamedNewbie]

Yes it will work, SFP / SFP+ Modules use LC Connectors and all LC Connectors are the same.

Thats exactly what I needed to know! Thanks a lot for the help, it made a big difference. I think I understand enough to start more research and playing around with this.

 

[ttabbal]

The vendor locking of modules is a douche move that I hate. There are only a few manufacturers of the actual modules, the only difference is if they have a vendor code on them. If they want to warn you about possible issues, fine, do so. It's probably BS, but whatever. At least on Linux with Intel cards, you can override the lock at the driver level. My Quanta LB4M doesn't seem to care, neither do my Mellanox ConnectX2 cards. Random SFP+ from ebay and a DAC work fine with both.

 

[TheUnnamedNewbie]

The vendor locking of modules is a douche move that I hate. There are only a few manufacturers of the actual modules, the only difference is if they have a vendor code on them. If they want to warn you about possible issues, fine, do so. It's probably BS, but whatever. At least on Linux with Intel cards, you can override the lock at the driver level. My Quanta LB4M doesn't seem to care, neither do my Mellanox ConnectX2 cards. Random SFP+ from ebay and a DAC work fine with both.

Seeing as you seem to have both Intel cards and Mellanox ConnectX2 cards: what are the differences? I've heard that with gigabit, the intel cards can do a decent amount of unloading and such, and will thus usually perform better (and have more driver options) as to say something by a "consumer" brand like d-link or whatnot. Does this apply to 10G SFP+ cards too? Do those Mellanox cards perform as well as say an intel x520 card?

 

[ttabbal]

I don't own any Intel 10G cards. From reading around the net, it sounds like Intel isn't as dominant in the 10G space. For 1G, there's almost no other option for a good quality card. For 10G, there are a few manufacturers that make good cards. Mellanox is one of them. As for performance, between 2 of them, direct or through my switch, I can get 9.85Gbit/sec in iperf without any tuning. That's good enough that I have no interest in fiddling with driver options. Running top in another window, I don't see any significant CPU use, so there seems to be some offloading built in.

You have to be a little careful with Mellanox cards, as they are often listed on ebay with just the OEM model numbers. Mine were HP branded and just had a generic HP model number in the description. After some looking around, I found that they are rebranded ConnectX2 cards. The problem is that they made a number of Infiniband cards that can't do Ethernet. Those are useless to me, as I have no interest in trying to deal with that. The cards with SFP+ ports seem to be fine though.

 

[TheUnnamedNewbie]

I don't own any Intel 10G cards. From reading around the net, it sounds like Intel isn't as dominant in the 10G space. For 1G, there's almost no other option for a good quality card. For 10G, there are a few manufacturers that make good cards. Mellanox is one of them. As for performance, between 2 of them, direct or through my switch, I can get 9.85Gbit/sec in iperf without any tuning. That's good enough that I have no interest in fiddling with driver options. Running top in another window, I don't see any significant CPU use, so there seems to be some offloading built in.

You have to be a little careful with Mellanox cards, as they are often listed on ebay with just the OEM model numbers. Mine were HP branded and just had a generic HP model number in the description. After some looking around, I found that they are rebranded ConnectX2 cards. The problem is that they made a number of Infiniband cards that can't do Ethernet. Those are useless to me, as I have no interest in trying to deal with that. The cards with SFP+ ports seem to be fine though.

Thats great to know. Melanox SFP+ cards in europe seem to go for about 40 euros a piece on ebay, some only in sets of 2 with a DAC cable thrown in. Seems like I'll start ordering tonight then. Thanks!

 

[Pete L.]

I will also second the cautionary note about OEM vs Retail Mellanox Cards also keep in mind that ConnectX-2 Cards work fine but they aren't upgrading the firmware anymore. ConnectX-3 Cards support a lot more of the newer features that offload a lot of the work from the CPU. I don't believe that OS makes a difference but you also want to make sure that you have the right Bus Connection in the system you are going to put the card in to get the best performance.

As for Intel vs Mellanox - I find that the Intel cards do seem to be a little better performance wise but they are picky as hell when it comes to the SFP Modules but much less picky when it comes to DAC Cables (go figure) and as mentioned you can probably get around it with a driver tweak but I just get my SFPs from FiberStore, I don't mind paying $16 for the "Right" ones =)

I have Mellanox ConnectX-3 Cards in 2 of my Servers and Intel Cards in my NAS and one other Server. They have all been stable but will say that they have all seemed to go crazy price wise over the last few months. I got my ConnectX-3 Cards for well under $100 and now they seem to be well over. I guess we aren't the only ones going 10G These days. Its been said that this year is the year of 10G.

 

[cheezehead]

The vendor lock-in is just a way for them to help profit margins. the same SFP+ module is sold to Cisco, Dell, HP, Mellanox, ect and each one has an EEPROM chip that locks it to a specific vendor. So for <Brand X> I can spend $1,500 on a "X branded" SFP+ module, or I can get a generic SFP+ module that's coded to work with <Brand X> for less than $100 (both new mind you).


Also worth mentioning is that when running optics, all modules run at specific wave-lengths and you'll need to match the wavelength on each end. A unique trait with fiber over copper is you can run multiple wavelengths on the same piece of glass to work around needing more pairs of fibers.

For DAC, I've only dealt with HP's implementation and there are different model DAC cables depending on which switching line the device connects to (ie provision vs comware). DAC cables can be picked up for less generally but personally I would just go with glass if I could.

 

[MiniKnight]

Cables, cable sizes and vendor-lock-in for lamps.

 

[Terry Kennedy]

The vendor lock-in is just a way for them to help profit margins. the same SFP+ module is sold to Cisco, Dell, HP, Mellanox, ect and each one has an EEPROM chip that locks it to a specific vendor. So for <Brand X> I can spend $1,500 on a "X branded" SFP+ module, or I can get a generic SFP+ module that's coded to work with <Brand X> for less than $100 (both new mind you).

Another possibility is to buy SFP+ modules that are field-recodable. I use Solid Optics (I had bad experiences with Fiberstore DWDM modules). There's an app that runs on my tablet and a USB-to-SFP/SFP+/XFP gizmo.

Also worth mentioning is that when running optics, all modules run at specific wave-lengths and you'll need to match the wavelength on each end. A unique trait with fiber over copper is you can run multiple wavelengths on the same piece of glass to work around needing more pairs of fibers.

Bench (actually floor) test of 2 40-channel DWDM muxes and 2 Catalyst 4500X switches:

Given the cost of xWDM SFP+ optics, most home users would be far better off running more fibers. The stuff in the picture is currently running on a 40km link that costs thousands of dollars / month, so DWDM provides a cost savings after a couple of months.

For situations where just doubling the fiber capacity is sufficient, BiDi SFP+ parts (bidirectional on a single fiber) will likely be more cost-effective, as long as the switch supports those parts.

 

[aero]

Are those DWDM wavelength transceivers in your switches? I don't see any transponders. What model muxes are those? Nice setup!

 

[Terry Kennedy]

Are those DWDM wavelength transceivers in your switches? I don't see any transponders.

Yup. DWDM SFP+.

What model muxes are those? Nice setup!

I'm not sure - Purchasing came up with them. I spec'd low-loss 40-channel DWDM w/ monitoring ports and these turned up. The test reports were all in Chinese (but the important stuff are the per-channel measurements, and I could read those).

 

[cheezehead]

Another possibility is to buy SFP+ modules that are field-recodable. I use Solid Optics (I had bad experiences with Fiberstore DWDM modules). There's an app that runs on my tablet and a USB-to-SFP/SFP+/XFP gizmo.

How much does the programming tool run? What kind of pricing are you getting on the short-range SFP+ modules?

 

[Terry Kennedy]

How much does the programming tool run? What kind of pricing are you getting on the short-range SFP+ modules?

I don't know, for 2 reasons: 1) I'm not the person who handles the purchasing, and 2) My company has a negotiated discount based on volume and some mutual technical assistance. I don't know why they don't provide pricing on their web page - some companies are just that way for some reason.

My guess is that they will charge a bit more than other places do, for commodity parts. Once you get into things like DWDM SFP+ where even Fiberstore wants $400+, they're probably more competitive.

 

[cheezehead]

We've only bought a couple SWDM SFP+'s but they were all ER's and with the HP coding were less than that I think. Ended up buying new SFP and SFP+ modules direct from China over the last few years...saves a ton.