AOC noob questions

[lunadesign]

I have experience with 10G SFP+ and 100G QSFP28 passive DAC cables but have absolutely no experience with optical cables.

I need to connect one ConnectX-5 Ethernet card to another (both have 100G QSFP28 ports) but I believe the systems are too far apart (approx 10 meters) for passive DAC cables.

I'm thinking of getting an AOC to handle the longer distance.

Questions:

1. Is there anything special I need to know / watch out for with AOCs besides making sure the ends are "coded" for the intended card manufacturer?
2. Is there any downside to ordering a longer cable (say 15m instead of 10m) than I need?
3. Any sources you recommend that can deliver to the US West Coast in a week or so?

Thanks in advance!

 

[DavidWJohnston]

You should be good, I've used various AOCs and transceivers with ConnectX-4s, QLogic Fastlinq NICs, and a Celestica DX010 Switch.

If you buy AOCs/QSFPs sold as Mellanox-compatible from a reputable supplier like fs.com I don't expect you to have any problems.

One thing to consider with 100G in general is the FEC mode, but with a short (not km), direct (no switch) run in your case I don't think it matters at all.

There is no downside to using a longer cable, except higher cost and needing to manage the slack.

Another option you could consider is CWDM4 transceivers. They connect with 2xLC SM. This would let you run any length of cheap duplex SM fiber you want, change it in the future, replace the run if it breaks, etc. I run the Kaiam XQX5170 100G-CWDM4-RT bought from eBay.

Supplier-wise, fs.com is usually a safe bet, though I'm not sure about shipping times. I've had good success buying from eBay, but YMMV on that one.

The one AOC I had a problem with compatibility-wise is this HPE-branded Finisar one:

Ethernet99: SFP EEPROM detected
        Application Advertisement: N/A
        Connector: No separable connector
        Encoding: 256B257B
        Extended Identifier: Power Class 4(3.5W max), CDR present in Rx Tx
        Extended RateSelect Compliance: QSFP+ Rate Select Version 1
        Identifier: QSFP28 or later
        Length Cable Assembly(m): 10
        Nominal Bit Rate(100Mbs): 255
        Specification compliance:
                10/40G Ethernet Compliance Code: 40GBASE-SR4
                Extended Specification compliance: 100G AOC (Active Optical Cable) or 25GAUI C2M AOC
        Vendor Date Code(YYYY-MM-DD Lot): 2017-09-09
        Vendor Name: FINISAR CORP
        Vendor OUI: 00-90-65
        Vendor PN: FCBR425QE1C10-HP
        Vendor Rev: A
        Vendor SN: 8C1736004Z

I couldn't get that to work at 100G... But it works perfectly on my HP-branded 4x25G quad NIC, so it's probably some kind of vendor lock-in. If you're buying new this won't be an issue for you.

Dave

 

[lunadesign]

Hi @DavidWJohnston --

Thank you VERY much for the detailed response. I'm going to give it a whirl!

BTW, is this stuff (optical cables) as fragile as I hear it is?

 

[DavidWJohnston]

Cool! It's fragile compared to copper cables, but it's really not that bad. They do tend to tangle really easily especially DACs; so avoid just grabbing one end and flinging it around to de-tangle it There is armored fiber, for tactical/outdoor use, probably overkill for your use. Corning has a type of ultra-flexible fiber you can tie in knots without breaking - But jamming it in a door would still do it. Cheap low-quality or old fiber is easier to break.

I've run fiber all over (I own a fusion splicer, purely for messing around, LOL) and the main issue I've had isn't the fiber itself - Its the ceramic connectors. Any little particle of dirt in there can cause the connectors to crack when you plug them together. (There's actually a lot of PSI between those tiny glass connectors) Those 1-click cleaners and low lint cloths + alcohol are super effective.

With AOCs/DACs, you don't have to deal with the glass connectors (the optical interface is sealed inside the transceiver assembly), but the transceiver ends are metal and somewhat heavy (especially the QSFP28s), if they fall on concrete/tile they can become bent and no longer plug in.

Also keep the bend radius down, if fiber is bent at a sharp angle the light can "leak" out and perform poorly, even if it doesn't break. You'd be surprised at how robust modern patch and AOC cables are though, it takes a lot to break them.

Dave