This is one that you are going to see on the main site over the rest of 2021. It is consuming a ton of time to plan and just get materials for. My goal is to go build something different and big so that way I can help give STH readers some inspiration on these items.
The project: Running (a lot of) fiber
Fiber Being Run
This is the fiber we have available as of now.
All of this is MTP-12 and will be used for in-wall installation. A quick breakdown:
The purpose of this build-out is to one get fiber to the two offices, the studio, and the server room. The other purpose, and this is quite important, is that I wanted to be able to do room-to-room direct networking without putting high-speed switches in the room. For example, in the YouTube studio, I did not want a 1U 100GbE switch. If I have something set up in my office, I wanted to be able to get a direct connection to another room easily. Sometimes we get new NICs, like a 200GbE NIC where we get two of them. Right now, I need both endpoints to be located next to one another. Scaling to four becomes a bigger challenge if they are in different rooms. This build-out effectively lets me have tons of fiber, and then manage connectivity in patch panels.
Again, given the number of devices at each location, putting 25/100GbE or 100GbE switches is cost-prohibitive from the initial purchase price standpoint as well as ongoing power and cooling costs. Further, when it comes time for the next upgrade cycle, it is cheaper to replace one switch than many.
Is this overboard, 100%. The original plan was to run 3x MM MTP-12 runs to five locations. Then I got a really good deal on the 144 fiber single-mode, and the 96 fiber multi-mode and so now those five locations are getting 240 fibers run to each.
Update 2021-09-16: We found a cable run that was too short for one of the big yellow single mode spools. Ordered a much longer one. The cable cost, had we used that unnamed vendor would have been $3,754.31. Instead dealing with a 2 day longer lead time to save around 90%. As a bonus, we can do another run with single-mode.
Key In-Wall Fiber Stats
Just for some sense of scale on the spools above, the plan is for:
Update 2021-09-16: Due to the extra cable, the revised plan is:
More on the patch panels soon. We are testing two different solutions that are rackmount solutions. What I *really* want is something more like this: Commscope FDH 3000 864 Port Quick Connect Indoor Fiber Distribution Cabinet | eBay
It is just out of the price range at the moment and even that one is not what I want.
Patch Cables
This is another area of a fairly large expense.
Wall Termination and Keystones
MTP keystones are very hard to get right now.
Switches
The actual goal and many are going to think I am crazy here, is to run the fiber network mostly off a single switch or at most two switches right now.
Something this build is really making me realize is that there are not a lot of good options for high-speed low-noise switches. 32x 100GbE switches are almost all 1U and therefore quite loud.
It feels to me like a decent business might be making switches for edge deployments that are perhaps 2-4U but are much quieter. A 1U 32x100GbE switch uses about as much power as a high-end desktop PC but screams.
Update 2021-09-16: Now have a Dell S5296F-ON 2U 96 port 25GbE + 8 port 100GbE 2U switch to test.
Optics
This is going to follow the switches. I currently have around 30 CWDM QSFP28 optics and have 30 SFP28 optics on order. I also have quite a few SR4 optics for multi-mode runs.
Test Equipment
Running anything at this scale basically requires that one does a lot of testing. Likely several Fluke products like the MultiFiber Pro. One of the challenges is that running both single-mode and multi-mode means that we need both the 850nm and 1310nm light sources plus the power meter, or something like the MFTK-MM850-SM1310 kit that runs ~$20K. The benefit of a kit like this is that it can test an MTP-12 cable in a single shot. Otherwise, there is a lot of time spent cleaning ends, getting reference points, and so forth.
The challenge will go well beyond the MTP-12 links in the walls. There also needs to be MTP-12 to LC cassettes so a MTP-12 cable can be split into six pairs. Not only do we need to test MTP, but we also need to test LC and that makes things extra challenging.
Update 2021-09-16:Looking more like we will be using Fluke test gear.
Stay Tuned
Hopefully, we will have fiber in the wall by Friday of next week. We are also doing a pretty solid set of re-wiring copper for more PoE+/PoE++ endpoints. I want to be able to test topologies of 6+ APs along with this. I also want to be able to have multiple camera feeds that we can use for video analytics / AI testing.
In case anyone missed the Plenum cable video, this is going to be an entire series that will run at least through 2021 on STH, but likely into 2022. Part of the reason this is so complex is just to have the infrastructure to do a ton of content. As great as 10Gbase-T is, for 25GbE and faster you certainly want fiber runs anywhere over ~10m. I think we have many STH readers asking for quieter infrastructure. When Rohit and I were doing the comparison, running a MTP-12 cable through a wall was ~2x the cost of the 2-fiber LC-LC cable but the real cost was in running each cable rather than the cables themselves. The basic premise is that instead of running a single LC-LC link between your office and rack and then needing a switch in the room, you can instead run MTP-12 and not need a loud switch that is using a lot of power in your office to service the two-strand back-haul.
MTP-12 is super cool, but also a bit scary. There are different fiber count MTP cables, one has to deal with polarity, male/ female plugs, couplers, cassettes, non-existent keystones and so forth. The payoff is that once it is set up, you get a ton of flexibility.
Update 2021-09-16: We should have some fiber in the wall by tomorrow, but we got delayed 1 day based on crew availability and then we pushed a day because of the long-run cable. That is the one to my office so it is one I have a big interest in seeing get run.
The project: Running (a lot of) fiber
Fiber Being Run
This is the fiber we have available as of now.
All of this is MTP-12 and will be used for in-wall installation. A quick breakdown:
- The 15x HPE spools on the left are OFNR LSZH OM4 50m spools
- The six big teal spools are 96-fiber (8x MTP-12) ONFP OM4 spools
- The six spools on the top right are OFNP OM4 100m spools
- The five yellow spools are 144-fiber (12x MTP-12) ONFP OS2 spools
The purpose of this build-out is to one get fiber to the two offices, the studio, and the server room. The other purpose, and this is quite important, is that I wanted to be able to do room-to-room direct networking without putting high-speed switches in the room. For example, in the YouTube studio, I did not want a 1U 100GbE switch. If I have something set up in my office, I wanted to be able to get a direct connection to another room easily. Sometimes we get new NICs, like a 200GbE NIC where we get two of them. Right now, I need both endpoints to be located next to one another. Scaling to four becomes a bigger challenge if they are in different rooms. This build-out effectively lets me have tons of fiber, and then manage connectivity in patch panels.
Again, given the number of devices at each location, putting 25/100GbE or 100GbE switches is cost-prohibitive from the initial purchase price standpoint as well as ongoing power and cooling costs. Further, when it comes time for the next upgrade cycle, it is cheaper to replace one switch than many.
Is this overboard, 100%. The original plan was to run 3x MM MTP-12 runs to five locations. Then I got a really good deal on the 144 fiber single-mode, and the 96 fiber multi-mode and so now those five locations are getting 240 fibers run to each.
Update 2021-09-16: We found a cable run that was too short for one of the big yellow single mode spools. Ordered a much longer one. The cable cost, had we used that unnamed vendor would have been $3,754.31. Instead dealing with a 2 day longer lead time to save around 90%. As a bonus, we can do another run with single-mode.
Key In-Wall Fiber Stats
Just for some sense of scale on the spools above, the plan is for:
- 121 MTP-12 runs total, half single-mode and half multi-mode
- 1,452 strands of fiber
- 5.3km of bundles
- 64km of individual fiber strands
Update 2021-09-16: Due to the extra cable, the revised plan is:
- 133 MTP-12 runs total, half single-mode and half multi-mode
- 1,596 strands of fiber
- 5.4km of bundles
- 83km of individual fiber strands
More on the patch panels soon. We are testing two different solutions that are rackmount solutions. What I *really* want is something more like this: Commscope FDH 3000 864 Port Quick Connect Indoor Fiber Distribution Cabinet | eBay
It is just out of the price range at the moment and even that one is not what I want.
Patch Cables
This is another area of a fairly large expense.
Wall Termination and Keystones
MTP keystones are very hard to get right now.
Switches
The actual goal and many are going to think I am crazy here, is to run the fiber network mostly off a single switch or at most two switches right now.
Something this build is really making me realize is that there are not a lot of good options for high-speed low-noise switches. 32x 100GbE switches are almost all 1U and therefore quite loud.
It feels to me like a decent business might be making switches for edge deployments that are perhaps 2-4U but are much quieter. A 1U 32x100GbE switch uses about as much power as a high-end desktop PC but screams.
Update 2021-09-16: Now have a Dell S5296F-ON 2U 96 port 25GbE + 8 port 100GbE 2U switch to test.
Optics
This is going to follow the switches. I currently have around 30 CWDM QSFP28 optics and have 30 SFP28 optics on order. I also have quite a few SR4 optics for multi-mode runs.
Test Equipment
Running anything at this scale basically requires that one does a lot of testing. Likely several Fluke products like the MultiFiber Pro. One of the challenges is that running both single-mode and multi-mode means that we need both the 850nm and 1310nm light sources plus the power meter, or something like the MFTK-MM850-SM1310 kit that runs ~$20K. The benefit of a kit like this is that it can test an MTP-12 cable in a single shot. Otherwise, there is a lot of time spent cleaning ends, getting reference points, and so forth.
The challenge will go well beyond the MTP-12 links in the walls. There also needs to be MTP-12 to LC cassettes so a MTP-12 cable can be split into six pairs. Not only do we need to test MTP, but we also need to test LC and that makes things extra challenging.
Update 2021-09-16:Looking more like we will be using Fluke test gear.
Stay Tuned
Hopefully, we will have fiber in the wall by Friday of next week. We are also doing a pretty solid set of re-wiring copper for more PoE+/PoE++ endpoints. I want to be able to test topologies of 6+ APs along with this. I also want to be able to have multiple camera feeds that we can use for video analytics / AI testing.
In case anyone missed the Plenum cable video, this is going to be an entire series that will run at least through 2021 on STH, but likely into 2022. Part of the reason this is so complex is just to have the infrastructure to do a ton of content. As great as 10Gbase-T is, for 25GbE and faster you certainly want fiber runs anywhere over ~10m. I think we have many STH readers asking for quieter infrastructure. When Rohit and I were doing the comparison, running a MTP-12 cable through a wall was ~2x the cost of the 2-fiber LC-LC cable but the real cost was in running each cable rather than the cables themselves. The basic premise is that instead of running a single LC-LC link between your office and rack and then needing a switch in the room, you can instead run MTP-12 and not need a loud switch that is using a lot of power in your office to service the two-strand back-haul.
MTP-12 is super cool, but also a bit scary. There are different fiber count MTP cables, one has to deal with polarity, male/ female plugs, couplers, cassettes, non-existent keystones and so forth. The payoff is that once it is set up, you get a ton of flexibility.
Update 2021-09-16: We should have some fiber in the wall by tomorrow, but we got delayed 1 day based on crew availability and then we pushed a day because of the long-run cable. That is the one to my office so it is one I have a big interest in seeing get run.
Last edited: