Hobby 100Gbps Network Build

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

dudemcbacon

New Member
Sep 19, 2024
3
1
3
Hi friends!

I'm putting together a list of parts I'll need to build a small 100Gbps network in my home lab. I'm pretty inexperienced with optics so I was wondering if someone could check my thinking on the parts list below. I've also created a simple network diagram that illustrates the addition to my network which is attached.

My main concern is the connection to the 'Desktop' client in the diagram. It seems like 100GBase-SR4 transceivers are the right choice but I was surprised how expensive the cables are (anywhere from ~$300-600 based on my limited Googling). Am I making the correct choice here? Are there cheaper alternatives? I briefly read about 100GBase-BiDi transceivers but those seem too expensive for the hobbyist (~$400) compared to the 100GBase-SR4 transceivers (~$50).

The run to the 'Desktop' client isn't quite 100m but it's difficult to estimate given the route the cable will have to take through my house (up to the ceiling, across the garage, through the wall to the outside of the house, up one story around a window, and then through the wall again into my office). I figured I'd just get the maximum allowable length to hedge my bets.

Here's the parts list:
QtyItemPriceNet Price
3Mellanox Connect-X 5 (MCX515A-CCUT)120360
210Gtek 100G QSFP28 DAC Cable2550
110Gtek 100G QSFP28 100GBase-SR4 (OEM) Transceiver5050
110GTek 100G QSFP28 100GBase-SR4 (Mellanox) Transceiver5050
1Plenum MTP Fiber Optic Cable, 8-Fiber, Single-mode, 100 Meter320320
110Gtek 40G QSFP+ DAC Cable2525
1MikroTik CRS504-4XQ-IN650650
Total1505

If you're curious what the purpose is, I have a flash storage array on my servers in the garage that I'd like to share with my desktop without a huge network bandwidth limitation.

Appreciate any advice and thank you all for being a great resource!
 

Attachments

  • Like
Reactions: lightsword

dudemcbacon

New Member
Sep 19, 2024
3
1
3
I'm mostly playing with theoretical numbers here but a 4TB Samsung 990 Pro can do sustained sequential reads of ~7GB/s (56Gbps). Imagine a few of those in an array and it's hard not to see myself getting limited by network bandwidth. That's my thinking, at least.

Also, this is mostly just for fun so I'm not tooooooo hung up on the numbers. :)
 

seany

New Member
Jul 14, 2021
20
9
3
What are you doing that needs 100Gb to the workstation?
I just upgraded to 25G and have a hard time keeping that saturated from a flash storage pool.
Not sure what OP is upto, but I've upgraded network speeds before not because we were running out of bandwidth but because we wanted lower latency for small packets and the raw line encoding rate was faster.
 

nexox

Well-Known Member
May 3, 2023
1,399
645
113
You can get CWDM4 transceivers for $5-10 on ebay, they only require duplex single mode fiber, which should be a whole lot less expensive. Note: I haven't used them yet, my 100G stuff is all DACs so far.
 

mvs123

New Member
Jul 28, 2024
9
1
3
Munich, Germany
100GBase-SR4 transceivers are made for multimode fiber... Look for 100GBase-LR4, if you want to use single mode MTP/MPO patchcord.
I would rather grab 3-4 used 100GBase-CWDM4 2km tranceivers on ebay and duplex LC-LC single mode UPC patchcord.

It makes also not much sense to buy 40G QSFP+ DAC, since 100G QSFP28 DACs are backwards compatible.
 

dudemcbacon

New Member
Sep 19, 2024
3
1
3
100GBase-SR4 transceivers are made for multimode fiber... Look for 100GBase-LR4, if you want to use single mode MTP/MPO patchcord.
I would rather grab 3-4 used 100GBase-CWDM4 2km tranceivers on ebay and duplex LC-LC single mode UPC patchcord.

It makes also not much sense to buy 40G QSFP+ DAC, since 100G QSFP28 DACs are backwards compatible.
Thank you for the tip! I will lookg at 100GBase-CWDM4 transceivers. Do I need to worry about signal attentuators for such a small run? I honestly know nothing about optics but I've read about people burning out their transceivers when the power levels were too high.

Also, good call on the 40G DAC, I didn't consider that!
 

dsrhdev

New Member
May 28, 2024
1
0
1

EBRAddict

Member
Jun 16, 2023
33
10
8
You'll need some extra cooling for the 100GB cards, they run hot. The 100GBase-CWDM4 transceivers also run hot but they are cheap as is the single mode fiber cabling. One pair of my 100GB cards has a 60C throttle limit and they idle at 55C with a fan blowing directly on them. Mine wouldn't play nice with sleep/hibernate on Windows, and you can forget about wake on LAN. They also require a dedicated x16 pcie slot.

The upside: move a ton of data with very low latency.