Help with cheap 40Gb links and network planning?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rootpeer

Member
Oct 19, 2019
73
13
8
Hello! I am moving some stuff around in my house and I need some advice.

I am situated it Europe and I live in an apartment on the 4th floor. I had a spare room temporarily set up as a server room but that has to go now. I am basically running just one server right now (Dell R720xd) which I am using as a NAS and Docker host.

The server is going to be moved to the basement. I used to have it there before but it was only on a 1Gb copper link so i moved it to the main house to connect it to 10Gb with MM fiber.

I have ordered and I am going to run a 4-core Single Mode LC fiber cable from the apartment to the basement because it was the only cheap outdoor rated cable I could find. With the cable being SM, it opens up the avenue to move to 40Gb instead of 10Gb. One core is going to be used to carry the CATV signal my fiber ISP is sending at 1550nm to the basement. That leaves 3 cores empty with 2 being usable for a duplex link unless I can find a way to inject the CATV signal at the apartment and extract it in the basement while being able to use that core for a duplex link as well.

There are two devices that need high speed networking, the NAS and my workstation. The plan for the future is to upgrade the apartment switch to a Brocade ICX6610, have a 40Gb DAC cable for the workstation and use optics to connect to the basement, either to some inexpensive 40Gb switch or directly to the NAS and bridge the NAS' 40Gb NIC to the 10Gb NIC to provide networking to other servers/devices.

Basically I need:
  • 2x 40Gb NICs as cheap as possible. Looking at some Mellanox/Fujitsu NICs but need advice on what model I should buy. Also, I would like them to have functional SR-IOV as I am currently using SR-IOV for Docker container and VM networking on both machines. I had tried SR-IOV with ConnectX-3 ENs but Windows guests would not work with the open source Linux host driver. Both workstation and server are running Arch Linux.
  • 2x 40Gb SM QSFP+ modules. I see these go for at least 130EUR on Ebay but I found some AOI 40G-LR4 Lite modules for 20EUR each. Has anyone messed with these with Brocade switches or Mellanox NICs?
  • Cheap basement switch recommendations. I would like a switch that can take the 40Gb link from the Brocade, connect to the NAS at 40Gb and then possibly have a breakout cable to provide 10Gb to other servers and to a Brocade ICX6450. Mellanox SX6012?

Links to what I found on Ebay: