Homebrew 10gbe switch / router

Discussion in 'Networking' started by fractal, Jul 8, 2016.

  1. fractal

    fractal Active Member

    Joined:
    Jun 7, 2016
    Messages:
    309
    Likes Received:
    67
    Continued from Homebrew 10GbE switch

    Most places don't like necro'ing year old threads so I am starting a new one and ask the mods to merge them if that is preferred.

    Has anyone else tried what Chuckleb suggested? I got to thinking after reading iLya's thread on daisy chaining servers and that was the only thread I could find on building your own 10GBE switch / router.

    I am new to 10 GBE networking so ran a few experiments to see what was up. I built a NAS4Free box out of spare parts with an i3-2100 and a small SSD. I then added an Intel X520-DA2 to give it 10GBE capability.

    I put a HP branded Mellanox CONNECTX-2 NIC in a spare PC and two more in a supermicro motherboard which happened to have enough X8 slots. I spotted the Mellanox NICs on eBay at 2 for 40 with a DAC cable included. What a deal for a weekends worth of fun. I wasn't sure whether I wanted low profile or standard profile so I bought both. That gave me enough parts for these tests.

    The first thing I looked at was raw throughput. I created a file larger than cache on the NAS4Free drive then measured how fast I could read it.

    Reading the file directly on the NAS gives an idea of the speed of the SSD.

    nas4free: ~# dd if=/mnt/vol1/bigfile of=/dev/null bs=1M
    25000+0 records in
    25000+0 records out
    26214400000 bytes transferred in 92.943910 secs (282045376 bytes/sec)

    Reading the file on my test machine over a gigabit network suggests I am network limited by gigabit.

    fractal@mutt:~$ dd if=/media/lab1g/bigfile of=/dev/null bs=1M
    25000+0 records in
    25000+0 records out
    26214400000 bytes (26 GB) copied, 223.032 s, 118 MB/s

    Reading the file on my test machine over a directly connected 10G DAC is the same speed as reading it on the NAS

    fractal@mutt:~$ dd if=/media/lab10g/bigfile of=/dev/null bs=1M
    25000+0 records in
    25000+0 records out
    26214400000 bytes (26 GB) copied, 93.694 s, 280 MB/s

    I then restructured the 10G network with one DAC connecting the NAS to the machine with two Mellanox NICs and another connecting that router to the test machine. I enabled ip forwarding on the router and set up static routes on both the NAS and the test machine so they would talk via the router. Traceroute shows the connection

    root@mutt:~# traceroute 10.100.1.1
    traceroute to 10.100.1.1 (10.100.1.1), 30 hops max, 60 byte packets
    1 10.100.1.5 (10.100.1.5) 0.156 ms 0.198 ms 0.241 ms
    2 10.100.1.1 (10.100.1.1) 0.389 ms 0.413 ms 0.364 ms

    Copying the file from the NAS via the router was pretty quick.

    root@mutt:~# dd if=/media/lab10g/bigfile of=/dev/null bs=1M
    25000+0 records in
    25000+0 records out
    26214400000 bytes (26 GB) copied, 93.3364 s, 281 MB/s

    The router did not even notice the traffic. Top showed an IRQ handler occasionally using a percent of the CPU.

    So, as I would have expected, making my own 10G switch/router did not have any measurable impact on real world sequential throughput. I would expect transactional type traffic to notice the extra latency.

    Which brings us back to the original question. It seems to me that the real limit for people wanting to build their own homebrew 10GBE switching gear is the number of PCIe slots on your motherboard along with the density of your NICs. Three to six x8 slots seem to be the norm for supermicro server boards. But, by the time you put a half dozen, dual NICs in a machine like this you are getting pretty close to the cost of a 10GBE switch and likely consuming a whole lot more power.
     
    #1
Similar Threads: Homebrew 10gbe
Forum Title Date
Networking Homebrew 10GbE switch Feb 17, 2015
Networking Help with home 10GbE network (10Gbase-T and SFP+) Nov 19, 2019
Networking The usual noob question(s) about cheap 10gbe again Nov 17, 2019
Networking 10GBe vs. 25GBe - Cost Analysis and Part Numbers Oct 31, 2019
Networking Broadcom BCM57416 10Gbe vs Intel X550 10Gbe adapters? Sep 29, 2019

Share This Page