Continued from Homebrew 10GbE switch
Most places don't like necro'ing year old threads so I am starting a new one and ask the mods to merge them if that is preferred.
Has anyone else tried what Chuckleb suggested? I got to thinking after reading iLya's thread on daisy chaining servers and that was the only thread I could find on building your own 10GBE switch / router.
I am new to 10 GBE networking so ran a few experiments to see what was up. I built a NAS4Free box out of spare parts with an i3-2100 and a small SSD. I then added an Intel X520-DA2 to give it 10GBE capability.
I put a HP branded Mellanox CONNECTX-2 NIC in a spare PC and two more in a supermicro motherboard which happened to have enough X8 slots. I spotted the Mellanox NICs on eBay at 2 for 40 with a DAC cable included. What a deal for a weekends worth of fun. I wasn't sure whether I wanted low profile or standard profile so I bought both. That gave me enough parts for these tests.
The first thing I looked at was raw throughput. I created a file larger than cache on the NAS4Free drive then measured how fast I could read it.
Reading the file directly on the NAS gives an idea of the speed of the SSD.
nas4free: ~# dd if=/mnt/vol1/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 92.943910 secs (282045376 bytes/sec)
Reading the file on my test machine over a gigabit network suggests I am network limited by gigabit.
fractal@mutt:~$ dd if=/media/lab1g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 223.032 s, 118 MB/s
Reading the file on my test machine over a directly connected 10G DAC is the same speed as reading it on the NAS
fractal@mutt:~$ dd if=/media/lab10g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 93.694 s, 280 MB/s
I then restructured the 10G network with one DAC connecting the NAS to the machine with two Mellanox NICs and another connecting that router to the test machine. I enabled ip forwarding on the router and set up static routes on both the NAS and the test machine so they would talk via the router. Traceroute shows the connection
root@mutt:~# traceroute 10.100.1.1
traceroute to 10.100.1.1 (10.100.1.1), 30 hops max, 60 byte packets
1 10.100.1.5 (10.100.1.5) 0.156 ms 0.198 ms 0.241 ms
2 10.100.1.1 (10.100.1.1) 0.389 ms 0.413 ms 0.364 ms
Copying the file from the NAS via the router was pretty quick.
root@mutt:~# dd if=/media/lab10g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 93.3364 s, 281 MB/s
The router did not even notice the traffic. Top showed an IRQ handler occasionally using a percent of the CPU.
So, as I would have expected, making my own 10G switch/router did not have any measurable impact on real world sequential throughput. I would expect transactional type traffic to notice the extra latency.
Which brings us back to the original question. It seems to me that the real limit for people wanting to build their own homebrew 10GBE switching gear is the number of PCIe slots on your motherboard along with the density of your NICs. Three to six x8 slots seem to be the norm for supermicro server boards. But, by the time you put a half dozen, dual NICs in a machine like this you are getting pretty close to the cost of a 10GBE switch and likely consuming a whole lot more power.
Most places don't like necro'ing year old threads so I am starting a new one and ask the mods to merge them if that is preferred.
Has anyone else tried what Chuckleb suggested? I got to thinking after reading iLya's thread on daisy chaining servers and that was the only thread I could find on building your own 10GBE switch / router.
I am new to 10 GBE networking so ran a few experiments to see what was up. I built a NAS4Free box out of spare parts with an i3-2100 and a small SSD. I then added an Intel X520-DA2 to give it 10GBE capability.
I put a HP branded Mellanox CONNECTX-2 NIC in a spare PC and two more in a supermicro motherboard which happened to have enough X8 slots. I spotted the Mellanox NICs on eBay at 2 for 40 with a DAC cable included. What a deal for a weekends worth of fun. I wasn't sure whether I wanted low profile or standard profile so I bought both. That gave me enough parts for these tests.
The first thing I looked at was raw throughput. I created a file larger than cache on the NAS4Free drive then measured how fast I could read it.
Reading the file directly on the NAS gives an idea of the speed of the SSD.
nas4free: ~# dd if=/mnt/vol1/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 92.943910 secs (282045376 bytes/sec)
Reading the file on my test machine over a gigabit network suggests I am network limited by gigabit.
fractal@mutt:~$ dd if=/media/lab1g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 223.032 s, 118 MB/s
Reading the file on my test machine over a directly connected 10G DAC is the same speed as reading it on the NAS
fractal@mutt:~$ dd if=/media/lab10g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 93.694 s, 280 MB/s
I then restructured the 10G network with one DAC connecting the NAS to the machine with two Mellanox NICs and another connecting that router to the test machine. I enabled ip forwarding on the router and set up static routes on both the NAS and the test machine so they would talk via the router. Traceroute shows the connection
root@mutt:~# traceroute 10.100.1.1
traceroute to 10.100.1.1 (10.100.1.1), 30 hops max, 60 byte packets
1 10.100.1.5 (10.100.1.5) 0.156 ms 0.198 ms 0.241 ms
2 10.100.1.1 (10.100.1.1) 0.389 ms 0.413 ms 0.364 ms
Copying the file from the NAS via the router was pretty quick.
root@mutt:~# dd if=/media/lab10g/bigfile of=/dev/null bs=1M
25000+0 records in
25000+0 records out
26214400000 bytes (26 GB) copied, 93.3364 s, 281 MB/s
The router did not even notice the traffic. Top showed an IRQ handler occasionally using a percent of the CPU.
So, as I would have expected, making my own 10G switch/router did not have any measurable impact on real world sequential throughput. I would expect transactional type traffic to notice the extra latency.
Which brings us back to the original question. It seems to me that the real limit for people wanting to build their own homebrew 10GBE switching gear is the number of PCIe slots on your motherboard along with the density of your NICs. Three to six x8 slots seem to be the norm for supermicro server boards. But, by the time you put a half dozen, dual NICs in a machine like this you are getting pretty close to the cost of a 10GBE switch and likely consuming a whole lot more power.