Homebrew 10GbE switch

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
So I've seen this discussion a few times now and I've thought about it myself. Let's ask the questions and see if we can get some testing/thoughts.

Overall question: Can we build an effective 10GbE switch out of cheap parts and have it work well?

- What are the limiting factors to make this work? CPU cores? Bus speed?
- Should I be able to slap in RouterOS or just turn on bridging in Linux and get this to work?
- How do I test saturation or performance? I don't even know how to generate enough traffic and to make sure my nodes are not the bottleneck.
- What are other problems or what else do I give up?

My goal would be quiet and relatively cheap with a low port count (6-8 ports). Power would be nice, but you can only get so much.

Here's a sample config:
MB: Supermicro X8DTL-i [$150]
CPU: L5520 [$20]
RAM: 3 x 4GB [$60]
SSD: 64GB [$35]
10GbE NIC: Brocade 1020 x 4 [$160]
Case/PS: Any.. [$100]

$525 for 8 ports so $65/port + dual GigE

If CPU power will be a factor, could double the CPUs and RAM (+$80).

This config is on my workbench right now (less the 10GbE NICs) and idles at 60W. Adding in a ConnectX3 makes it 65W at idle. I'd estimate full config at about 90W idle with case fans and all.

Don't know if I have time to build this to test, but would love to get the discussion started.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
*drool*...

80Gb saturates the x8 slot though and this cheap board doesn't have PCIE 3.0 I think. This is why the connect x3 boards say to use the dual ports as failover, not same time.

Yes, would kill the budget but use less power and noise. Would love to mix done 10GbT in here but those cards aren't cheap.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
I'm reasonably certain the limiting factor is going to end up being the CPU, the 12GB of RAM in that config is probably far more than required. And of course there's always also the option of adding some quad-port GigE NICs to the mix as well - customizable to the exact ratio of 1G to 10G ports for each use-case.

I believe just enabling bridging is all that is required for basic switch functionality, but if I were building it I would probably base it on pfSense. Built out of a PC it is as capable of being a full-featured router as it is a dumb switch may as well use software that will not just enable that functionality but give it a nice web GUI as well. Especially if virtualized hosts are part of the mix it might be nice to have this box do VLAN trunking as well as inter-VLAN routing.

As for performance testing, I expect it to be hit hardest just trying to keep up with the rate of packets with a few 10G connections, and so would configure any synthetic benchmarks to blast away with a small packet size and see just how much CPU power gets used up. With 9K jumbo-frames I would hope to be able to saturate the pipe before running out of CPU. For a more real-world test, in my experience the most bandwidth intensive thing I use on a regular basis is vMotion - get a couple of ESXi hosts connected to it and move a VM with 64+ GB of RAM back and forth, with multi-NIC vMotion if you really wanna abuse the switch.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
It will work, but getting reasonable performance will be a challenge. I think your biggest issue will be packet forwarding latency. With the Linux Bridge approach you'll see something close to a full millisecond of forwarding latency. With a GigE configuration this is probably no biggy, but at 10Gbe it's deadly to performance.

For example, at 10Gbe a full 9k jumbo frame is transmitted in just over 7ms, meaning that if you have 1 ms of forwarding latency you waste 17% of the available bandwidth...with smaller frames sizes you'll waste upwards of 50% or more.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Well I just emailed MicroTik folks to see if they could offer suggestions for this as well with their RouterOS. I'm curious if they have tricks since they have 10Gb ports on their products as well.
 

mervincm

Active Member
Jun 18, 2014
159
39
28
I found the best way to do this was VYOS, a fork off Vyatta. I ran them in a VM to bridge between 2 10G ports before I bought a MicroTik
the switch was easy to config, my notes look like this

login vyos:vyos
configure
edit int eth eth0
set bridge-group bridge br0
exit
edit int eth eth1
set bridge-group bridge br0
exit
set system host-name vyos1-10G
set system domain-
set system name-server 192.168.110.1
set system gateway-address 192.168.110.1
set int bridge 'br0'
set int bridge br0 address '192.168.110.19/24'
set service ssh port 22
commit
save
 
  • Like
Reactions: Patrick

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Well I just emailed MicroTik folks to see if they could offer suggestions for this as well with their RouterOS. I'm curious if they have tricks since they have 10Gb ports on their products as well.
Their products all have a switch/bridge chips on them that do layer-2 forwarding without having to pull the packets up into the CPU. By using them they get rid of the packet forwarding latencies introduced by the Linux bridge and/or routing layers. They do most of the layer-3 work in the CPU so as a "router" they also suffer from latency issues.
 

jtreble

Member
Apr 16, 2013
93
10
8
Ottawa, Canada
... I think your biggest issue will be packet forwarding latency ...
+1. My big concern as well. I'm hoping, however, to use a XL710QDA2 and do all of my switching "intra-nic" (max. 8@10GbE ports). I'm really not sure about this but I'm guessing that as soon as I would need to hit the PCIe bus (i.e., inter-nic switching) performance would fall off a cliff.

BTW, kudos to Chuckleb for starting this thread - it's long overdue.
 
Last edited:

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
+1. My big concern as well. I'm hoping however, to use a XL710QDA2 and do all of my switching "intra-nic" (max. 8@10GbE ports).
Can you use breakout cables to get 4 to 1? I bought cables (and still have them) but was not able to get Mellanox cards to break them out. They are designed for the switch side... to break out a 40Gb port into 4x 10Gb. I don't think you get 4 Ethernet MAC addresses basically.
 

Entz

Active Member
Apr 25, 2013
269
62
28
Canada Eh?
One issue you may find with RouterOS is lack of support for the Brocade cards. Vyatta would be a better choice in that regard.

Im curious how you would measure the forwarding latency. I have a setup right now with a couple of ConnectX-2 EN cards bridged in Ubuntu. Works great for my uses ( >1gbit <= 10gbit) but may not be optimal, especially if you go from say 2 to 6 or 8 ports..

Anyways +1 for this, definitely long overdue. Will be interesting to see how it scales out.
 

DolphinsDan

Member
Sep 17, 2013
90
6
8
Does pfsense do ConnectX-2 EN cards? You'd probably save money on SFP+.

Is there a good way to really test switches made like this? Will it matter port to port?
 

capn_pineapple

Active Member
Aug 28, 2013
356
80
28
Specifications

Product code
  • XL710QDA2
Connections
  • Single or Dual QSFP+ cages for:
    • SFP+ SR fiber-optic transceivers
    • SFP+ LR fiber-optic transceivers
    • SFP+ Direct Attach cables
Physical Layer Interface
  • 40GBASE-SR
  • 40GBASE-LR
  • SFF-8431:
    • 40GSFP+ CR4 (Direct Attach Copper)
    • 40GSFP+ CR4 to 4x SFP+ (Breakout Cable) [Awesome]
Data rate(s) supported per port
  • Optical: 10 GbE/40 GbE
  • Direct Attach: 40 GbE
 

Patriot

Moderator
Apr 18, 2011
1,450
789
113
Does pfsense do ConnectX-2 EN cards? You'd probably save money on SFP+.

Is there a good way to really test switches made like this? Will it matter port to port?
You would need clients and load generators.
I would hypothesize if you used 2 cards crossing the pcie bus to the other card would have higher latency than staying on the same card.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
@capn_pineapple Yeah I think that's still one way. This lets you hook the card into a switch that only has SFP+ ports, not QSFP+ ports.

"This cable type allows customers to bridge the gap between 10GbE and 40GbE cables, switches and adapters, and economically links a QSFP+ port with an upstream 10GbE SFP+ switch"

Intel PDF

The cards are competing against the Mellanox ConnectX3 cards basically, but lower power, etc. I tried this with a Mellanox card and couldn't figure out how to get it to make 4 ethernet devices. The port is only visible as one device.

The RouterOS folks recommended this card which gives you 6 SFP+ cages:
HotLava > Ethernet Adapters > Maximize Your Network's Bandwidth
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Just some food for thought on this project. Not responding to anyone in particular - just comments on a few common themes that are being discussed:

  • No matter what you do for multi-port NICs you are still going to be dealing with CPU-based forwarding (and the latency that it will introduce). Multi-port NICs do not do layer-2 forwarding between ports. You can't "keep the packets on the NIC" unless the NIC has layer-2 switch capabilities on it. None of the multi-port NIC cards being discussed do this natively.

  • Using RoS or Vayatta or whatever is OK. You'll get a decent switch/router that will forward packets and will work OK. You'll get quit a bit of benefit over a 1Gbe configuration, but you won't get full line-rate performance due to the packet forwarding latencies. You'll do much better with large packets than with small ones (duh!).

  • Testing a setup like this just requires a "source" and "sink" system to pump packets and measure the results. A couple more systems with 10Gbe NICs and software like Seagull (open source protocol test platform) or Landslide (a proprietary test suite). While these test suites are really geared towards testing higher-level protocols they do quite well performance testing simple protocols like TCP.

  • The 6-port HotLava adapters are really expensive. You can buy a real 10Gbe switch for the same money a couple of these cost. It sorta defeats the purpose. Well, the cost effective purpose. You are still left with the "fun" and "challenge" part if money is not in your way.

  • ConnectX-2 EN are probably the lowest cost used cards that will work reliably with RoS and/or Vyatta. The Brocade cards might work but YMMV. Costs go up fast with ConnectX-3 EN, ConnectX-3 VPI and the various versions of Intel-based card.
 

bds1904

Active Member
Aug 30, 2013
271
76
28
AFAIK brocade 10xx and all connectx cards are confirmed not working in routeros. Only 10gb card that is confirmed is intel x520 based cards.
 
  • Like
Reactions: Chuckleb

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
The cards are competing against the Mellanox ConnectX3 cards basically, but lower power, etc. I tried this with a Mellanox card and couldn't figure out how to get it to make 4 ethernet devices. The port is only visible as one device.
I think the ConnectX-3 cards cannot do 1->4 using a cable, but the mellanox FDR VPI switches can. (my memory may be off as I am not sleeping much)

On the XL710 side - they are absolutely awesome. I only have one which the main site Fortville content was based off of, but I am a big fan. They are perhaps one of the most underrated pieces of tech out there today.
 

azev

Well-Known Member
Jan 18, 2013
768
251
63
The one way i know to test and saturate switch backplane is to do snake test. You still need packet generator, but it will give you end to end throughputs and latency.