Three server 10 GbE network without a switch. Possible?

[herby]

I currently have three servers at home: two Xenservers, and a Freenas box. With the promised I/O improvements in the next Xenserver release and Carp/Hast being added to Freenas I'm interested in virtualizing my storage and moving up from a multipathed Gigabit storage network to 10GbE.

To avoid using a switch I was thinking of adding a 2 port 10 GbE nic of some kind to each server, connecting in a ring, and using Open Vswitch to span/mirror the traffic to both ports.

Would this work?

 

[Scott Laird]

If you set it up in a ring with each node as a 2-port switch, then you'll either (a) have a switching loop (bad) or (b) have STP turned on, in which case it'll turn down one link to break the loop. Unless openvswitch is especially bright, then you'd probably be better off setting each link up as its own /30 and then adding static routes as needed. It's probably be easier than a switch, anyway, but it'll behave poorly if any of the links come unplugged. Dynamic routing would fix that, but that's probably overkill.

 

[herby]

I figured using span it would work like this:

1) One server would request something using both ports
reaching both servers:

2) Desired server would respond on both ports:

 

[Mike]

You can do that with Openflow yes. Create static flows in openvswitch and you do not even need a controller or STP for that matter, but watch out with creating loops. I'm not sure how efficient the 10gbe bridging would be though.
Another option would be to have redundant paths with different priorities.

 

[Lost-Benji]

I would just go with "Get a switch" and take out software and system dependencies.

 

[Mike]

I would just go with "Get a switch" and take out software and system dependencies.

Most 10gbe switches may use as much power as his entire cluster and OpenVswitch is already a system dependency to have network connectivity?

 

[PigLover]

Get a switch. They are not as expensive and power-hungry as you might think. The Juniper EX2500 in my rack draws less than 100w. There are quite a few GigE switches with 4x SFP+ 10Gbe uplinks that could service these three nodes. Also, sometimes there are deals to be had on more capable switches (example: a 24 port Juniper EX2500 just like the one I use recently went on eBay for $399).

Takes some shopping & diligence but 10Gbe switches are no longer out of reach.

 

[geeker342]

So, if I wanted to set up an alert in eBay for a 10Gbe switch, what parameters would you recommend to eliminate the chaff? Is there a good list of "starter switches" in a home lab environment? I see plenty of stuff with 2x 10Gbe SFP+ ports but something with 4x would be better, and more is almost always better.

 

[Lost-Benji]

Look for any with quad SFP+ ports.
The Mikrotiks might be an option for you to keep costs down and use a lot less power than the 100W suggested.