Multi-interface server box feasibility

[Peter_U]

Hey all,

I'm kinda thinking about doing something different. I have a few different servers (~6) at home. I also have a completely eclectic mix of network cards. 40gb IB, 10 gig SFP and Cat6, 56GB IB/ 40G Ethernet QSFP+.

Each interface I've gotten over time but usually only 2-3 cards each. I'm kinda thinking I want to take my dual E5 server and make like a Noah's Ark of network interface cards. One of each dual port interface type. I would probably run Windows Server 2012 R2 on this since that's what I'm most comfortable with.

Then I want to use the second cards of each pair to connect the other servers. It would look like:
Server 0: all interfaces
Server 1 and 5: SFP+ 10G
Server 2: CAT6 10G
Server 3: QSFP 40GB IB/ Ethernet
Server 4: QSFP 56GB IB/ Ethernet

Here's the thing, I KNOW I can get a switch. But how hard would it be to have Windows just manage all of this on one subnet and let Server 4 talk to Server 1?

Does Server 0 just become the gateway instead of my router?

 

[NotMine999]

Hummm....

If you operate all ports in the same subnet, the "server0" is not routing and your "gateway" remains the same. If "server0" performed routing between all of these interfaces, then it would perform a "gateway" function and the add the complexity of configuring routing to this challenge.

Having said that, with the different cards that you have, you will encounter a networking phenomena called "speed mismatching" and possibily even "MTU mismatching". Your "server0" would be expected to buffer traffic from "high speed" cards to "low speed" cards, regardless of MTU settings. You could always consider "bonding" (802.3ad) multiple "low speed" ports into a larger "high speed aggregate" port, but then you encounter another networking phenomena called "load balancing across links"...and "load balancing across links" (and an aggregate bundle has to be built from links all of the same speed) doesn't always behave as you expect.

As for the challenge of trying to do all of this on Windows, well ummm...I have tried hard to forget everything I have learned about that OS. What I do remember is this: If Windows can do something in the networking area, it won't be easy, might not work as expected, and troubleshooting tools can be "slim or none". Ok, my personal choice for an OS in this case would be Linux, but this challenge isn't any easier on a Linux OS.