Infiniband vs 10G

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

talsit

Member
Aug 8, 2013
112
20
18
I am in the process of moving and racking my home lab.

Right now I have a mix of equipment stacked on a workbench in my garage, a Dell c2100 with dual L5639's, HP DL120G7 with E3-1230, a full tower with SM X8DTI and dual L5639's and a recently added Dell R320 with E5-2407.

Honestly, in their current location, I have been connecting to them via USB wifi dongle while they do their thing. They are used primarily for learning and Folding At Home right now. They get turned off when outside temperatures stay above 80 because it isn't cool enough in the garage for them to run at full load above 80.

I'm moving this equipment into a rack in my new, much better climate controlled basement and I will probably add another server or two that currently resides in my office to the rack. Those office servers are loaded with my storage (Tiered Storage Spaces) and VMs for learning and I run Plex off one server which I also use as a workstation

Right now, the office servers are on a 1GB wired network with an unmanaged switch connected to a wifi router which is also where Internet access comes in via cable modem.

In the interest of education, what would be the best way to go on stacking this equipment in a rack? Be happy with wired GB Ethernet? 10GB cards and direct connections in the rack, 1GB out to clients? 40GB Infiniband in the rack via direct connect?

In the long run, I want to setup some VMs that are home mission critical that will be shared out to thin clients/dumb clients/RDP on the network but nothing too taxing. Those will probably be clustered with some sort of failover.

I'm waiting on a quote to pull cable from the basement through the house but (as I understand it) I can get Cat6 pulled and use it for 1GB or 10GB, right?

I want to maintain and expand centralized home file storage and backup. If I can get everything into one chassis in the rack, I can then back that up locally to another chassis and remotely.

I will probably also setup pfsense or some other firewall as I move forward.

Thoughts, opinions?
 

Scott Laird

Active Member
Aug 30, 2014
317
147
43
If you're happy with WiFi, then 1G is probably fast enough and it's basically free. At some point, 10Gbase-T will be reasonably priced, and you can consider upgrading if you need the bandwidth.

Doing mixed Ethernet/Infiniband networks is a pain in the neck. I like to think that I know what I'm doing, and I was constantly having weird little issues; one day one machine would re-enable IP reverse path filtering and silently blackhole some traffic, the next week an IB driver would hang, and so forth. It's all manageable, and if you really need the bandwidth for cheap then it's not a bad choice, but you'll end up spending a lot more time then you expect on it.

It's possible that single-homing all of your servers with IP and having pfsense connected to both networks wouldn't be that bad (assuming pfsense does IB, I haven't checked), but it's still a much bigger pain then just using Ethernet.
 
  • Like
Reactions: talsit

Scott Laird

Active Member
Aug 30, 2014
317
147
43
Wow. Those have fallen like a rock since I last looked. $800 still isn't *cheap*, but it's a lot less work then routing everything yourself or dual-homing.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,058
113
I paid around that 6 or so months ago too, just have to submit offers and negotiate as most had them listed for 1200+ it seemed.
 

talsit

Member
Aug 8, 2013
112
20
18
Thanks for the info. I'll keep an eye on the 4036E. It's EOL so I probably need to start hoarding documentation?
 

Jerry Renwick

Active Member
Aug 7, 2014
200
36
28
43
Thank you for the nice information about the 4036E, I have gained some new information about it!;)