My Home Server Farm

[x86guru]

Build’s Name: x86guru a.k.a. Adrian a.k.a 3ph_power
Operating System/ Storage Platform: Debian 10 w/custom kernel
CPU: Mostly E5-2697 V2s, E5-2690 V2s, E5-2680 V2s, and a few V3s, and V4s.
Motherboard: All PowerEdge, three Intel R2208, and one 4U super-micro
Chassis: R720, R620, R630, R740, Intel R2208, Super Micro SC468
Drives: NVMe
RAM: Several Terabytes. Each server has either 64GB, 128GB, 192GB, or 256GB.
Add-in Cards: 108 Mellanox ConnectX-3 FDR 56G cards. 8 systems have 6 FDR 56G cards installed, the remaining all have 3 FDR 56G cards installed.
Network: Three Mellanox MSX6036F 36-port FDR 56Gbit/s switches (All 108 ports on all 3 switches are in use).
Power Supply: Lots of 750w and 1100w units all powered from 4 PDUs fed from two 50A 240VAC circuits
Other Bits: My servers are located in my home office's walk-in closet. Cooled with a 12K BTU A/C system

Usage Profile: For work and play. I design/develop ultra-high end enterprise class software for work and also for fun/play.












List of equipment (output from my own management utility)

 

[maxermaxer]

This is epic! Can you show how you use the ducts to exhaust the heat? Where do the two ducts connected to?
Do you run all the rigs at the same time and how much power do they draw?

It is a very nice and tidy rack!

 

[x86guru]

This is epic! Can you show how you use the ducts to exhaust the heat? Where do the two ducts connected to?
Do you run all the rigs at the same time and how much power do they draw?

It is a very nice and tidy rack!

Thanks! I use cable management arms so that I can quickly and easily always fully extend any of the servers for re-configuring PCIe cards, memory, etc

The two ducts (inlet/exhaust) go to a 12K BTU A/C unit off to the side of the racks in my closet. See the picture below. I do run everything at the same time for some tasks but mostly I'm only running a few servers and only when I need them. When all servers are running, and all cores are active and pinned everything draws around ~60 AMPS from the 240V PDUs. My A/C is 120V and it draws ~9 AMPS.

 

[x86guru]

One more picture of my 6 terabit fabric with 108 56Gbit/s connections

 

[nry]

This is mind blowing! As a software developer myself who gets by with 2 dual Dell R720 servers, I'd love to know more about what you do which requires so much compute!

 

[rskoss]

Thanks! I use cable management arms so that I can quickly and easily always fully extend any of the servers for re-configuring PCIe cards, memory, etc

The two ducts (inlet/exhaust) go to a 12K BTU A/C unit off to the side of the racks in my closet. See the picture below. I do run everything at the same time for some tasks but mostly I'm only running a few servers and only when I need them. When all servers are running, and all cores are active and pinned everything draws around ~60 AMPS from the 240V PDUs. My A/C is 120V and it draws ~9 AMPS.

View attachment 17190

How do you vent the AC outside? I have a similar AC but don't use it because installing the vent in a window is a huge security hole in my house.

 

[Whaaat]

I hope this carpet is fire- and spark- proof

 

[csp-guy]

OMG, beautiful cable management porn, and it seems, the stuff has owner.
Congrats!

 

[x86guru]

How do you vent the AC outside? I have a similar AC but don't use it because installing the vent in a window is a huge security hole in my house.

I vent the AC outside a window that is behind my servers. No security issues here, but it would be easy to make it secure -- at least as secure as a normal window.

 

[x86guru]

I hope this carpet is fire- and spark- proof

 

[x86guru]

OMG, beautiful cable management porn, and it seems, the stuff has owner.
Congrats!

I need to completely redo the three Infiniband switches and use some type of cable arms and maybe stager the switches so that I can actually see the ports. It's really hard to actually see the ports. I should have given myself 2U between the switches but I wired up the switches last after everything else was racked up.. Something like this would be great to help see the switches

 

[213NSX]

****.

 

[BoredSysadmin]

Did you hack your electrical company and they give you free electricity?

 

[Sean Ho]

This is a really impressive setup in a small amount of floor space! Do you have whole-home surge suppression? I assume most of this rack is usually powered down, and any servers that are on 24/7 are protected by a UPS somewhere?

 

[x86guru]

This is a really impressive setup in a small amount of floor space! Do you have whole-home surge suppression? I assume most of this rack is usually powered down, and any servers that are on 24/7 are protected by a UPS somewhere?

I only run one system 24/7 (My SuperMicro 846 file server) and that is powered through a Liebert GXT4 3KW UPS with double-conversion technology so it's well isolated form any line-power issues. There is no protection for the other servers as they are only used as-needed for testing new software I'm developing and if they get damaged I can just re-load images/replace hardware as needed. I do power all of the other servers with switched Server Technology power distribution units which might have some kind of surge protection; but I don't think it does.