Building infrastructure from zero.

GCM

Active Member
Aug 24, 2015
137
43
28
So, I'm building out some new server infrastructure for one of my clients. They pretty much have nothing, and had all 220ish users on Dropbox or Google Drive... The engineering team will be sharing the rack space once they move into the office come December.

Here is what I came up with:

Hyper-visor OS - Ubuntu Server
VM - KVM
Directory Services - OpenDJ
Storage - ZFS from Ixsystems
Network - Core network will be 10GbE, branching off to 1GbE. Video editors *may* get 10GbE to the desktop.
Rack - It's completely self contained, fully sealed. Very interesting to work with.

Parts list:

SWITCH x2 : Juniper ex3300-24t
STORAGE: TrueNAS Z30 - Usable 18TB - 128GB RAM - 2x SSD l2arc - SSD ZIL - 2x 10GbE
SERVER x2: RAX XS4-1130V3-SH (E3 CPU, 32GB RAM, Raid 10 with HGST 4TB Drives)
UPS x2 : EATON 5P UPS 2200VA
RACK: CC42U8000M6-R2

So, here are a few pictures of what I've got going so far:




 

GCM

Active Member
Aug 24, 2015
137
43
28
Is that a Neat Patch I spy in the rack shot? I love those things.

I'd be interested to hear more of what you think of that sealed rack system.
It is! I literally put them in EVERYWHERE I touch the network. I've lost count of how many I've purchased...

And we'll see in a week how the sealed rack is working out. I can tell you one thing, that rack is the heaviest rack I've dealt with, and that AC unit was incredibly hard to get on by myself due to the weight. The rack itself is very nice though, completely sealed, swinging doors both front and back, adjustable depth.
 

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
So, I'm building out some new server infrastructure for one of my clients. They pretty much have nothing, and had all 220ish users on Dropbox or Google Drive... The engineering team will be sharing the rack space once they move into the office come December.

Here is what I came up with:

Hyper-visor OS - Ubuntu Server
VM - KVM
Directory Services - OpenDJ
Storage - ZFS from Ixsystems
Network - Core network will be 10GbE, branching off to 1GbE. Video editors *may* get 10GbE to the desktop.
Rack - It's completely self contained, fully sealed. Very interesting to work with.

Parts list:

SWITCH x2 : Juniper ex3300-24t
STORAGE: TrueNAS Z30 - Usable 18TB - 128GB RAM - 2x SSD l2arc - SSD ZIL - 2x 10GbE
SERVER x2: RAX XS4-1130V3-SH (E3 CPU, 32GB RAM, Raid 10 with HGST 4TB Drives)
UPS x2 : EATON 5P UPS 2200VA
RACK: CC42U8000M6-R2

So, here are a few pictures of what I've got going so far:
I see that either yourself or the client is trying to do things cheap possibly.

That being said, the server you are looking at is already screaming alarms. You will need a smaller and separate array to place the Host OS and VM's on. Do not even try installing the lot to a 8TB volume. A pair of Mirrored SSD's to hold Host OS and VM's, create another VHD and attach from the big array that will NOT handle 10GbE very well as you will only see around 220-230MB/sec out of the array with limited I/O depending on the RAID controller.
 

GCM

Active Member
Aug 24, 2015
137
43
28
I see that either yourself or the client is trying to do things cheap possibly.

That being said, the server you are looking at is already screaming alarms. You will need a smaller and separate array to place the Host OS and VM's on. Do not even try installing the lot to a 8TB volume. A pair of Mirrored SSD's to hold Host OS and VM's, create another VHD and attach from the big array that will NOT handle 10GbE very well as you will only see around 220-230MB/sec out of the array with limited I/O depending on the RAID controller.
Sort of both.

Honestly, the services running on the VM will be minuscule compared to what most people are running. I did crunch the numbers of what would be required. There is zero reason for the VM's to utilize the 10GbE array except when backing up nightly. Really, I mean it when I say those two servers are going to be hosting bare bones VM's.

What you do not see yet is an actual proper speced KVM infrastructure that is currently hosted elsewhere, and will be brought into the rack come December. This build was literally meant to hold them over until then, and then utilize the two servers for other purposes.

That being said, the ZFS box will stay. I'll be doing a separate post with a full write up on it, since it's a TrueNAS system. I'll post the price, full specs, and detail shots.
 

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
What you do not see yet is an actual proper speced KVM infrastructure that is currently hosted elsewhere, and will be brought into the rack come December. This build was literally meant to hold them over until then, and then utilize the two servers for other purposes.
Not sure if there is bad terms being used here, are you referring to PCoIP in place of the KVM you mention? I ask because I actually use a Dell R5400 as my daily rig and very familiar with these setups and love them, just not used enough.

If there is to be any more gear, the UPS will be way under speced for other workstations/servers.
 

GCM

Active Member
Aug 24, 2015
137
43
28
Not sure if there is bad terms being used here, are you referring to PCoIP in place of the KVM you mention? I ask because I actually use a Dell R5400 as my daily rig and very familiar with these setups and love them, just not used enough.

If there is to be any more gear, the UPS will be way under speced for other workstations/servers.
UPS's will also be brought over too! What you see now is one set up, with a second being transplanted soon.
 

Dajinn

Active Member
Jun 2, 2015
512
78
28
29
Were they opposed to using Microsoft software/os for virtualization and active directory?
 

GCM

Active Member
Aug 24, 2015
137
43
28
Were they opposed to using Microsoft software/os for virtualization and active directory?
Very opposed.

Their entire core infrastructure for the site / app is on AWS, and averages about 35 million uniques a month.