So I am working on a full rebuild of all of my systems here. Putting aside the main file server (in another thread) I am trying to get the virtualization gear replaced now. Below are the three designs I am about 99% settled on at this point. Thankfully Supermicro refreshed their line and the SC113M looks to be a much better fit for me than other options. I do need to determine how loud the four included 40mm fans are and if needed replace them with quieter ones. The chassis can take up to eight 40mm fans so I could replace the four original ones with eight quieter (less effective) ones.
As the MB10-DS3 has SFP+ and gig RJ45 ports, I will connect one SFP+ directly to the VM_SAN box below for data to ESX_Core and then one RJ45 to my network for mgmt. ESX_Core however will have the one SFP+ tied up for the NFS data while the second will uplink to the network (no 1GB RJ45 being used). For now it is a small setup and with the ESX_Core and one node the direct link to VM_SAN should be fine. At some point when I add a third node, I will go back to the network and redesign the 10GB connectivity.
Build’s Name: VM_SAN
Operating System/ Storage Platform: FreeBSD 10.3
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 2x Seagate ST1000NX0323 1TB RAID1 | 4x Seagate ST91000640SS 1TB RAID10 | 1x ST91000640SS 1TB HSP
RAM: 2x M393A2K43BB1-CRC0Q (16GB modules, 32GB total)
Add-in Cards: LSI 9361-4i Raid Controller
Power Supply: Supermicro supplied in chassis
Usage Profile: VM Storage System
Other information: This system will act as the file server for VM's. The RAID1 array will be the OS installation and the RAID10 will be used for the storage. I have 6 of the ST91000640SS drives here and while I could create a 6 drive RAID10, I would be left with no spare drives. Since they are out of warranty, for now I don't need any more than ~1.8TB and the performance should be fine for the load I will throw at it I figured I would go that route. I will have one additional drive bay left over and I could put in the last 1TB for 2 HSP's but I see no reason to put hours of use on it for likely very little gain. Other than that, it will run FreeBSD and serve the ESX storage up via NFS.
Build’s Name: ESX_Core
Operating System/ Storage Platform: ESX 6.0
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 16GB USB Flash Drive | 1x Samsung 850 Pro 1TB | 4x Seagate ST2000NX0243 2TB RAID5 | 2x Samsung 850 Pro 1TB RAID1 | 1x Samsung 850 Pro 256GB
RAM: 2x M393A4K40BB1-CRC0Q (32GB modules, 64GB total for now)
Add-in Cards: LSI 9361-4i Raid Controller
Power Supply: Supermicro supplied in chassis
Usage Profile: Core ESX System
Other information: Primarily this system will run the core VM's. For now it will run all VM's until I build a second ESX chassis (same specs as this machine, just without all of the local hard drives). The local drives are purposed in the following manner:
16GB USB Flash Drive: ESX OS
1x Samsung 850 Pro 1TB: Passthrough Dedicated Live Surveillance Storage
4x Seagate ST2000NX0243 2TB RAID5: Passthrough Dedicated Archival Surveillance Storage
2x Samsung 850 Pro 1TB RAID1: Local VM Storage
1x Samsung 850 Pro 256GB: Passthrough Dedicated Disk for a VM
The reason for local storage on this box is to keep the hardware requirement as small as possible for booting. This system will run the backend VM's for core systems (virtualized router and pbx, network services, etc) and needs to be online regardless of any other systems (obviously if the switches are offline then I am screwed but if the VM_SAN goes down then whatever). A large number of VM's will be kept on local storage though the remainder will be stored on the VM_SAN in preparation of adding another ESX chassis and migrating them to that chassis.
Build’s Name: ESX_n+1
Operating System/ Storage Platform: ESX 6.0
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 16GB USB Flash Drive
RAM: 2x M393A4K40BB1-CRC0Q (32GB modules, 64GB total for now)
Add-in Cards: None
Power Supply: Supermicro supplied in chassis
Usage Profile: ESX Node
Other information: All non core VM's will be generally moved to this chassis and if needed additional ram will be installed. As needed additional nodes will be built and installed and vMotion enabled to handle loads better.
As the MB10-DS3 has SFP+ and gig RJ45 ports, I will connect one SFP+ directly to the VM_SAN box below for data to ESX_Core and then one RJ45 to my network for mgmt. ESX_Core however will have the one SFP+ tied up for the NFS data while the second will uplink to the network (no 1GB RJ45 being used). For now it is a small setup and with the ESX_Core and one node the direct link to VM_SAN should be fine. At some point when I add a third node, I will go back to the network and redesign the 10GB connectivity.
Build’s Name: VM_SAN
Operating System/ Storage Platform: FreeBSD 10.3
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 2x Seagate ST1000NX0323 1TB RAID1 | 4x Seagate ST91000640SS 1TB RAID10 | 1x ST91000640SS 1TB HSP
RAM: 2x M393A2K43BB1-CRC0Q (16GB modules, 32GB total)
Add-in Cards: LSI 9361-4i Raid Controller
Power Supply: Supermicro supplied in chassis
Usage Profile: VM Storage System
Other information: This system will act as the file server for VM's. The RAID1 array will be the OS installation and the RAID10 will be used for the storage. I have 6 of the ST91000640SS drives here and while I could create a 6 drive RAID10, I would be left with no spare drives. Since they are out of warranty, for now I don't need any more than ~1.8TB and the performance should be fine for the load I will throw at it I figured I would go that route. I will have one additional drive bay left over and I could put in the last 1TB for 2 HSP's but I see no reason to put hours of use on it for likely very little gain. Other than that, it will run FreeBSD and serve the ESX storage up via NFS.
Build’s Name: ESX_Core
Operating System/ Storage Platform: ESX 6.0
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 16GB USB Flash Drive | 1x Samsung 850 Pro 1TB | 4x Seagate ST2000NX0243 2TB RAID5 | 2x Samsung 850 Pro 1TB RAID1 | 1x Samsung 850 Pro 256GB
RAM: 2x M393A4K40BB1-CRC0Q (32GB modules, 64GB total for now)
Add-in Cards: LSI 9361-4i Raid Controller
Power Supply: Supermicro supplied in chassis
Usage Profile: Core ESX System
Other information: Primarily this system will run the core VM's. For now it will run all VM's until I build a second ESX chassis (same specs as this machine, just without all of the local hard drives). The local drives are purposed in the following manner:
16GB USB Flash Drive: ESX OS
1x Samsung 850 Pro 1TB: Passthrough Dedicated Live Surveillance Storage
4x Seagate ST2000NX0243 2TB RAID5: Passthrough Dedicated Archival Surveillance Storage
2x Samsung 850 Pro 1TB RAID1: Local VM Storage
1x Samsung 850 Pro 256GB: Passthrough Dedicated Disk for a VM
The reason for local storage on this box is to keep the hardware requirement as small as possible for booting. This system will run the backend VM's for core systems (virtualized router and pbx, network services, etc) and needs to be online regardless of any other systems (obviously if the switches are offline then I am screwed but if the VM_SAN goes down then whatever). A large number of VM's will be kept on local storage though the remainder will be stored on the VM_SAN in preparation of adding another ESX chassis and migrating them to that chassis.
Build’s Name: ESX_n+1
Operating System/ Storage Platform: ESX 6.0
CPU: Xeon D-1541
Motherboard: Gigabyte MB10-DS3
Chassis: SC113MFAC2-605CB
Chassis Rails: MCP-290-00056-0N
Chassis Riser Card: RSC-RR1U-E16
Drives: 16GB USB Flash Drive
RAM: 2x M393A4K40BB1-CRC0Q (32GB modules, 64GB total for now)
Add-in Cards: None
Power Supply: Supermicro supplied in chassis
Usage Profile: ESX Node
Other information: All non core VM's will be generally moved to this chassis and if needed additional ram will be installed. As needed additional nodes will be built and installed and vMotion enabled to handle loads better.