Basic C6100 deployment to consolidate home network

[briancl]

I am building a home server to take on multiple tasks for my home network: network router/security, media storage (both archival and real-time DVR), system backups for about 15 systems, a number of VMs for various household services, and some work-related workspaces. Performance is important, but I believe the gear I've selected is up to the task. Reliability and redundancy are also important, but my needs aren't mission critical. I'm often away for a few days at a time traveling for work, and my wife still needs this stuff operational since its integrated into my house. Downtime would really suck, but I don't need HA redundant pairs of each node with cold spares. Power and noise are medium priority issues.. I think I know what I'm getting on that front.

6 Months ago when I started reaching the limit of my current setup, I did the research and figured I'd just go with and AiO ESXI powered by a 1240v2 Ivy Bridge Xeon. I never loved the idea, and just started deleting stuff off my file server to put off the need for the new server. I was hoping something better would come along, and I think the C6100 may be just that. Plus, I've run out of things to delete.. I really need to get a new server up and running.

Here is a rough outline of what I've got planned:

C6100 with 4 nodes, 2x L5520, 24GB per node

- Node 1: Networking - pfSense - Embedded install, boot from USB/DOM

- Node 2: Storage - OmniOS w/ Napp-it; Boot Drives: 2x40gb Raid 1 Intel 320 SSD

• ZFS Data Store pool: 8x3TB RaidZ2 = 18TB effective (considering WD Reds)
• ZFS VM OS Drive pool: 2x500GB Raid1 = 500GB effective (considering Samsung 840 Pro)

- Node 3: Primary VM - Proxmox, StackOps, OpenStack, ESXi??

• Linux + Windows workspaces
• Web Server
• Plex/media/entertainment
• Home Automation
• Telephony
• Misc Services

- Node 4: Sandbox VM - Proxmox, StackOps, OpenStack, ESXi??

• Misc VMs for staging
• Test new hypervisors

Networking gear:
- HP ProCurve 1810 or 1910
- Possibly a Ubiquiti EdgeMAX hardware router in lieu of pfSense

1) Will an embedded pfSense install work on a node?
2) Any concerns about the storage layout? Can I wire up the drives so that they all go to Node 2? Any significant benefit to running the VM OS Drive Pool directly on Node 3, as sort of an AiO zfs/VM host node?
3) Any other suggestions on the OS drive for OmniOS? 2x40gb Intel 320 SSD seems good to to me, but I'm open to other ideas. Mirroring here is smart because if the filer is down, everything is down.
4) What are my expandability options for storage? How would I go about adding another 8 drives for another vdevl? Can I design the system better now for future expansion? I know I will use up all of this space as some point, so I want to consider future growth in the build.
5) What is the natural bottleneck here given my use and design? If I could do something better, what is that? What's the best bang for the buck upgrade or change? What's the best way to add more reliability or redundancy?
- I'm considering upgrading the RAM.. any advantage to doing it now, or can I wait on that?
- Would 10gb be useful (Infiniband)? I feel like maybe the throughput and IOPS will be just fine with the 1gbe links. If I went with additional NICs in each node, is there any benefit to Link Aggregation?
6) What else? Am I missing anything? What would you do differently?

 

[Patrick]

Check this one out: Colocation Architecture Installation - ServeTheHome 2013

Using single L5520 pfsense nodes with 12GB each (WAY too much.) USB DOM is a bit harder since there are no internal headers yet, but folks are working on that http://forums.servethehome.com/processors-motherboards/1638-c6100-extender-board-usb.html

On adding drives, one thing dba may shed some light on is his 16 lane LSI 9202-16e setup. You can also see his great JBOD guide.

 

[dba]

Sounds like fun!

You can save some power by using a single CPU in those nodes that don't do serious processing - the pfsense node for sure and almost certainly the storage node. You can also save power by using fewer DIMMS and tweaking the OS and MOBO power settings.

With all that horsepower, and multiple VMs hitting your storage node, dual gigabit links will almost certainly be a bottleneck. I found that a pair of c6100 nodes, each with dual 5520s and 96GB RAM, barely breaks a sweat while running my 37 VMs, but performance was awful until I gave those VMs some disk IO. My initial deployment used a single dedicated VM disk on each node - ~200MB/S and 200 IOPS total across both nodes) which was of course too slow. The first experiment was to quickly slot in an external SAS RAID with ~1,000MB/S and around 600 RAID6 IOPS. This was almost as bad as using local disk. The second experiement was to up the RAID to around 8,000 IOPS. This was much much better - quite usable really. The third experiment was to try SMB3 over Infiniband to a very serious chunk of SSD storage with more throughput and IOPS than anyone would want for VMs. Very nice but very expensive. These experiments proved what we already know: Running multiple VMs requires both throughput and IOPS. But then the surprising part:

Lastly, and not thinking it would work very well, I tried replacing the external RAID storage with a single large internal SSD drive in each c6100 node. This decidedly low-tech solution ended up being my favorite - simple, low power, and with surprisngly snappy performance. In the end, I am using five smaller SSD drives per node, all directly attached (24-bay c6100 with LSISAS2008 Mezz card). This gives me scary good performance - 2,600MB/S sequential read speed and 300,000 4kb IOPS per node. I am not using RAID at all, which might surprise you. Instead, I run Hyper-V replication to my storage node every five minutes, with a VSS (snapshot) replication point every four hours, keeping about the last 16 hours worth of snapshots live on the storage node. This, plus a twice-daily backup are my guards against failures and human error. This cheap-and-simple strategy would not be appropriate for a transaction system, but I highly recommend it for VMs. I can recover from any error or failure up to and including a completely dead node by "spinning up" the failed VM in another node in about ten seconds, using any of the snapshots.

 

[briancl]

Hmm.. All great info. Thanks guys.

dba,
The SSD drive pool itself certainly will support enough IOPS, but what you're saying is that the network connection (dual GB) would limit the effective performance, i.e., my dozen or so VMs would be slow at times. Is it throughput or IOPS that the 2x1gbe would limit?

What if I were to move the 2x512GB SSD ZFS VM OS drive pool from Node 2 (Storage) to Node 3 (Primary VM)? This would effectively make Node 3 an AiO build with both storage and VM on one node. Then Node 2 is really just slow big storage and nothing more. Node 4's demand on the drive pool would be pretty low, so I could run that across the network to the drive pool on Node 3.. or just toss in a single small SSD to run local on Node 4.. it's not a production node anyway.

The other option is mitigating the network bottleneck. I suppose I'd do that with an Infiniband card.. one each in Node 2 (Storage) and Node 3 (Primary VM), directly connected. That should sustain whatever IOPS my VMs need, right? However, if I use up my only pcie slot with a network card, then I lose my storage upgrade path to add more drives to the storage node via HBA + SAS expander

 

[Mike]

The HP 1810 range is the 'real' procurve series and some people swear by it. Note that you have the v1 and v2 with some minor differences, mainly size, power usage and featureset. It does have a small buffer though.

Also I would save some power by virtualising Pfsense. Doable on vmwarez but slightly harder on the other hypervisors.

 

[Patrick]

Have two of the HP 1910-24G's.Thus far very stable and both have lifetime warranties which is nice.

 

[dba]

You'd have a throughput bottleneck for sure while IOPS would probably be OK. Best case, your two gigabit ports are good for a bit above 200MB/Second - about the same as two SATA drives, which isn't much when divided amongst three, five, ten VMs. You would have a noticeable throughput bottleneck. As for IOPS, my rough guess is that you'd go from 100K raw SSD read IOPS to between 5K and 20K after adding in network latency and ZFS overhead. With a different SSD and a single network connection I saw 7K IOPS over SMB3, for example. 10K IOPS is good enough for ten average VMs according to my rough rule of thumb.

Moving the SSDs to the VM machine would make the VMs very happy. Using your original architecture but adding more gigabit connections would also help incrementally (add a quad card and you'd triple your maximum throughput) and of course adding Infiniband would solve the bandwidth and latency problems once and for all. If you take that approach and are worried about PCIe real estate, hunt down the Dell c6100 IB Mezzanine card. With two ports per card, you can directly connect up to three c6100 nodes in a ring without needing a switch of any kind. I saw 85K IOPS with a Windows share over IB in my very first try.

Hmm.. All great info. Thanks guys.

dba,
The SSD drive pool itself certainly will support enough IOPS, but what you're saying is that the network connection (dual GB) would limit the effective performance, i.e., my dozen or so VMs would be slow at times. Is it throughput or IOPS that the 2x1gbe would limit?

What if I were to move the 2x512GB SSD ZFS VM OS drive pool from Node 2 (Storage) to Node 3 (Primary VM)? This would effectively make Node 3 an AiO build with both storage and VM on one node. Then Node 2 is really just slow big storage and nothing more. Node 4's demand on the drive pool would be pretty low, so I could run that across the network to the drive pool on Node 3.. or just toss in a single small SSD to run local on Node 4.. it's not a production node anyway.

The other option is mitigating the network bottleneck. I suppose I'd do that with an Infiniband card.. one each in Node 2 (Storage) and Node 3 (Primary VM), directly connected. That should sustain whatever IOPS my VMs need, right? However, if I use up my only pcie slot with a network card, then I lose my storage upgrade path to add more drives to the storage node via HBA + SAS expander

 

[briancl]

So 2 or 3 of these makes my existing architecture work? Dell Mellanox QDR VPI 2 Port 40GB s Daughter Card for Dell PowerEdge C6100 JR3P1 | eBay

Also, looking at my original architecture, I need something to help me wire up the extra drives (beyond the first 6). I can't find an LSI 9202-16e, which appears to be a strongly recommended card here. I could get away with half as many slots with an M1015, right? I don't have as much upgrade headroom with the M1015, but it will work for now. No matter what, I need an HBA.. what options should I consider?

I am 95% committed to the C6100 at this point... I just need to figure out what bits and pieces I need in order to make it work for me. You can't beat the economics of it when comparing it against a powerful AiO, or even more modest separate storage and VM servers.

 

[dba]

That is the card. Two or three of those are one way to make your existing architecture work. Another is to add a quad-port Gigabit card to each node and use LACP - not nearly as fast but sticks with familiar technology.

The particular auction you pointed to is a good one since it includes the riser board and the little metal bracket that is absolutely required but is often missing and does not have its own part number.

You are limited to a maximum of six internal drives per node. If you have a 24-bay chassis then all of your nodes get six drives while if you have a 12-bay chassis then each node gets three drives, though you can re-wire it with Dell parts to re-distribute the drives - as discussed in other posts in this forum. The motherboard ports are 3Gbit SATA of course. There are six-port 3Gbit and 6Gbit SAS/SATA mezzanine cards available instead, but the 6Gbit cards are rare and expensive and neither card works if you have already fitted an Infiniband Mezzanine card. Otherwise, you can use an internal-port RAID card or HBA instead of the motherboard ports, with a cable whose part number I do not know**, or you can use an external port card like the 9202-16e with 16 ports or 9200-8e/9205-8e/9207-8e with eight ports. If you want both Infiniband and 6Gbit drives then using an Infiniband Mezzanine and a PCIe SAS card will be cheaper than a Mezzanine SAS card and a PCIe Infiniband card.

If you are bulk storage-oriented then the 12-bay 3.5" chassis will be appealing. If you are focused on SSD drives then the 24-bay 2.5" chassis will be appealing.

**Has anyone found the cable set used when you have an LSI PCIe SAS card connected to the internal drives? The Dell docs talk about such a setup. The cable has to be different than the one for the SAS Mezzanine card.

So 2 or 3 of these makes my existing architecture work? Dell Mellanox QDR VPI 2 Port 40GB s Daughter Card for Dell PowerEdge C6100 JR3P1 | eBay

Also, looking at my original architecture, I need something to help me wire up the extra drives (beyond the first 6). I can't find an LSI 9202-16e, which appears to be a strongly recommended card here. I could get away with half as many slots with an M1015, right? I don't have as much upgrade headroom with the M1015, but it will work for now. No matter what, I need an HBA.. what options should I consider?

I am 95% committed to the C6100 at this point... I just need to figure out what bits and pieces I need in order to make it work for me. You can't beat the economics of it when comparing it against a powerful AiO, or even more modest separate storage and VM servers.