homelab upgrades to 10gbe + SAN

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AllenAsm

Member
Jul 30, 2018
55
3
8
I need a SAN for my home lab so that I can run lots of virtual machines (30ish currently) and have failover, etc. I tried backing them into a traditional NAS (linux machine with local 7200 rpm drives) over 1gig network and the virtual machines were very slow despite having 2 hyperv machines with 32 cores (64 logicals) and 192gigs of ram each. I did some research and found that the bottleneck was the NAS and the networking to that NAS.

So, I've upgraded all of the servers to these 10gbe 'converged' cards (so I can have HBA and RDMA)
https://www.amazon.com/gp/product/B01HMGWOU8/ref=oh_aui_detailpage_o08_s00?ie=UTF8&psc=1

I also upgraded the switch to this 10gbe switch
https://www.amazon.com/gp/product/B0787TKSQ9/ref=oh_aui_detailpage_o00_s01?ie=UTF8&psc=1

I however need to add a SAN and that is where I am struggling. Should I just get a vanilla tower or 2u and get some sort of SAS 12gb/s hardware raid care to connect all the drives via? Should I try and put a ZFS set of disks together?

upload_2018-7-31_10-59-56.png

Any advice on the SAN side would be greatly appreciated.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
I however need to add a SAN and that is where I am struggling.
Should I just get a vanilla tower or 2u and get some sort of SAS 12gb/s hardware raid care to connect all the drives via? Should I try and put a ZFS set of disks together?
Without sounding condescending...please don't get me wrong...Your concept of SAN does not seem to relate to your understanding of hardware.

I'd first map out your VMs and their storage requirements, both from a quantity and speed perspective.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Without sounding condescending...please don't get me wrong...Your concept of SAN does not seem to relate to your understanding of hardware.

I'd first map out your VMs and their storage requirements, both from a quantity and speed perspective.
No offense taken as I know enough to be dangerous but there are for sure holes in my understanding (why I'm posting here). The SAN requirement comes from working at a large company and hearing the IT guys talk about having RDMA for direct machine to machine ram access which speeds up the vm's considerably. Apparently iSCSI can be really slow if its over the transport layer and doesn't directly access RAM.

My requirements are as stated, I need the virtual machines that all do a fair amount of disk access when processing things to run smoothly even when there are many of them.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
No offense taken as I know enough to be dangerous but there are for sure holes in my understanding (why I'm posting here). The SAN requirement comes from working at a large company and hearing the IT guys talk about having RDMA for direct machine to machine ram access which speeds up the vm's considerably. Apparently iSCSI can be really slow if its over the transport layer and doesn't directly access RAM.

My requirements are as stated, I need the virtual machines that all do a fair amount of disk access when processing things to run smoothly even when there are many of them.
Sure, but what does the translate to? Heavy IOPS? Heavy writes? mostly reads? multi GB media files or a lot of small files? Does it need to be highly available (now, you're talking two of them)? What about data redundancy?

You have two 512GB SSDs in raid0 for OS. Why? At best you need two 80gb disks for the OS, assuming you want to mirror them.

You have 2x 2TB SSDs as the caching tier (which will hopefully be your most used data). Is that striped? mirrored? Are these SAS2/3 disks?

You have 4x 12gb/s SAS disks as your data tier. How did you come up with 4 disks? mirrored/striped/raidz2/zfs (You're on Windows)/reFS/NTFS?? Why 12gb/s?? Because it's the fastest available?

Are you going to store your VMs on this SAN or are they going to be stored on the Raid1 SSD on your boxes? If you're storing them on the boxes, why? What is the SAN for then?

Why have 2x1tb SSD for your DC? At best you need two 80gb disks for the OS, assuming you want to mirror them. Then use the SAN for data. Isn't that why you're building it?

What's the total size of your data?

etc etc.

That's what I meant by mapping your VMs to their storage requirements.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Sure, but what does the translate to? Heavy IOPS? Heavy writes? mostly reads? multi GB media files or a lot of small files? Does it need to be highly available (now, you're talking two of them)? What about data redundancy?

You have two 512GB SSDs in raid0 for OS. Why? At best you need two 80gb disks for the OS, assuming you want to mirror them.

You have 2x 2TB SSDs as the caching tier (which will hopefully be your most used data). Is that striped? mirrored? Are these SAS2/3 disks?

You have 4x 12gb/s SAS disks as your data tier. How did you come up with 4 disks? mirrored/striped/raidz2/zfs (You're on Windows)/reFS/NTFS?? Why 12gb/s?? Because it's the fastest available?

Are you going to store your VMs on this SAN or are they going to be stored on the Raid1 SSD on your boxes? If you're storing them on the boxes, why? What is the SAN for then?

Why have 2x1tb SSD for your DC? At best you need two 80gb disks for the OS, assuming you want to mirror them. Then use the SAN for data. Isn't that why you're building it?

What's the total size of your data?

etc etc.

That's what I meant by mapping your VMs to their storage requirements.
these are great questions, I apologize for not being more specific. Let me try my best to answer.

Raid1 for OS because I had a VM machine with windows server 2008 that blew through a 128g raid 1 disk array with just OS upgrade files and such. I wanted to make sure that didn't happen again. Also was thinking that I wanted it to have plenty of high speed storage for SWAP files if needed. Its overkill, but piece of mind you know...

The 2x2tb SSD's on each are expected to also be RAID1. This is a total SWAG and something I was hoping would be right. My plan was to adjust the size by adding more RAID1 ssd's if I needed to. This was also a driving reason behind going with a SAS3 / 12gbps 2.5 cage so I could seamlessly add more SSD storage with a quick visit to the CO-LO instead of having to power down and crack the case open.

The 4x SAS 6tb drives are to be RAID5 so I can start with some 18tb of storage. I currently use about 12ish TB of storage for both virtual machines and the data they use/product. This is also the total current size of my data.

The particular VM storage requirements vary significantly so I'm trying to get a one size slays them all if you will on this. :) Some are IOPS heavy while others are straight up massive file processing heavy. One of my recent test environments dealt with graph databases and various overlay performance characteristics for instance while another processed massive web advertising logs.

As stated previously, my existing setup virtual machines are slow and it appears the network and NAS are the bottleneck. I could go into all the reasons I need this but the easy answer is that my personal consulting business is booming and I do a LOT of tests and theory work against this lab. I'm willing to spend some bucks on making this extremely fast in support of that.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
That clarifies a few things.

That said, since you called this a "lab", I'm assuming HA and redundancy are not top requirements?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
I would not buy the networking gear and instead buy 10TB drives for each system. 2x10TB mirrored in each, you already have 2x2TB for "fast" stuff storage, throw in another SSD if you want/need to cache the 10TB drives. You could buy 3x 10Gig 2 port nics and connect them together for 'backups' to the other local server too, or to a NAS device if you want to make backups/moving lots of data faster.

You'll eliminate the networking bottleneck, which WILL still be there especially at 10Gig.
You'll eliminate the cost of networking.
You'll keep things much more simple to configure, maintain, troubleshoot, and get the most performance out of.
You'll have the option to add optane cache tier which will totally blow any SSD out of the water and not be severely delayed by networking.

When people at large enterprises are talking about a high performance SAN utilized by the VM hosts it's not really the same as a home SAN with 4 drives in it :) I had a home/business SAN with 24 drives in it and it still didn't make sense to keep it that way vs. go hyperconverged for my relatively "small" setup. I saved boatloads of power and increased performance exponentially, and Id idn't need to upgrade to 40Gig to get to the 'start' of performance I wanted ;)

Not for everyone, but thought I'd share another option :)
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
I would also note that if you need high IOPs and/or consistent performance your SSD drives should NOT be consumer but enterprise, I like Intel myself :) but I hear good things about the newer-generation Samsung that don't have bad latency like the older models.

> performance use NVME (newer gen Intel are MUCH faster than original-release like p3700... like the P4600 is great 'bargain')
>> performance use Optane :)
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Huge generalization here, SAN is really mostly used for HA clusters that need shared storage, If your application supports any over network replication (SQL server always on for example) then local NVMe, Optane if it works with the volume is really the goods.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Need for HA isn't that necessary since it is a LAB but with that said, i've had disks fail before and I don't want to have to rebuild everything which is why you see raid 1 and 5 everywhere in this setup.

10gbe cards and switche are already purchased and in the lab and i love how fast they are. So that part has been a success.

The problem I'm having with hyperconverged is that my plan is to add more hardware to the lab so I can do larger more meaningful tests and within that context I need self-leveling. Hyper-V seems to be begging me to have a single location for disk images so that it can move to a different host seamlessly if one of the physical machines starts to pin out. I'm not an expert with Hyper-V though so I may or may not be assuming those things correctly.

I'm still leaning towards making a SAN for the RDMA aware hyperv cluster but the arguments against are compelling as well. I really appreciate all of the input.

If I do go the SAN route, what SAS3 Raid card would you guys recommend? There seem to be several options out there.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
My opinion is that there's no point in any of that with a single point of failure still in your 10Gig switch AND single NIC per-server which not only you have setup to use for your storage network, but also your 'normal' network.

What's the point of load balancing or ability to migrate VMs if all it takes is 1 switch and ALL your servers, storage, etc, go down ???
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
As much as I have fun building bargain-basement SANs, unless you've got an absolute requirement for shared storage and HA (and from everything you've said so far, you don't), I'm going to side with T_Minus and the rest and say don't waste your time and money on one at this stage. It's adding a significant amount of complexity that, reading between the lines, I'm not sure you're quite ready for.

Raid1 for OS because I had a VM machine with windows server 2008 that blew through a 128g raid 1 disk array with just OS upgrade files and such. I wanted to make sure that didn't happen again. Also was thinking that I wanted it to have plenty of high speed storage for SWAP files if needed.
Semi-offtopic, and it's possibly a contentious opinion given the frequent arguments I still have with some people about it, but personally I make people apply for special dispensation if they want boxes with more than 4GB of swap (windows or *nix). If you don't have enough physical memory and eat into swap for any significant portion of the dataset being held in RAM, performance will tank and then you're also chewing into your precious IO. Large swathes of swap are only useful IMHO if you've got applications habitually leaking memory, whereupon the leaks will (hopefully) eventually get paged out to disk and never read back. But if you've got memory leaks it's far more efficient (not to mention cheaper) to fix them or come up with a mitigation strategy than it its to waste money on "fast" swap space.

The 4x SAS 6tb drives are to be RAID5 so I can start with some 18tb of storage. I currently use about 12ish TB of storage for both virtual machines and the data they use/product. This is also the total current size of my data.
Four spindles in an 18TB RAID5 will likely be pretty dire for random IO unless it's got a lot of RAM and SSD in front of it as cache (and with drives that size you're at great risk of losing the array during a drive replacement/rebuild but that's a topic for another thread but you should look into using RAID6 or better to lessen the chances of running into a URE). Even with caching, ultimately you're still going to be limited by the rate at which things can actually be written to the backing discs. If your IO requirements are as high as you seem to think then four SAS drives for the bulk of your storage is not enough (unless I've misunderstood and they're 6TB SAS SSDs?).

By way of comparison, my humble home server-cum-NAS uses 6x6TB drives in RAID10 with an 128GB SSD writeback cache - I've used it as an iSCSI target before, but if I was making a SAN for the sort of VM hosts you're talking about I'd be wanting at least 16 spindles and 1TB of SSD fronting it (see many other threads on this site for people using FreeNAS and similar systems to do this very effectively).

What sort of numbers are we actually talking here, in terms of required IOPS and throughput?

my existing setup virtual machines are slow and it appears the network and NAS are the bottleneck.
What exactly was it they were bottlenecking on, network or NAS, throughput or IOPS? I've seen plenty of people waste money on 10GbE when the NAS behind their NICs couldn't even saturate 1GbE - usually because it had nowhere near the amount of platters/SSDs needed to keep up with lots of random IO - and any IO from more than a couple of VMs will be as near to random as makes no odds. Your average bog-standard linux NAS box with less than a dozen spindles and little in the way of cache will choke on that pretty quickly.

Hence a lot of people recommending you keep it local if you can - VMs running off local SSDs are pretty much as good as it gets performance-wise, add a SAN layer (be it FC, iSCSI or anything else) and you're immediately sacrificing potential performance and reliability for the sake of having a SAN. If it's not a crucial component, try and leave it out - your hairline will thank you ;)

I'd propose you attempt to segregate your storage and your VMs; put some of the SSDs into one host and run off a local SSD data store there, and use that box for your hefty random IO boxes. Then run your big fat HDD RAID array and some SSDs (possibly in an SSD-fronted array if windows or your RAID controller supports that) and use that for processing your large dataset stuff.

There's a lot of ways to skin this particular cat, but jumping straight from "NAS is slow" to "I need to build build an RDMA-enabled hyperconverged SAN on windows using only four SAS drives and no dedicated storage fabric" seems like you might be trying to skin the cat using a water pistol whilst the cat is still alive and with its claws perilously close to your nether regions :)

(All my £0.02, YMMV, IMHO, yadda yadda)
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
"Dedicated SAN Fabric" -- There the 3 words I was looking for, ha ha :D
 
Last edited:

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
As long as you do not connect the JBOD enclosures to multiple servers. Sure S2D will work that way. I had an 8 node cluster doing exactly that, where my SSD's were on-board the server and the HDD were in a SAS JBOD.

if you do connect multiple hosts (up to 4 - depends on enclosure) to multiple enclosures you can just run clustered storage spaces. I had a cluster like that for a while (minus the Hyper-V) and it worked well.

Chris
 

gregsachs

Active Member
Aug 14, 2018
559
192
43
As long as you do not connect the JBOD enclosures to multiple servers. Sure S2D will work that way. I had an 8 node cluster doing exactly that, where my SSD's were on-board the server and the HDD were in a SAS JBOD.

if you do connect multiple hosts (up to 4 - depends on enclosure) to multiple enclosures you can just run clustered storage spaces. I had a cluster like that for a while (minus the Hyper-V) and it worked well.

Chris
Whoops, yeah clustered was what I was thinking first, simpler than s2d.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Just an update to this. I ended up buying a larger qnap NAS server for the backend of this. It has 2tb nvme cache, 4tb ssd tiering and 18tb of total useable space. With the purchased 10gbe qnap card I was able to get all of the RDMA/ROCE stuff working pretty well.

I got all of it working (eventually) including the RDMA part with the qnap which ended up being WAY harder than I think it should have been. Its all running really fast and I must say the QNAP with all of the tiering is outperforming my expectations. I was able to learn a lot with this build as well on networking in general and that helped my overall knowledge.

Business has been good however, and now I need to expand the lab as performance testing is pushing the current 3 machine setup, so i'm thinking of making lab2 entirely with S2D instead to simplify things. Will post more later on a new thread to get advice.

thanks to everyone who gave me really good advice and helped me achieve success with this build
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Just to sew this up. This is the qnap I bought, "QNAP 12 Bay NAS/iSCSI IP-SAN, Intel Skylake Core i5 3.6GHz Quad Core, 16GB RAM, 10G-Ready (TVS-1282-i5-16G-US)" and the 10g nic for it "QNAP QXG-10G1T Single-Port (10Gbase-T) 10GbE ".

I ran a dedicated storage network over the netgear smart switch with dedicated ports.

Can give more info if anyone wants it.
 
  • Like
Reactions: TomUK