HA SAN build recommendations
- Thread starter legen
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
How is SAS intended to be expended when using external enclosures?
The JBOD chassi is Supermicro | Products | Chassis | 3U | SC837E26-RJBOD1
Now if we want to scale this with another enclosure do i add yet another LSI 9207-8e in each machine (if i dont want to use the free SFF-8088 port on the first LSI card) like this:
The other way seems to be using sas chaining but that would seriously limit the performance since one SFF-8088 port only have 4x6Gbit/s?
Using the JBOD chassi above i can only connect to it using one SFF-8088 port thus from each server node i am limited to 4x6Gbit/s (or 3GByte/s) to the array. Using 24x7200rpm spinners (each with 180MB/s) in a 12 vdev setup i have a theoretical max of 2160MB/s.
So in my case the arrays theoretical max of 2160MB/s is below the SFF-8088 max of 3 GByte/s. Sounds correct?
The JBOD chassi is Supermicro | Products | Chassis | 3U | SC837E26-RJBOD1
Now if we want to scale this with another enclosure do i add yet another LSI 9207-8e in each machine (if i dont want to use the free SFF-8088 port on the first LSI card) like this:
The other way seems to be using sas chaining but that would seriously limit the performance since one SFF-8088 port only have 4x6Gbit/s?
Using the JBOD chassi above i can only connect to it using one SFF-8088 port thus from each server node i am limited to 4x6Gbit/s (or 3GByte/s) to the array. Using 24x7200rpm spinners (each with 180MB/s) in a 12 vdev setup i have a theoretical max of 2160MB/s.
So in my case the arrays theoretical max of 2160MB/s is below the SFF-8088 max of 3 GByte/s. Sounds correct?
That certainly looks like it would work. I would think the main reason to not use the secondary port on the existing card would be if you're overrunning the PCI-e capability of that single card/slot.
Certainly on paper, you're getting 50% less performance. But in the real world, I suspect you're not going to see that as a 'limit'. Your original post suggested you're currently on 1GbE NFS, even 2x10GbE NFS would still be less than 4x6Gbit.
Aren't most 7.2K SATA disks 3Gbit/sec not 6Gbit/sec like SAS? So you'll be at 4x3Gbit? And right now you're running 8xM500 SSD's but are considering going to 24x7.2K HDD?
My general (legitimate) questions, as I'm curious how this build works out:
* The current picture shows 1 JBOD being shared by two head units - are the 7.2K Spinners going to be dual ported NL-SAS? Can this be done with single ported SATA disks, I wouldn't think so.
* You mention 24 disks, but a 28 disk JBOD - so assuming 4x SSD to handle the caching? I just didn't see the SSD's mentioned. Same dual port question applies.
* Your original post suggests you're using this for 1GbE NFS, and it's shared to 10 XenServer hosts, running ~ 20 VM's (really 1-2 VM's per host? That seems like horrible density...). But you point out your desired speeds in sequential MB/sec throughput - which has nothing really to do with running VM's. Largely you're probably going to care a ton more about IOPS and Latency of 4K'ish workloads for the virtual machines, unless you're streaming large data, which is probably never going to be sequential on a shared SAN.
* Does your $15K budget include just the SAN/storage, or does it also include a bump up to 10GbE on the hosts, and SAN, and switching? Even with home lab pricing and equipment, the NIC's, Cabling, and Switches are likely to eat up ~ $5000+ of that (~$200 2x10GbE NIC's x 12, with 2x cabling each, 2x 24x10GbE switching)
* If you're providing 2x10GbE out to the hosts (and can you do the full 20 or will you be doing 10 with failover?) should you be worried about theoretical limits of 12 or 24 or 48Gbit/sec to the JBOD's, when you're not able to push that much out through the front hole to network? Granted, internal SAN tasks don't traverse that, but if you're honestly running at peak that often, you need to build for a size up, no?
I'm speaking all of this from a VMware background, and where I use storage appliances (eg: NetApp, Nimble, etc) vs build my own. So I have an honest curiosity in some of the above answers and outcomes, as I'm not sure I know the answers.
I think the biggest one I have is the "why are you so focused on sequential read/write, when that's likely _never_ going to be what you actually need?".
Hi, Thank you for your detailed response. We will be going with 10GbE for now but may move to 40GbE using switches that have 24x 10GbE ports and 2x 40GbE ports. In that scenario the bottleneck would occur in the SAS chain.
We are looking at 7200rpm SAS drives since we require HA capabilities for this build. sadly SAS SSDs are to expensive so the only route forward is hdds in a shared JBOD enclosure (for OmniOS+RSF-1, Solaris+RSF-1, Nexentastor, Quantastor, Microsoft Storage Spaces)
Yes they are dual-ported SAS drives (i.e. Seagate Constellation SAS or Ultrastar 7K6000 SAS).
Yes we use 24 drives in a raid10 config (12 vdevs each using 2 disk mirror). We use 2 extra as hot-spares and 2 slots for mirrored zeusRam (a total of 28 3.5'' slots).
Microsoft Storage spaces would indeed require another setup with SSDs for tiering (maybe 4 or so).
Yes the density is low due to licensing issues for the software running on the VMs (we prefere few VMs with much resources in favor of many VMs with less resources available).
I agree (and know) IOPS is the crucial thing here which we are trying to accommodate through our use of 12 vdevs (raid10). I am interessted in the sequential performance since i do not know how/if a bottleneck here will also impact the IOPS in a negative way. If i manage to have no bottlenecks i know it wont affect the IOPS (i.e. during high sequential loads).
The budget was increased to ~25 000$
We realized this was impossible with the previous budget. We have a separate budget for switches. If we manage to get this SAN build for 20 000$ we will have 5000$ extra for the switches which would be nice.I have not yet picked the switches for this so i do not know if we can do active/active or active/passive :/
If we go with the 40 GbE switch we will have a problem with the theoretical limit of the SAS 6.0 port.
I agree that we might not have to dimension our storage for the peaks. However our situation now is that we invest a quite large sum for the SAN which we hope will last us the next 5 years (only expand with more storage during that period). Thus yes today we do not see this traffic, in 5 years we do not know so better to build big if its possible
Do you have some recommendations on how i should focus my work? Using the 12 vdev raid-10 array + zeusRAM i think i have maximized the IOPS, but again i do not know how the IOPS will be affected if we are bottlenecked during sequential throughput.
Thanks!
So I'm just one guy, doing it one way, I'm sure YMMV. But...
- I just about NEVER see anyone maxing out 10GbE for ISCSI/NFS, even big shops. Maybe during a boot storm or "patch day" or something, but even then it's never maxxed out. I don't know you'll ever 'realistically' see 10GbE or 40GbE be your bottleneck. Spend more money on SSD/Caching.
- VM's seldom do anything that looks like sequential IO - I'm not sure still why you're focused on that. Unless you're cloning a VM or backing up the entire thing (both of which the hypervisor usually throttles to the preference of VM based IO) and not using differential/CBT type solutions, there's no reason you'd really need sequential IO for anything, so designing to hit that is usually the wrong answer. I see this typically in folks who run an Atto style benchmark and go for the maximum 2MB block size numbers, completely ignoring the 4-16KB block range where they actually operate 99% of the time.
- While I'm not a ZFS guy and I'm not sure how the cache plays in, 12 mirrors of 7.2K drives is still only going to yield a maximum of around 1920 IOPS (80 IOPSx24) - and the mirroring will eat up half of that. So you're building all of this for ~ 1000 IOPS based on disk - or 100 IOPS/host. That's giving each host the equivalent of a desktop SATA disk of performance (assuming they're all balanced, all the time, etc)
If your workload is 4KB blocks (you don't mention, I'm just making an assumption) then you're going to get about 4MB/sec 4K Random throughput or about 500mbit/sec - half of 1GbE. Your problem won't be 6 or 12Gbit SAS or 10/40GbE any time soon. It looks like there is a long way to go before you even max out multi-NIC 1GbE on the SAN side. Especially if you have say 10x hosts with 2x1GbE and 2x SAN head units with also 2x1GbE - that'll be your choke point. Adding another 4x1GbE NIC to the SAN would probably help more than changing the infrastructure to 10GbE across the board.
Do you know what your workload profile looks like? IS it all streaming/sequential, even though it's 20 VM's on 10 hosts, virtualized, on a common SAN in an "IO blender"? Even if the VM's _are_ sequential, is that what the SAN sees/feels?
Could anyone enlighten me on the more ZFS focused aspect of this, and if it changes the theory much? As stated before I don't typically roll my own storage, but the concepts seem similar. It seems like this is being built for capacity/throughput and not performance?
- I just about NEVER see anyone maxing out 10GbE for ISCSI/NFS, even big shops. Maybe during a boot storm or "patch day" or something, but even then it's never maxxed out. I don't know you'll ever 'realistically' see 10GbE or 40GbE be your bottleneck. Spend more money on SSD/Caching.
- VM's seldom do anything that looks like sequential IO - I'm not sure still why you're focused on that. Unless you're cloning a VM or backing up the entire thing (both of which the hypervisor usually throttles to the preference of VM based IO) and not using differential/CBT type solutions, there's no reason you'd really need sequential IO for anything, so designing to hit that is usually the wrong answer. I see this typically in folks who run an Atto style benchmark and go for the maximum 2MB block size numbers, completely ignoring the 4-16KB block range where they actually operate 99% of the time.
- While I'm not a ZFS guy and I'm not sure how the cache plays in, 12 mirrors of 7.2K drives is still only going to yield a maximum of around 1920 IOPS (80 IOPSx24) - and the mirroring will eat up half of that. So you're building all of this for ~ 1000 IOPS based on disk - or 100 IOPS/host. That's giving each host the equivalent of a desktop SATA disk of performance (assuming they're all balanced, all the time, etc)
If your workload is 4KB blocks (you don't mention, I'm just making an assumption) then you're going to get about 4MB/sec 4K Random throughput or about 500mbit/sec - half of 1GbE. Your problem won't be 6 or 12Gbit SAS or 10/40GbE any time soon. It looks like there is a long way to go before you even max out multi-NIC 1GbE on the SAN side. Especially if you have say 10x hosts with 2x1GbE and 2x SAN head units with also 2x1GbE - that'll be your choke point. Adding another 4x1GbE NIC to the SAN would probably help more than changing the infrastructure to 10GbE across the board.
Do you know what your workload profile looks like? IS it all streaming/sequential, even though it's 20 VM's on 10 hosts, virtualized, on a common SAN in an "IO blender"? Even if the VM's _are_ sequential, is that what the SAN sees/feels?
Could anyone enlighten me on the more ZFS focused aspect of this, and if it changes the theory much? As stated before I don't typically roll my own storage, but the concepts seem similar. It seems like this is being built for capacity/throughput and not performance?
I belive a ZFS based solution without an SSD logdevice will give you terrible write performance , unless you configure it with sync=disabled . But if you go with sync=disabled the HA will not work because if you have a failover for any reason the cached write data in memory not commited to disk yet will be gone. so the VMs will not notice because the NFS volume is still available but up to 5 seconds of data is gone (= a bad day).
Have you only looked at the ZeusRam SSD's ? Because one nice feature with Solaris 11.2 is the ability to do parallel writes to multiple log devices , if you need more write iops just add another SSD and that makes it possible to use cheaper SSD's (E-bay FTW !
) . and also from Solaris SRU 11.2.8.4.0 the L2ARC is persistent and survives reboot. We have been using the Cisco Nexus 5010 as 10Gbit ToR switches for 5 years now and i love them, rock solid . so if i would build a low cost environment now i would buy used Nexus 5010's ( or 5020's ) from E-bay (have seen them as low as 1200 EUR).The budget was increased to ~25 000$
Sounds like a good idea.
You can check geas post on page 2 about how the ZIL works. Basically ZFS will cache 5 sec of writes in RAM thus transforming the writes from small random writes to sequential writes. The only thing that should be hitting my disk array with a dedicated ZIL would be the cached writes that exist in RAM
I agree that the performance of the array will be ~1000 IOPS with my given raid configuration. But the boost you gain from caching the writes in RAM will boost this (hard to get any numbers on how much though!). The reads will also be served >90% from the read cache so the idea here is that not that many small random read/writes will hit the array in the end.
I am unsure about the workload block size distribution. I will try to find some DTRACE scripts that can show me this. Then i will sample for a day or two. I can use zpool iostat for iops and sequential bandwidth but need to find something that shows be the block size hitting my current SAN.
Again here i am unsure how ZFS caching will affect 4K random throughput but in theory it will cache 5 sec of it in RAM and then write it sequentially to the disk array (which should improve performance alot avoiding small random writes!)
Let me get back to you on this when i have found some way of measuring the workload distribution in omniOs
Typically know my workload is copying files and directories, un-zipping files. But i think real numbers from the current SAN would show this in real numbers in a much better way.
EDIT: I might add a question. What would you say is a good solution? I am cost-wise limited to HDDs when going the SAS shared JBOD way which is required by MS storage spaces, Zfs+RSF-1, Quantastor and nexentastor.
The only other alternative (if not using non-supported SATA SSDs with interposers in the JBOD) would be to build two nodes each with its own SATA SSD storage. Problem here is that the only HA solution that supports this is Zetavault.
Ah yes. We will require sync writes and with the zeusRAM this should not be an issue. Good point though that HA will require the ZIL and sync=always (which is the default from the xenserver NFS mounts).
We did actually look into several other but the zeusRam is in a league of its ownHave you only looked at the ZeusRam SSD's ? Because one nice feature with Solaris 11.2 is the ability to do parallel writes to multiple log devices , if you need more write iops just add another SSD and that makes it possible to use cheaper SSD's (E-bay FTW !
. Its a shame that Solaris is now closed since it gets a lot of new nice features.I will keep the door open for turning this into a Solaris+RSF-1 solution. Will probably start with OmniOS since it is 100% free but its easy to import/export the pool if we decide to change
Thanks for the tip. Will put the nexus on the list of possible candidates!
"EDIT: I might add a question. What would you say is a good solution? I am cost-wise limited to HDDs when going the SAS shared JBOD way which is required by MS storage spaces, Zfs+RSF-1, Quantastor and nexentastor."
(Damned phone client...)
So again my 'not a ZFS guy' disclaimer...
It doesn't make a lot of sense to plan for 10gbe and then put on the other end shared storage with an expectation of 1000 IOPS. Yeah, the ZIL will help but it's pretty small. That 'few seconds of writes' may cache but what happens when you have minutes and minutes of steady activity? Seconds of cache won't matter. Your current specs sound like they'd feel a lot like the NetApp FAS2040 I have in the lap - 12 SATA or SAS disks and 2Gb of memory. Doesn't take much to overwhelm that cache and start being disk bound.
You mention being budget limited to HDD vs SSD. I did a search on eBay for enterprise SAS SSD as I couldn't find a quick post here on the forums for a current deal, and I'm finding 400gb SAS SLC Enterprise stuff for $400-600 CAD. Cheaper if I'm picky or careful. I suspect you can fit the SAS SSD in the budget?
(Damned phone client...)
So again my 'not a ZFS guy' disclaimer...
It doesn't make a lot of sense to plan for 10gbe and then put on the other end shared storage with an expectation of 1000 IOPS. Yeah, the ZIL will help but it's pretty small. That 'few seconds of writes' may cache but what happens when you have minutes and minutes of steady activity? Seconds of cache won't matter. Your current specs sound like they'd feel a lot like the NetApp FAS2040 I have in the lap - 12 SATA or SAS disks and 2Gb of memory. Doesn't take much to overwhelm that cache and start being disk bound.
You mention being budget limited to HDD vs SSD. I did a search on eBay for enterprise SAS SSD as I couldn't find a quick post here on the forums for a current deal, and I'm finding 400gb SAS SLC Enterprise stuff for $400-600 CAD. Cheaper if I'm picky or careful. I suspect you can fit the SAS SSD in the budget?
Yeah, I was trying to explain this to a friend the other day. I'm willing to live (if needed) with sync=disabled in my current setup if I need better performance (the s3700 SLOG device limits me to 180MB/sec writes over 10gbe, whereas sync=disabled can hit 500MB/sec). But I am not in the HA space at this time. The idea that you could silently lose several seconds of writes (which might very well be client filesystem metadata - e.g. corruption anybody???) is a complete showstopper for me.
p.s. interestingly, the SSD itself doesn't seem to be the choke point. I even tried creating a 4GB ramdisk and saw the same performance. Someone mentioned that this was likely some internal ZIL throttling or whatever, but he wasn't very specific, and I had no idea what to look into, so I dropped it...
I think i have to explain how the zil and zfs handles writes in more detail (how i understand it). If we saturate the 10GbE link for a long time we have a constant rate of 1.25GByte/s cached in RAM and to the 8GB ZeusRAM. Every 5 seconds this data is written sequentially to disk. Even lots of small random writes will go to RAM and be limited only by the zeusRAM (which has very fast writes).
That gives us 10Gbit/s = 1.25Gbyte/s. 1.25Gbyte/s * 5 seconds = 6.25GByte of data (fits within the Zeusram zil). Now this data needs be written to the array before the next 5 seconds with new data arrive. Thus the array needs to be able to write at a minimum 1.25GByte/s sequentially.
Using 12 vdevs we get the sequential write performance of a single disk times 12. I.e. 180MB/s * 12 = 2160 MB/s
This shows that we can saturate the 10GbE link indefinitely without any dip in performance (in theory)
SAS SSDs from ebay might be a way forward i agree. The problem here is that it might be hard finding 12-16 similar SAS SSD drives to use, finding replacements when they break and stuff like that :/
EDIT: On ebay i think the Smart Storage Optimus 920GB Enterprise SAS drive would be the best option if we aim to get to 10TB usable storage (would require ~14 drives if using raid-z2).
Last edited:
You are absolutely correct and i overlooked the fact that it is capped to 400MB/s Single-port and 800MB/S dual-port.
I guess the dual port speed only applies if one has two active/active head nodes and not active/passive?
Sorry i misunderstood your earlier post when you said SAS SSD's are to expensive , i thought you meant no SAS SSD's at all , not even for ZILAh yes. We will require sync writes and with the zeusRAM this should not be an issue. Good point though that HA will require the ZIL and sync=always (which is the default from the xenserver NFS mounts)
We did actually look into several other but the zeusRam is in a league of its own
Have any one tried these HP Optimus SAS drives?
Smart Storage Optimus 920GB Enterprise SSD SAS 2 5" TXA2D20920GA60HP | eBay
They have HP part numbers and are alot cheaper than the other non-hp optimus drives on ebay.
Smart Storage Optimus 920GB Enterprise SSD SAS 2 5" TXA2D20920GA60HP | eBay
They have HP part numbers and are alot cheaper than the other non-hp optimus drives on ebay.
Hey everyone! Time for an update
We have started moving forward with the shared enclosure solution using 2 head-nodes and a JBOD.
My budget allows me to use the new 44 bay SAS 3 supermicro chassi (Supermicro | Products | Chassis | 4U | CSE-847E2C-R1K28JBOD
Like before we will use ZeusRAM for zil.
With this i will use Seagate Constellation 1TB HDD drives. Would it be a bad idea to run 36 of these in raid10 with 3 for hot-spares?
That would increase my iops using 18 vdevs to about 1800 IOPS.
I have not seen that many people use that many drives in a single raid10, so is it a bad thing to go above 24 drives in a raid10 (with 3 hot-spares)?
The software we will be evaluating are: Open-E JovianDSS, OmniOs+RSF-1, Solaris 11+RSF-1, Quantastor and Nexentastor. JovianDSS has a very nice license so that will be the first we try.
We have started moving forward with the shared enclosure solution using 2 head-nodes and a JBOD.
My budget allows me to use the new 44 bay SAS 3 supermicro chassi (Supermicro | Products | Chassis | 4U | CSE-847E2C-R1K28JBOD
Like before we will use ZeusRAM for zil.
With this i will use Seagate Constellation 1TB HDD drives. Would it be a bad idea to run 36 of these in raid10 with 3 for hot-spares?
That would increase my iops using 18 vdevs to about 1800 IOPS.
I have not seen that many people use that many drives in a single raid10, so is it a bad thing to go above 24 drives in a raid10 (with 3 hot-spares)?
The software we will be evaluating are: Open-E JovianDSS, OmniOs+RSF-1, Solaris 11+RSF-1, Quantastor and Nexentastor. JovianDSS has a very nice license so that will be the first we try.
We still "only" go with 2 ZeusRAM drives. We will do some experiments with not using sync writes on some workloads and try to "only" force i.e. database VMs to use the ZIL. Will also try the performance using them in mirror vs raid-0.
The other option i was looking at was the s840z 16GB BUT It looks like it has reached end-of life and is no longer sold by HGST
!Shame since this makes the zeusRAM the only option if one needs SAS.....
I see i'm a little too late, I recently went through something similar with a rebuild from a homebrewed mess to something a bit more manageable, we moved to a thinkserver RD650(which was reviewed here recently) and I've seen impressive performance for it although we didn't go for HA which would require a second identical unit, I have to say i'm very impressed with cachecade using a pair of the "lenovo read intensive ssd's"(they're just intel ssd dc3500's I'd have liked faster units but i couldn't just buy empty trays) the writes are a bit short due to the read intensive drives and use of RAID6 for the rest of the storage but faster SSD's would certainly remedy that and storage spaces has been mentioned for HA already, I'm fairly impressed with WS2012R2 as a storage platform especially the data deduplication