Best disk requirements for a ZFS setup excluding actual shared storage.

[RimBlock]

So I am looking to turn one of my C6100 nodes in to a ZFS server.

I have storage connectivity via a 16 by DAS (8 Drives - SAS2 - No expanders & 16 drives - SAS2 - expander with 8 channels to 16 drives).

Not sure what to use as a boot disk. Any real benefit from using a SDD and if so how big ?. If not then I will probably use a WD Blue 320GB I have knocking around.

I will probably use 2x Intel 520 60GB SSDs for ZIL. Sound reasonable ?. (max bandwidth is 2.5GB/s *10 seconds = 25GB requirement).

L2ARC will reside on a Intel 250 120GB drive. Is there a way of disabling caching for various disk pools. One pool is likely to be made up of multiple SSDs so the L2ARC could well be slower.

Anything else to consider before I start buying ?.

Thanks
RB

 

[wyluliraven]

You hit the major notes correctly, but just to answer you. Yes, you want to use an SSD for the boot drive. However for your ZIL drive, you should look at SLC SSD, not MLC like the Intel 520 is. The ZIL is not a very large file, but gets absolutely hammered, so the longevity of SLC is a noted benefit here.

 

[PigLover]

Why do you propose SSD boot for a ZFS server? Once booted the rpool is pretty much never touched. Swap isn't an issue either because ZFS will only use RAM for the ARC. Unless there are other things running on it I disagree - there is absolutely no reason to use SSD for the boot drive on a ZFS server.

You DO want to use two drives for boot. While you don't need speed, it is a royal PITA to reload and recover all your settings if the boot disk dies. By running a mirror you can have one fail and your still all good. Its far, far easier to replace a drive and resilver than it is to do a reload. Google mirrored rpool boot for instructions to set up.

 

[cactus]

If you have spare drives, there is no use to buy SSDs for boot. If you are getting everything new, why not go with two 32GB SSDs(or any cheap SSDs)? They are basically equal price and you save 10W idle power.

 

[PigLover]

Fair play to that - always SSD if the price is close. In fact, we are not far from the day that 2.5" spinning drives will be a thing of the past...Seagate announced this week they are dropping their 7200RPM 2.5" line - and the slower ones won't be around much longer.

I was really just questioning the idea that there was a need for - or even any good reason for - preferring SSD for the boot drive of an ZFS server. There isn't.

 

[cactus]

I cant think of another reason for SSD boot besides power. It would be nice to be able to boot from RAIDZ for disk limited system like the HP MicroServer, but I guess gea's NappIt To Go is a good solution there.

 

[nitrobass24]

Might want to use SSDs that have Supercaps for your ZIL, so you dont get hosed if the power goes out.

 

[RimBlock]

THanks for the replies.

So for the Zil I may look at getting a couple of Intel 313 24GB SLC drives. Any others worth looking at ?.

For boot I have a couple of Scorpio Black 320GB drives or a Barracuda ES 500GB (will get a second).

RB

 

[nitrobass24]

Intel 320s are the cheapest out there with super caps

 

[cactus]

I would skip the 311/313 because it lacks a supercap. ZFS sends a cache flush to make sure you dont get into problems, but I have read about SSDs, especially consumer level drives, ignoring the flush command sent by the OS. I believe this is for your own lab, so an Intel 320 mirror with a lot of over prevision seems like the best bet for slog. For prod stuff, stec zeusram($$$).

Edit: nitrobass beat me to it.

 

[RimBlock]

Just remembered I have the two Vertex II which I can use for a mirrored boot drive.

I will look for a couple of 80GB 320s then for the ZIL.

An Agility 3 for the L2ARC.

Thanks guys.

 

[RimBlock]

Another question.

More ram and bigger L2ARC or a SSD ZFS array ?.

The L2ARC will benefit any array but if flushed will take time to build up and if something is not in there then slowdowns may occur.

I am thinking (please correct me if needed);
Movie storage - HDD
Minecraft server (rapid multiuser reads and writes) - SSD array (or part of).
Database partition - HDD

Just trying to work out where to place the cash for the best effect. I do already have 4x Intel 520 120GB drives (big enough for a ZFS array with one drive redundancy, for now).

I also see the Samsung 840 500GB ssds are also quite cheap @ US$299 each.

 

[Biren78]

What about an Intel 710 100GB or S3700?

 

[cactus]

Another question.

More ram and bigger L2ARC or a SSD ZFS array ?.

The L2ARC will benefit any array but if flushed will take time to build up and if something is not in there then slowdowns may occur.

I am thinking (please correct me if needed);
Movie storage - HDD
Minecraft server (rapid multiuser reads and writes) - SSD array (or part of).
Database partition - HDD

Just trying to work out where to place the cash for the best effect. I do already have 4x Intel 520 120GB drives (big enough for a ZFS array with one drive redundancy, for now).

I also see the Samsung 840 500GB ssds are also quite cheap @ US$299 each.

How much space does minecraft take up?

 

[cactus]

What about an Intel 710 100GB or S3700?

Both are good options in my opinion, but they are 2x or more the cost or the 320.

 

[Patrick]

Both are good options in my opinion, but they are 2x or more the cost or the 320.

You can get 710's for ~$120 if you have patience.

 

[RimBlock]

How much space does minecraft take up?

Not big at this point by any means. Under 1GB. Is the L2ARC purely for read ?. If so I would be slightly concerned with the updates not being cached fast enough compared to using a direct SSD. Minecraft is not a major requirement but is an ok example of a multiuser environment that can stress and benefit from storage configurations. One players updates need to be available to all other players fairly quickly or their is potential for glitches to occur.

If I end up going for full mechanical hard drives and a large ssd L2ARC, is there any mileage in raiding some SSDs (I still have these 4x Intel 520s which it would be cheaper to utilise than sell and replace).

I may use one locally for each ESXi server though for boot / swap space although it really is overkill IMO.

RB

 

[RimBlock]

You can get 710's for ~$120 if you have patience.

Unfortunately in my case I also have to make sure the seller is willing to ship international and it is getting quite frustrating with the two sellers I am dealing with on EBay for the two C6100s and extras I am ordering with us only being able to pass 1 message per weekday due to the 12 hour time difference. It is doubly frustrating when some of my questions aren't even answered...

The other issue with buying on EBay is that you have no initial idea on how thrashed the drives may have been.

The 710 is also only SATA II. The S3700 looks interesting but current EBay pricing is on par with local distributor pricing here. It is also around US75 more than the 320s.

I may put a stopgap solution in place for the short term and then reassess when cash becomes available. This is only a test system after all.

RB