Which SLOG (ZIL) to improve our sync writes?

[legen]

Hello everyone!

We are running ZFS on OmniOS with 8xM500 in a raid-z2 setup. We have 8 (soon 16) xenserver machines accessing the SAN over NFS. We have sync=always thus all our writes to the VM datastores are synchronous.

We now need to improve our sync-write speed (we only get about 130 MB/s or so currently). Settings sync=disabled gives us better write speeds (1.2GB/s).

I have tried to read up on what to use for a ZIL. I have found the following alternatives,

• ZeusRAM, seems to be a great device for it, http://www.napp-it.org/doc/manuals/benchmarks.pdf.
  
  
• s840z, a 2.5’’ zil alternative s840Z 2.5″ ZIL SAS SSD | HGST Storage
  
  
• DDRdrive?
  
  
• Marvell WAM module?
  
  
• Intel S3700 or S3500

I have looked mostly at zeusRAM, s840z and intel SSDs. We will soon check how to build a HA cluster with another OmniOS machine (RSF-1 probably but have not looked into that yet). Our cluster-setup will probably use a shared SAS array where the ZIL should be shared between both servers, thus I think it is better to use a SSD device than a PCI device.

The ZIL should sustain our writes for 10 seconds (have not checked our zfs_txg_timeout since im on the road , source Nex7's Blog: ZFS Intent Log)

We hope to buy 40Gbit/s infiniband for storage traffic soon to. But currently its 10Gbit ethernet. Ideal this means we need to have a ZIL of 25-50 GB or so in the future. I doubt we will have that much heavy writes but in theory that is our wanted ZIL size…..

Now the s840z specification indicate a higher write latency, 39μs, than the zeusRAM (>23μs). But the s840z is twice the size and half the price. Have anyone tested it?
It seems to be a better alternative than the intel SSDs (65μs).

What do you guys think?

 

[Patrick]

I have not tested that drive yet. I did want to try getting a test setup ready to do this kind of testing on SSDs.

 

[Biren78]

What about this? Fusion IO 160GB SSD SLC PCI E Solid State Drive Free Shipping | eBay

SLC PCIE SSDs

 

[legen]

What about this? Fusion IO 160GB SSD SLC PCI E Solid State Drive Free Shipping | eBay

SLC PCIE SSDs

Looks to be a great drive. But i think i need a SAS device that can be shared between two omnios boxes for ha

 

[bds1904]

ZeusRAM is the traditional solution. It will give you sustained performance over the entire write until the ZIL is near full. Very low latency. As far as size, if you need more than 8GB ZIL then you also need to focus on the sequential write speed of your pool. In your case, I think you'll be fine.

When it comes to ZIL latency is everything. Be sure to go by latency before speed figures.

Generally once you start maxing out a single ZIL setup you need to start adding other pools with additional ZIL's. You may need to start thinking about using 2 pools and 4 ZIL's.

 

[gea]

I own two DRAM based ZeusRAM with 10 GbE in my SSD pools and they are perfect.
Main problem for me: I use more and more 2,5" enclosures from SM and ZeusRAM is 3,5".

I use also some high-performance Winkom Sata 32 GB SLC SSDs in a special powercap edition and some Intel 3700-100 SSDs, a quit good and cheaper option- but the s840z seems a good but not cheap alternative in SAS.

With SAS I would first try the s840z. With 10 Gbit/s=1GB/s= about 10 GB in 10s - one 8GB ZeusRAM is ok.
With 40Gb, you would need 4-5 x ZeusRAM or 2-3 s840 or 1 x Sata Intel S3700-100/200

No SSD can compete against DRAM based ZIL but the price difference is heavy.

Please keep us updated.

 

[legen]

ZeusRAM is the traditional solution. It will give you sustained performance over the entire write until the ZIL is near full. Very low latency. As far as size, if you need more than 8GB ZIL then you also need to focus on the sequential write speed of your pool. In your case, I think you'll be fine.

When it comes to ZIL latency is everything. Be sure to go by latency before speed figures.

Generally once you start maxing out a single ZIL setup you need to start adding other pools with additional ZIL's. You may need to start thinking about using 2 pools and 4 ZIL's.

Yes our current sequential write speed is more than 1 single ZeusRam can output (900MB/s or 475MB/s according to the datasheet). But i dont expect heavy writes to last long enough to saturate 1 zeusram (not in the near future)

I own two DRAM based ZeusRAM with 10 GbE in my SSD pools and they are perfect.
Main problem for me: I use more and more 2,5" enclosures from SM and ZeusRAM is 3,5".

I use also some high-performance Winkom Sata 32 GB SLC SSDs in a special powercap edition and some Intel 3700-100 SSDs, a quit good and cheaper option- but the s840z seems a good but not cheap alternative in SAS.

With SAS I would first try the s840z. With 10 Gbit/s=1GB/s= about 10 GB in 10s - one 8GB ZeusRAM is ok.
With 40Gb, you would need 4-5 x ZeusRAM or 2-3 s840 or 1 x Sata Intel S3700-100/200

No SSD can compete against DRAM based ZIL but the price difference is heavy.

Please keep us updated.

We have the same problem. Our current case is 2.5'' so the ZeusRAM wont be relevant until we have decided on our future storage (san HA, SAS vs SATA, SSD vs spindles etc etc etc...)

So in summary i think we will start looking at the s840z. And if we can wait (using sync=disabled for a while) we can get a zeusRam or so instead if we decide to invest in 3.5'' enclosures.

 

[T_Minus]

What did you end up doing?

What's your performance now?

 

[T_Minus]

@legen any updates? I see you're around!! Update us pleeeeeez

My ZuesRAM is coming this week,!!

 

[whitey]

@legen any updates? I see you're around!! Update us pleeeeeez

My ZuesRAM is coming this week,!!

Sniffs...I miss her already! Ohh btw if you see a big fat kiss on the package that was me giving her a parting kiss goodbye one last time before I pitched her in the mail to you hahahah j/k!

 

[T_Minus]

 

[legen]

@legen any updates? I see you're around!! Update us pleeeeeez

My ZuesRAM is coming this week,!!

I apologize for not being that active on the forum I have been swamped with ordinary work and the server project we started nearly 1 year ago now. It has finally started to give results which (to me) is very exiting

We actually postponed this and simply turned of sync writes for the time being. We prioritized to get our services online in favor of this for the time beeing.

I was actually just about to create a new thread asking for SAN recommendations. We have actually gotten a budget where we will try to build a HA SAN setup (if we go with ZFS zeusRam will be the number one choice).

Would love to see some performance benchmarks when yours arrive