Looking for others to help up vote a general purpose write-caching method for TrueNAS.
Log in with Atlassian account
Log in with Atlassian account
These really have limited effect, as you pay a lot of performance penalty for the safety of ZFS.Anyway, into the blue, I'd say this is pointless. There are already several effective methods in place that speed up writes and some of them are cache or cache-like / cache-adjacent methods.
Writes are already collected in transaction groups before they're written to disk and that happens in RAM. Data is collected until the TXG is too big or the timeout is reached. You can tune these settings to cache more dirty data in RAM if you like.
Well, if someone made the decision to use ZFS, they know about its properties and probably made it a deliberate choice. It's not as if ZFS is inherently slow, it's just slower than some alternatives but it all depends on your setup. And you say that TXGs have limited effect, but that's why I said you can tune those to whatever you think is appropriate. I don't know if there are any hard caps, but the default TXG timeout is 5 seconds and you can increase this. Some people use timeouts in the range of several minutes, which isn't necessarily the intention of ZFS, but if this fits your use case, you can do it. It will allow you to keep minutes worth of changes in RAM, basically caching everything at RAM speeds, if it fits in your RAM and doesn't exceed the TXG size limit.These really have limited effect, as you pay a lot of performance penalty for the safety of ZFS.
I'll have to read up again, but when I checked it seemed like *persistent* L2Arc (newish to TrueNAS) behaves differently and ends up being a frequency based read cache; any read cache is only as good as the cachehit ratio, and we do lose the idea of a HOT tier (recently written data being automatically in the read cache by nature. Not flawless but also fully "within" ZFS. Recent changes reduce the RAM needed for persistent L2arc as well.Be aware
- With enough RAM, L2Arc only helps when you reboot quite often and you have many users with many volatile files
- For a pure filer, sync write is not needed and only lowers write performance without gain.
- A special vdev is not a cache but the only place where this data is stored. Unlike L2Arc or Slog, a special vdev failure is a pool lost.
The safest and cheapest way to "long term archive" a "massive amount of data" is tape. If you actually need spinning-rust speed access to the data on a regular basis, then it's not really archive.ZFS was made (and is still intended for) hosting a massive amount of data which is important to be safe. Think long term archival of critical business data (in EU company’s have to store their business files for 10 years) Mostly on spinning rust.
Yes, if you truly ingest 24/7 at faster than the pool write speed, then you will eventually slow to that. In that case, though, you designed your system incorrectly.And the end of the day, every kind of cache is only a temporary solution. If your pool isn't fast enough and you have a constant stream of new data and changes, your pool will never catch up and when your cache is saturated, performance will drop down to pool speeds. No cache in the world will prevent this, you can't have an arbitrarily slow pool and think that caching will fix this for you.
Oh, absolutely. But following your example, it's not just ingesting 24/7 and then eventually hit the limit at some point. It could be as little as minutes, maybe even seconds, until you've reached the point. Or never achieve expected speeds in the first place. Incorrect design and incorrect expectations happen very frequently.Yes, if you truly ingest 24/7 at faster than the pool write speed, then you will eventually slow to that. In that case, though, you designed your system incorrectly.
Again, there are many datacenter NVMe drives than can sustain close to 2GB/sec for the entire size of the drive. A 3.8TB drive could handle that speed for over 30 minutes. This is a trivial solution for up to 25Gbps on the ingest network.It could be as little as minutes, maybe even seconds, until you've reached the point.