SLOG/L2ARC in Ubuntu w/Docker File-/Media Server

norsemangrey

New Member
Apr 27, 2021
4
0
1
Hello experts. I'm have recently built a server to be used as file-/media server for myself and a few family members. It will be running on Ubuntu Server with ZFS. I will be using Docker to run quite a few services for serving and streaming. Although I have already built the server I am not quite certain yet on the best utilization of my SSD drives. My intended setup is shown in the drawing below. I have two main questions:

1) Which type of SSD should I use for Root and which for the fast storage (Samsung 970 Pro NVMe / Intel D3-S4610)?
2) Since I will not have any data stored on my root drive (meaning redundancy is not critical, I can live with some downtime) should I use one of the drives a SLOG or L2ARC device for the HDD pool?
3) Any other comments on the setup?


1619522777844.png
 

NateS

Active Member
Apr 19, 2021
159
87
28
Sacramento, CA, US
My first suggestion would be to use something smaller and cheaper as a boot drive -- there's no reason to use a high end device for something you're going to write once and rarely touch again, and your base install will probably be <<16GB. If you've got a spare x2 NVMe M.2 slot, I like these $7 tiny Optanes** as boot drives (https://www.amazon.com/gp/product/B0854RKKZY/). Or if you've got a spare SATA, a cheap 120GB consumer drive works well ($23 example, but just get whatever is cheap: https://www.amazon.com/Silicon-Power-Performance-Internal-SU128GBSS3A55S25AC/dp/B07D7VTDNB/).

As for L2ARC caching, I'd personally use the S4610s for that over the 970s, since the S4610s are high endurance enterprise drives, whereas the consumer 970s will wear out faster in that role. But if you value read speed over drive lifetime, go ahead and use a 970. Note that in either case, no redundancy is needed, since losing the cache drive just means you have to go to disk instead, but no data is lost. Which means if you do use both the S4610s in this role, I'd recommend striping them rather than mirroring (or just use a single one, and free up a SATA port for a boot drive).

I wouldn't bother with an SLOG device unless your workloads are doing a ton of sync writes. If so, then using mirrored S4610s here would make sense.

For your docker data pool, two mirrored 970s would work very well.

So in summary, if you're out of ports or you want to just use what you have right now, I'd do one S4610 as a boot drive (or ebay it and get something cheaper), 2x 970s for your docker data pool, and the other S4610 as an L2ARC for your file storage pool. If you're able to squeeze one more drive in the system, I'd get a separate boot drive and mirror the two S4610s as L2ARC. In either case I wouldn't bother with SLOG unless you really need it, in which case you'll really want to add more drives, since that should be mirrored.

** Full disclosure: I work for Intel, on Optane devices. That said, that particular model of drive I linked is discontinued, and it looks like it's just being liquidated by an OEM, so neither myself nor Intel will make any money if you buy it.
 
Last edited:
  • Like
Reactions: Marsh

norsemangrey

New Member
Apr 27, 2021
4
0
1
Thanks for your thorough reply!

My first suggestion would be to use something smaller and cheaper as a boot drive -- there's no reason to use a high end device for something you're going to write once and rarely touch again, and your base install will probably be <<16GB. If you've got a spare x2 NVMe M.2 slot, I like these $7 tiny Optanes** as boot drives (https://www.amazon.com/gp/product/B0854RKKZY/). Or if you've got a spare SATA, a cheap 120GB consumer drive works well ($23 example, but just get whatever is cheap: https://www.amazon.com/Silicon-Power-Performance-Internal-SU128GBSS3A55S25AC/dp/B07D7VTDNB/).
Not so many spare NVMe ports and SATA ports unfortunately. I've got 2 x NVMe ports that I use for the Samsung 970 PROs. I have 2 spare SATA ports, but I plan to use them for future expansion of my HDD pool.

Both the Samsung and the Intel drives are, as you indicate, probably overkill for the Ubuntu install. I was very uncertain about my setup (going back an fourth between Proxmox, Ubuntu Server and FreeNAS) and I guess I just purchased something to cover many scenarios :) I only new I wanted to utilize ZFS where I could.

As for L2ARC caching, I'd personally use the S4610s for that over the 970s, since the S4610s are high endurance enterprise drives, whereas the consumer 970s will wear out faster in that role. But if you value read speed over drive lifetime, go ahead and use a 970. Note that in either case, no redundancy is needed, since losing the cache drive just means you have to go to disk instead, but no data is lost. Which means if you do use both the S4610s in this role, I'd recommend striping them rather than mirroring (or just use a single one, and free up a SATA port for a boot drive).
I'll go for the S4610 as the L2ARC caching then. To be honest I am not sure whether or not I will have any benefit of the L2ARC for my use case. I've been reading a lot about it, but there seems to be no clear answer and about 50/50 in the opinion of whether or not there will be any performance gain for a small home server.

I wouldn't bother with an SLOG device unless your workloads are doing a ton of sync writes. If so, then using mirrored S4610s here would make sense.

For your docker data pool, two mirrored 970s would work very well.
Noted.

So in summary, if you're out of ports or you want to just use what you have right now, I'd do one S4610 as a boot drive (or ebay it and get something cheaper), 2x 970s for your docker data pool, and the other S4610 as an L2ARC for your file storage pool. If you're able to squeeze one more drive in the system, I'd get a separate boot drive and mirror the two S4610s as L2ARC. In either case I wouldn't bother with SLOG unless you really need it, in which case you'll really want to add more drives, since that should be mirrored.

** Full disclosure: I work for Intel, on Optane devices. That said, that particular model of drive I linked is discontinued, and it looks like it's just being liquidated by an OEM, so neither myself nor Intel will make any money if you buy it.
I assume you agree then with using one device as a L2ARC instead of setting it up as a mirror device for the root. Even if using Root on ZFS (which would then be installed on single drive). Oh and any tip for how to configure the L2ARC? Is it best to use overprovisionning to increase the longevity the drive?
 
Last edited:

NateS

Active Member
Apr 19, 2021
159
87
28
Sacramento, CA, US
I'll go for the S4610 as the L2ARC caching then. To be honest I am not sure whether or not I will have any benefit of the L2ARC for my use case. I've been reading a lot about it, but there seems to be no clear answer and about 50/50 in the opinion of whether or not there will be any performance gain for a small home server.
Yeah, whether and how much you'll benefit from an L2ARC is going to depend pretty heavily on your workload as well. Since it sounds like you want this build to be flexible for a lot of different workloads, and it usually doesn't hurt even if it's not especially helping, I'd go for it. Or better yet, test it both ways, and if you find you're not getting much benefit from it, you can mirror the root instead.


I assume you agree then with using one device as a L2ARC instead of setting it up as a mirror device for the root. Even if using Root on ZFS (which would then be installed on single drive). Oh and any tip for how to configure the L2ARC? Is it best to use overprovisionning to increase the longevity the drive?
Yeah, I'd only bother mirroring the root if it was critical to have no downtime. Otherwise, I'd just back it up to an HDD, and restore if there's ever an issue (which in your case is not likely, since you're overprovisioning endurance heavily for the root drive).

As for the L2ARC, if you're wearing out it's endurance, then that means your workload is thrashing the cache, and you're not getting much benefit from it anyway. In that case, I'd recommend just disabling it rather than overprovisioning it. Or, possibly you just need a bigger cache so as not to thrash it. Or more ram, so that your L1ARC can do more of the heavy lifting. When it's tuned correctly with an appropriate workload, the L2ARC should only be written to very occasionally, and read from frequently.

ZFS is a very flexible, tunable filesystem, so a lot of what's best for it comes down to what workload you run on it. There's really no substitute for taking performance measurements with your actual workload.
 

norsemangrey

New Member
Apr 27, 2021
4
0
1
Yeah, whether and how much you'll benefit from an L2ARC is going to depend pretty heavily on your workload as well. Since it sounds like you want this build to be flexible for a lot of different workloads, and it usually doesn't hurt even if it's not especially helping, I'd go for it. Or better yet, test it both ways, and if you find you're not getting much benefit from it, you can mirror the root instead.

Yeah, I'd only bother mirroring the root if it was critical to have no downtime. Otherwise, I'd just back it up to an HDD, and restore if there's ever an issue (which in your case is not likely, since you're overprovisioning endurance heavily for the root drive).

As for the L2ARC, if you're wearing out it's endurance, then that means your workload is thrashing the cache, and you're not getting much benefit from it anyway. In that case, I'd recommend just disabling it rather than overprovisioning it. Or, possibly you just need a bigger cache so as not to thrash it. Or more ram, so that your L1ARC can do more of the heavy lifting. When it's tuned correctly with an appropriate workload, the L2ARC should only be written to very occasionally, and read from frequently.

ZFS is a very flexible, tunable filesystem, so a lot of what's best for it comes down to what workload you run on it. There's really no substitute for taking performance measurements with your actual workload.
Thanks again for your input. I guess I just have to get cracking then and then do some testing once I get everything up and running and maybe adjust later. Are there any recommended methods or applications for testing the performance for a system like this?
 

NateS

Active Member
Apr 19, 2021
159
87
28
Sacramento, CA, US
Well, there are a lot of benchmarks you could potentially optimize against, but how helpful that'll be will depend on how closely the benchmarks match your actual workload. In your case, since your workload isn't well-defined yet, I'd recommend just building something that's good for the "average" case (and there are lots of guides out there for this), and then measure and adjust over time once you can collect real performance data from the actual applications you're using.