Woke up this morning, found 2 more XCH, total of 4 XCH in 16 days.
I have 1400 plots now.
Good for you - I've had two days of "fun" - my setup basically exploded ...
O/c I have made the mistake to simply deploy a few full nodes and then move the plots to a central single large directory...
This has been working without issues, I'd say up to 6-700 plots.
Then first filer became slow (zfs filled up), so copy times became worse - o/c Gui looked fine...
Then found the comment re mergerfs so built an OMV box as secondary storage... bad idea - mergerfs is significantly slower than individual disks when written to and - more importantly - I had no idea re options, so one night the drive said full despite only a single disk being full...
So then I had some 50 plots not moved over + first suspicions that the single ZFS filer might be a problem...
so i started moving plots around...
Now 100G plots on spinners dont move fast.... so this is taking ages...
then i noticed that my proof & copy times became abysmal
After playing around for two days I came to the conclusion that
1. running multiple full nodes is a bad idea for shared storage, as all will try to proof and being application agnostic the disk tries to do all of them at the same time
2. Multiple read/write operations at 100G size on a single disk ea is not really good - especially when proofing is running in parallel
3. UnionFS/MergerFS is slower then expected, don't use it exclusively. I now have a local harvester including all individual disks and just push via unionfs for space balancing
4. Do a reboot after installing a harvester's init -c - else it wont connect
But better now, only need to fix the 100T TrueNAS filer to use a local harvester somehow, or migrate the plots or convert to ZoL
Just as a warning - all looks good until it goes south - check the logs