BTFRS Opinions...

[ideabox]

What are your opinions on BTRFS in production?

My opinion on BTRFS is on paper looks much better for my clients than ZFS,
One thing which I may be wrong on but how BTRFS unlike ZFS it distributes whole files over the drives which is less write intensive over all the drives and in theory we could use 'Pro' consumer SSD's (INTEL S3520)? What is your opinion on that?
Also if one drive fails only the files on that drive are missing of the array given there is a back-up then we are ok.

My
BTRFS Pro:
Ability to expand pool as needed
Distributes whole files not bytes over drives
Use Intel S3520

BTRFS Cons (ZFS Pro):
User Share Quotas

Happy to hear any thoughts on this...

 

[Evan]

As a file system ? Ie on top of other raid or as a raid filesystem ?
The former yes and the latter I would not, stay with ZFS

 

[ideabox]

No raid at all, Just a file system with a drive Pool and Cache.

 

[Evan]

No raid at all, Just a file system with a drive Pool and Cache.

Seems to be stable, used also by Synology in this way.
I would say ok for production but the project is clear that parity raid is absolutely not supposed to be ready for production, of course lots of people may argue that.

BTRFS
BTRFS RAID 5/6 found to be unsafe

 

[Stephan]

So you like features btrfs has over say ext4... well I did try to use them. Two days ago I managed to hard-lock a btrfs system by running a deduplication daemon (bees) and wiping free space within a windows VM at the same time. Locked up all I/O to the file system, did have to reset the machine. Swiftly did a backup and switched to ext4. Here is my list:

PRO
- In-kernel (compared to ZFS)
- Capable to be run as root fs
- Transparent compression (lzo, zlib)
- Deduplication
- Checksums for metadata and data (ext4 has metadata crc32 now though)
- Snaphots

CON
- Unstable under load
- Compression and deduplication make i/o latency a gamble (on a 2 TB P3520!)
- Can't use a swap file without workarounds
- Copy On Write will severely fragment in-place workloads like VM images and databases (and if you turn COW off, no more checksums)
- No LZ4 compression with 3-5x faster decompression vs lzo, which is a show stopper for anyone using an NVMe drive nearing 2000 MByte/s read speeds
- df will misreport free space, in effect breaking traditional cli semantics
- The subvolume concept will make even die-hard users of Linux scratch their heads for a long time
- Only capable of RAID0/1-like mode, no usable RAID5/6
- No built-in encryption of subtrees for e.g. /home like ext4 has now
- Would not trust any boot loader's btrfs read code at this point, so booting kernel, ucode and initramfs from an EFI System Partition is non-optional imho

Would I use it today in 2017? No. Not in production and not even at home. Maybe in 5-10 years, after the developers have ironed out a few things.

 

[nk215]

I actually use BTRFS in production (RAID 1).

At home, I also use BTRFS (various RAID configs) and I really like it. In fact, BTRFS is the only way for me to get an all flash array inexpensively. I can’t get all the similar size SSD at once inexpensively for ZFS. With BTFRS, there’s no issue with mismatch hard drive size. I can change from one RAID configs to another w/o any issue. I can take a drive one if I need it somewhere else (basically, any SSDI have in my BTRFS is also a spare for somewhere else) or add a drive if I see a good deal.

Can’t do any of that with ZFS.

 

[vl1969]

I would add that , personally I would not use anything other than single / raid-1/ raid-10 in production.
this 3 modes work well.
if you use individual drives use single and than pool them with something like MergerFS
you can use snapraid to add a little peace of mind. all of the above and you can add the subvolume replication, just like ZFS,
for a bit more data safety use raid-1 or raid-10. yes you do lose usable space but it works well. and you do not need extra for pooling the drives. BTRFS just like LVM and ZFS will pool the drives into single volume for you, what even more, it will do it with drives of different size and speed and you still get a decent performance out of it.
need more space, btrfs volumes are easily expandable live, on working system.
connect the drive, add it to the volume and rebalance, all while the system is chugging along. even better you can expand one drive at a time, unlike ZFS that do need pairs in vdevs, for redundancy.

I agree, if you need raid-5 or 6 btrfs is not the FS for you. maybe later, maybe never, who knows.
but single drive, raid 1 or 10 it works and works well.

 

[Blue)(Fusion]

I use various "raid" modes in Btrfs in several home PCs and servers. Right now, I am scrubbing a 7-drive "raid10" array (yes, odd number of drives) and achieving 1300MB/sec read performance.....on top of dm-crypt/LUKS encryption (using AES-NI). (4TB 5900RPM Seagate NAS drives, btw)

Thus far, I've had zero issues with Btrfs "raid5" either, however I opted to change it to "raid10" for the issues found this past summer. As far as I've read on the mailing list, most if not all of the raid56 corruption issues have been squashed in the patches submitted for kernel 4.12.

As mentioned above, I view it as stable when using Btrfs on a single device or in raid0,1, or 10 modes, not using quotas, and not a huge number of subvolumes/snapshots.

 

[McKajVah]

I'm running two 2x2TB raid1 arrays and 10 single disks with BTRFS. On top of this I'm running Snapraid.

I keep my "important" stuff on the two Raid1 arrays and "media" files on the ten single disks. These 12 2TB are then configured as data-stores in Snapraid with two parity drives in case somthing happens.

Has been super stable and really like it.

edit: Forgot to mention that I'm running MergerFS on top of all this again to pool the drives. No problems and everything running super stable. All this is done on a Proxmox 5.0 beta server...

 

[jseba]

I can second Stephan's I/O lockups and instability with BTRFS.

We are using BTRFS on top of our SAN to host our new development sandboxes at work with each dev getting a LXC container. It was okay at first as people slowly started to migrate (1, maybe 2 per host) but now everyone has migrated and with a bunch of new devs, we're up to 3-4+ per host. We've had so many issues to the point that we are moving all the build artifacts back to an EXT4/XFS partition and eventually moving the distributed build system off the hosts that have developers and onto their own dedicated machines.

BTRFS simply cannot keep up with the I/O load of people compiling, continuous integration and running the decreased subset of the production system for testing. I've hard locked entire hosts multiple times due to a previous bug when simply trying to grep through our source code; upgrading the kernel mostly solved it, but I still run into the bug if I'm not careful with my searches.

From what I've seen working with the systems engineers, it's that BTRFS's CPU usage under sustained heavy random I/O just skyrockets; the kernel worker threads for the module just sit at 100% on all cores, starving the system for both I/O and CPU time. I've seen loads on a 24-core box easily surpass 300+ simply due to BTRFS just trying to keep up. I frequently have vim freeze up on me because the load climbs up over 80 multiple times a day.

 

[JustinH]

I looked at it a while ago (maybe outdated info by now) as storage for my VM images. Performance was terrible! I believe it does well on small files or more typical workstation type workloads, but hosting VM imagines I was 10-20Mb transfer speeds inside a VM when the image was on BTFRS. Move it to XFS and the performance returned (to the 100Mb range)


Sent from my iPhone using Tapatalk

 

[whitey]

I'm running two 2x2TB raid1 arrays and 10 single disks with BTRFS. On top of this I'm running Snapraid.

I keep my "important" stuff on the two Raid1 arrays and "media" files on the ten single disks. These 12 2TB are then configured as data-stores in Snapraid with two parity drives in case somthing happens.

Has been super stable and really like it.

edit: Forgot to mention that I'm running MergerFS on top of all this again to pool the drives. No problems and everything running super stable. All this is done on a Proxmox 5.0 beta server...

BTRFS and Snapraid and MergerFS...OH MY (lions and tigers and bears parody). Makes my head spin. Think I'll stick w/ ZFS wink wink/ya'lls killin' me smalls :-D

 

[Blue)(Fusion]

I looked at it a while ago (maybe outdated info by now) as storage for my VM images. Performance was terrible! I believe it does well on small files or more typical workstation type workloads, but hosting VM imagines I was 10-20Mb transfer speeds inside a VM when the image was on BTFRS. Move it to XFS and the performance returned (to the 100Mb range)

Btrfs uses Copy On Write, so every small modification to the VM images, requires the whole chunk to be rewritten at a new point on the drive, update the B-tree and update the free space cache. It's slow for this purpose. Only way to get decent performance with it as a VM host is use nodatacow mount option or set chattr +C on the directory containing your VM images to disable CoW (BEFORE you write/create the files!). It got me back to ext4 performance, less a few MB/sec. I do currently currently use XFS for VM image host on an MDRAID1 setup, however. It's just better purposed.

 

[vl1969]

BTRFS and Snapraid and MergerFS...OH MY (lions and tigers and bears parody). Makes my head spin. Think I'll stick w/ ZFS wink wink/ya'lls killin' me smalls :-D

Whitey, you do understand that "BTRFS and Snapraid and MergerFS" setup is optional and not truly needed
it just gives you an extra options on how you want to use your disks and extra flexibility that you might not get with ZFS. what ZFS gives you that you can not do with BTRFS except raid-5/6 type of setup?
and I am not trying to jump on zfs bashing bandwagon here. but over all with just a few shortcomings both can work for you.

IMHO, BTRFS is more flexible for home use.
you can not expand raided ZFS pool by adding a single disk it must be vDev with proper number of disks as the raid dictates. you can not mix different size disks in vdevs, and on and on and on...
in the enterprise it might be ok, as you have a budget for this, but at home I might only have one drive available. btrfs works out better.
you can ads a single drive to the pool and rebalance it live, disks do not have to match etc...
it give you the CoW FS as ZFS, a bit rot protection and other things just like ZFS

you have option of
#1. run as any other Linux FS, i.e. each drive is formated on it's own

#2. setup a pool of mixed drives as needed with no protection(no raid), similar to LVM, MergerFS and ZFS drive pooling.

#3. build raided pool using RAID 0/1/10 (5 and 6 exists but not stable yet.) raid-0 gives you speed but no protection, raid 1 and 10 get you speed and data protection(within a reason of course)

#4. use SnapRaid with individual disks and pool them with MergerFS or similar software

what is wrong with having options ?

 

[JustinH]

Btrfs uses Copy On Write, so every small modification to the VM images, requires the whole chunk to be rewritten at a new point on the drive, update the B-tree and update the free space cache. It's slow for this purpose. Only way to get decent performance with it as a VM host is use nodatacow mount option or set chattr +C on the directory containing your VM images to disable CoW (BEFORE you write/create the files!). It got me back to ext4 performance, less a few MB/sec. I do currently currently use XFS for VM image host on an MDRAID1 setup, however. It's just better purposed.

Yeah. I realize the work arounds but then you loose most of the compelling features of it. Might as well stick with ZFS or XFS with LVM volumes etc!


Sent from my iPhone using Tapatalk

 

[djbon2112]

BTRFS is a mess for anything except a desktop system and has deep systemic flaws in its checksumming with RAID; so basically, it's a ZFS clone that can't do half of what ZFS can (like guarantee your data). Production? Never.

Edit: Some posters mention "squashing bugs" about this in 4.12, but that's my point - if there's major bugs being fixed right now, ten years into development, that cause active data corruption, the system is not fit for production by any measure and is fundamentally flawed. Besides, that ignores the holes in the write checksumming logic which are baked into the design: Btrfs RAID 5/6 Code Found To Be Very Unsafe & Will Likely Require A Rewrite - Phoronix is not even a year old yet. If this has actively changed I'd love to know but all I hear is occasional chatter about it "getting better but still not there". I've been personally waiting for years and *wanting* to move to BTRFS, but problem after problem keeps cropping up. And for prod, your data integrity is the *only* thing that matters.

ZFS has native modules - you just have to compile them yourself, or use the DKMS packages (official in Debian but older, or self-build from the repo). Don't be scared to use them if ZFS is the best tool for the job. They're rock-solid and power my Ceph cluster's OSDs and my backup array without an issue in years.

If you want a root partition for a VM, then OK BTRFS will do (though full disclosure, I use ext4 because it works well and is hands-off to manage). But use ZFS for datastores and RAID, especially in production, and, if you care about your data, in your homelab too.

Or, if you're a homelabber reading this and want to try something new and great and help the development of filesystems, because it's a homelab and not really "production", check out bcachefs (no typo), which is exactly what it says on the tin: a file system based on bcache by the original author which is trying to do btrfs' job right. It's still early but worth looking into and supporting; maybe not for homelab (and especially not prod) data, and it's still missing features like RAID, encryption and compression, but btrfs is honestly never going to get there due to baked-in bad design, so I'm spreading the word about bcachefs to help get interest, testers, and more development!