I have fallen in love with MooseFS

[wings]

my moosefs-leader is a VM and it does not seem to causing me any problems. VM host is promox on a intel E5v4 based cpu system with ddr4. My usage of moosefs is mostly vm backups from promox or media files for plex. Current moosefs cluster is 135tb total, 80tb free. 150k fs objects 563k chucks. The chuckservers are all proxmox hosts again intel E5v4 based cpu system with ddr4, 19 disk spread over 3 nodes.

It works in the literal sense, but you should really try it as bare metal. You will notice a huge difference in latency and performance when doing any kind of metadata operations, and this effect is compounded as you get to millions of objects. My cluster has 1.1 billion objects.

 

[zunder1990]

It works in the literal sense, but you should really try it as bare metal. You will notice a huge difference in latency and performance when doing any kind of metadata operations, and this effect is compounded as you get to millions of objects. My cluster has 1.1 billion objects.

For me it meets my needs in terms of speed and performance. Since it is only a single leader install having it in a VM allows me to moving the VM between hosts as I need to updated or hardware changes.

 

[wings]

Fair enough, that's roughly why I was running it as a VM for a while too. At scale you notice it more. (Metadata wise, not data size wise)

 

[amalurk]

For what it's worth -

I'm running MooseFS Pro at home (150TiB licence, 1.3PB cluster) and at work (1.5PB licence, 1.8PB cluster) and the erasure coding and high availability is solidly in "just ****ing works" territory. Failover is reliable. I sleep better with it than I do with my Ceph cluster.

Not only that, it actually *works* (read: provides reasonable, usable performance) on hard drive bricks/OSDs/disks, unlike Ceph... My 2c.

There's tons of development on Moose, it just isn't trickling down yet as MooseFS CE 4.x isn't a thing and won't be until... something? happens. I'm unclear on when they'll actually open source MFS4, but in the meantime it's by far the best SDS I've used. Happy customer.

Post some benchmarks we can use to compare with Ceph.

 

[wings]

Post some benchmarks we can use to compare with Ceph.

https://gist.githubusercontent.com/Zorlin/55024e6cccc6096c9efd7cc26c2fc95e/raw/910f6ebd7617ebafe33d48cea751dbff04750682/fio-read-sequential-4MB

this'll give you some idea, this is not erasure coded however. I expect with erasure coding you'd see slightly higher latency and maybe 880MiB/s.

 

[wings]

here it is with 128k blocksize instead of 4MB

https://gist.githubusercontent.com/Zorlin/55024e6cccc6096c9efd7cc26c2fc95e/raw/8671f6272edc24ce1fb37beb805641dbf3a661e9/fio-read-sequential-128k

 

[zunder1990]

using the same fio command as @wings on my moosefs cluster
read test root@sickchill2:/mnt/bigdata/media/testing# fio --name=seqread --rw=read --bs=4M - Pastebin.com
READ: bw=1060MiB/s (1112MB/s), 62.9MiB/s-69.9MiB/s (66.0MB/s-73.3MB/s), io=62.2GiB (66.8GB), run=60001-60040msec

write test root@sickchill2:/mnt/bigdata/media/testing# fio --name=seqread --rw=write --bs=4 - Pastebin.com
WRITE: bw=1038MiB/s (1089MB/s), 61.1MiB/s-67.9MiB/s (64.0MB/s-71.2MB/s), io=60.9GiB (65.4GB), run=60002-60054msec

my cluster
3 nodes running proxmox intel e5 v4 ddr4
19 disk spread among the 3 nodes
all disk are spinning rust ranging from 2tb to 16tb with about half 8tb drives
135tb total space
moosefs master is hosted as a VM one of the proxmox nodes
all servers are connected using 10gb to a swtich

client for testing was also a vm hosted on proxmox cluster
client mount was in a goal 2 folder

 

[gb00s]

Are you guys connecting MASTER, METADATA and CHUNK servers through IB or ETH? Because I find the provided performance figures a little bit on the lower side.

 

[zunder1990]

Are you guys connecting MASTER, METADATA and CHUNK servers through IB or ETH? Because I find the provided performance figures a little bit on the lower side.

my 3 hardware nodes are connected together are connected to a 10gb ethernet switch.

 

[tjk]

Does anyone have a contact # or email address to someone at MooseFS? I've sent several emails to contact@ and filled out their web form and haven't heard back in 3+ weeks.

They sure make it tough to buy a license, I hope support is way more responsive.

 

[gb00s]

Does anyone have a contact # or email address to someone at MooseFS? I've sent several emails to contact@ and filled out their web form and haven't heard back in 3+ weeks.

I will DM you contact details. If this does not work, just open a ticket at Github. They will jump

 

[gb00s]

So the Master HA feature from the Pro version could be done by a 2-node DRBD/Pacemaker setup if you don't intend to run it as a vm in a e.g. Proxmox cluster.

 

[tjk]

So the Master HA feature from the Pro version could be done by a 2-node DRBD/Pacemaker setup if you don't intend to run it as a vm in a e.g. Proxmox cluster.

Is that how they handle it in the pro version?

 

[gb00s]

Is that how they handle it in the pro version?

This is what I'm trying to figure out.

 

[elvisimprsntr]

@zunder1990 Congratulations! When is the wedding?