I have fallen in love with MooseFS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

wings

Member
Mar 2, 2021
33
12
8
my moosefs-leader is a VM and it does not seem to causing me any problems. VM host is promox on a intel E5v4 based cpu system with ddr4. My usage of moosefs is mostly vm backups from promox or media files for plex. Current moosefs cluster is 135tb total, 80tb free. 150k fs objects 563k chucks. The chuckservers are all proxmox hosts again intel E5v4 based cpu system with ddr4, 19 disk spread over 3 nodes.
It works in the literal sense, but you should really try it as bare metal. You will notice a huge difference in latency and performance when doing any kind of metadata operations, and this effect is compounded as you get to millions of objects. My cluster has 1.1 billion objects.
 

zunder1990

Active Member
Nov 15, 2012
219
74
28
It works in the literal sense, but you should really try it as bare metal. You will notice a huge difference in latency and performance when doing any kind of metadata operations, and this effect is compounded as you get to millions of objects. My cluster has 1.1 billion objects.
For me it meets my needs in terms of speed and performance. Since it is only a single leader install having it in a VM allows me to moving the VM between hosts as I need to updated or hardware changes.
 

wings

Member
Mar 2, 2021
33
12
8
Fair enough, that's roughly why I was running it as a VM for a while too. At scale you notice it more. (Metadata wise, not data size wise)
 

amalurk

Active Member
Dec 16, 2016
313
117
43
102
For what it's worth -

I'm running MooseFS Pro at home (150TiB licence, 1.3PB cluster) and at work (1.5PB licence, 1.8PB cluster) and the erasure coding and high availability is solidly in "just ****ing works" territory. Failover is reliable. I sleep better with it than I do with my Ceph cluster.

Not only that, it actually *works* (read: provides reasonable, usable performance) on hard drive bricks/OSDs/disks, unlike Ceph... My 2c.

There's tons of development on Moose, it just isn't trickling down yet as MooseFS CE 4.x isn't a thing and won't be until... something? happens. I'm unclear on when they'll actually open source MFS4, but in the meantime it's by far the best SDS I've used. Happy customer.
Post some benchmarks we can use to compare with Ceph.
 
  • Like
Reactions: zunder1990

zunder1990

Active Member
Nov 15, 2012
219
74
28
using the same fio command as @wings on my moosefs cluster
read test root@sickchill2:/mnt/bigdata/media/testing# fio --name=seqread --rw=read --bs=4M - Pastebin.com
READ: bw=1060MiB/s (1112MB/s), 62.9MiB/s-69.9MiB/s (66.0MB/s-73.3MB/s), io=62.2GiB (66.8GB), run=60001-60040msec

write test root@sickchill2:/mnt/bigdata/media/testing# fio --name=seqread --rw=write --bs=4 - Pastebin.com
WRITE: bw=1038MiB/s (1089MB/s), 61.1MiB/s-67.9MiB/s (64.0MB/s-71.2MB/s), io=60.9GiB (65.4GB), run=60002-60054msec

my cluster
3 nodes running proxmox intel e5 v4 ddr4
19 disk spread among the 3 nodes
all disk are spinning rust ranging from 2tb to 16tb with about half 8tb drives
135tb total space
moosefs master is hosted as a VM one of the proxmox nodes
all servers are connected using 10gb to a swtich

client for testing was also a vm hosted on proxmox cluster
client mount was in a goal 2 folder
 
  • Like
Reactions: Aluminat

gb00s

Well-Known Member
Jul 25, 2018
1,217
623
113
Poland
Are you guys connecting MASTER, METADATA and CHUNK servers through IB or ETH? Because I find the provided performance figures a little bit on the lower side.
 

zunder1990

Active Member
Nov 15, 2012
219
74
28
Are you guys connecting MASTER, METADATA and CHUNK servers through IB or ETH? Because I find the provided performance figures a little bit on the lower side.
my 3 hardware nodes are connected together are connected to a 10gb ethernet switch.
 

tjk

Active Member
Mar 3, 2013
485
200
43
Does anyone have a contact # or email address to someone at MooseFS? I've sent several emails to contact@ and filled out their web form and haven't heard back in 3+ weeks.

They sure make it tough to buy a license, I hope support is way more responsive.
 
  • Like
Reactions: zunder1990

gb00s

Well-Known Member
Jul 25, 2018
1,217
623
113
Poland
Does anyone have a contact # or email address to someone at MooseFS? I've sent several emails to contact@ and filled out their web form and haven't heard back in 3+ weeks.
I will DM you contact details. If this does not work, just open a ticket at Github. They will jump ;)
 
  • Like
Reactions: Auggie and tjk

gb00s

Well-Known Member
Jul 25, 2018
1,217
623
113
Poland
So the Master HA feature from the Pro version could be done by a 2-node DRBD/Pacemaker setup if you don't intend to run it as a vm in a e.g. Proxmox cluster.
 
  • Like
Reactions: zunder1990

tjk

Active Member
Mar 3, 2013
485
200
43
So the Master HA feature from the Pro version could be done by a 2-node DRBD/Pacemaker setup if you don't intend to run it as a vm in a e.g. Proxmox cluster.
Is that how they handle it in the pro version?