ZFS on Linux

[cactus]

I have moved over to using ZoL simply because I am more comfortable in Linux than the SunOS children.

I currently am using ZoL on three systems. My desktop(G630/16GB RAM) with a 4 drive RAIDZ that was created on OpenIndiana. A HP Microserver N40L(4GB RAM) with a two drive mirror created in Linux. My VM datastore(G34 6128/32GB RAM), also created on OI.

My experience has been rather painless with one exception. I added RAM to my single G34 system, 16GB to 32GB, and started experiencing hardlocks. Browsing through the bug reports, it seems there is a problem with NUMA machines with large amounts of RAM. From what I have been able to surmise, each G34 chip is seen as two NUMA nodes.

Anyone else using ZoL? If so, what are your experiences?

 

[Patrick]

I started playing with ZoL a few weeks back on a 2P G34 but have not touched NUMA setting yet.

My big worry is that ZoL does not gain traction. If it does, very exciting.

 

[sotech]

We moved our main server across after a period of testing (from OpenIndiana) and couldn't be happier with it. The decision was made based on the fact that most of us here have a lot of experience with Ubuntu whereas few were confident with Solaris... It has been just as stable and the array performance has either stayed the same or improved. One adventurous customer has moved his across, too, and he has had a very similar experience. He did record more of a performance improvement but he is running on quite different hardware to our own.

 

[Patrick]

Yea I know, we need main site ZoL posts... something that I have been working on but really short on time for. I think I did manage to do a how-to installation video... but still waiting for the new Youtube pre-roll to land.

One of the biggest things for me is that Linux hardware support v. OpenIndiana is a world of difference.

 

[gea]

Yea I know, we need main site ZoL posts... something that I have been working on but really short on time for. I think I did manage to do a how-to installation video... but still waiting for the new Youtube pre-roll to land.

One of the biggest things for me is that Linux hardware support v. OpenIndiana is a world of difference.

I would not declare the hardware to be a problem.
Suggested Solaris Hardware is nearly always the most stablest on Windows or Linux too. Its mostly enough to support simply the best like LSI storage controllers and Intel Nics or Intel server chipsets like Apple does (nearly all other mainboard chipsets works well, other hardware like Realtek Nics may cause problems but who cares?). I miss the availability of well tested and supported software beside pure storage use cases like for example HA in enterprise use or media servers at home use.

For a pure NAS/SAN regarding performance, stability and "it just runs at best performance without tweaking" I would name Solaris/OI and the other OpenSolaris derives OS's as unique and simply the best option. This is also the reason why I use a virtualized OI NAS/SAN together with other virtualized systems like BSD, Linux or Windows in an all-in-one setup at home and at work in my university and lab configs.

For these applications and if you are experienced to manage Linux, ZFS on Linux is a real option. (File system is the most important decision for data security).

For me its not an option. I tried Linux/SAMBA some times to replace my Windows AD 2003 filers. Nexentastor2 or newer Nexenta*/OI/or whatever with Solaris CIFS were the first systems where I would say it is worth the effort and I have now more advantages than troubles. (Today I would not go back to a Windows 2008/12 filer)

 

[cactus]

Just to note, ZoL does not suport direct file I/O (O_DIRECT) currently. Stumbled across this doing some benchmarks.

 

[Thatguy]

I added RAM to my single G34 system, 16GB to 32GB, and started experiencing hardlocks. Browsing through the bug reports, it seems there is a problem with NUMA machines with large amounts of RAM. From what I have been able to surmise, each G34 chip is seen as two NUMA nodes.

Anyone else using ZoL? If so, what are your experiences?

I'm running ZoL on Ubuntu 12.10 (was CentOS 6.3, RHEL 6 before that) on my dual e5 2620 w/ 64GB ram. No random issues when running it bare metal. I recently moved to having Ubuntu virtualized under esxi using pci passthrough. I had a crash earlier today w/ ZoL and kswapd. Not sure if it's because the VM only has 16GB of ram or what, but we'll see.

Just to note, ZoL does not suport direct file I/O (O_DIRECT) currently. Stumbled across this doing some benchmarks.

Sure doesn't. It also has some goofyness with the linux AIO implementation I think. Definitely had problems with KVM virtual disks and ZoL. It only likes setting writeback cache, or doesn't work if you set writeback. I can't remember exactly and hated KVM, so moved away from it in a hurry.

 

[cactus]

I'm running ZoL on Ubuntu 12.10 (was CentOS 6.3, RHEL 6 before that) on my dual e5 2620 w/ 64GB ram. No random issues when running it bare metal. I recently moved to having Ubuntu virtualized under esxi using pci passthrough. I had a crash earlier today w/ ZoL and kswapd. Not sure if it's because the VM only has 16GB of ram or what, but we'll see.

I didn't keep track of the exact bug that said it had to do with NUMA, but last I searched the problem was occurring without NUMA. There were non-NUMA safe calls, IIRC they were corrected, but the problem persisted.

I have been moving a lot around and only have my desktop(now an E3-1245v2) running ZoL or ZFS for that matter. I also have spindown working with the pool.

As to KVM, I am going to try out OmniOS and SmartOS, but thats another thread.

 

[cactus]

Actually, see issue #223, AIO is not implemented yet with ZoL.

 

[Biren78]

Actually, see issue #223, AIO is not implemented yet with ZoL.

Does that destroy performance w/ no async handlers?

 

[gigatexal]

Does that destroy performance w/ no async handlers?

has o-direct been fixed/added?

 

[Aluminum]

ZoL is pretty awesome for just being able to do simple setups on existing systems with better data integrity than a cheap card or mdadm, not just dedicated SAN/filer stuff.

Like "hey I got this workstation/whatever server with a couple non-boot drives, I'd like to mirror them".

As for gaining traction, it seems to be getting more popular along with all kinds ZFS in general, which is good in my book. I also like that the open source ZFS players have banded together and given oracle the middle finger it deserves.

 

[StephD]

Seems like a great option for the linux based all in one station I have in mind. This pretty much removes the need for VT-d support which is missing on Avoton based solutions.

 

[HellDiverUK]

Just starting to set up ZoL. I use NAS4Free at work (on a NL40, just about to hit a year uptime), so I'm pretty happy with ZFS. I'll be using napp-it on Debian. I have played with napp-it on OpenIndiana in the past too.

I'm trying ZoL on Debian Wheezy, currently in a VM. If it all goes well, I'll be putting it on an i3 machine to replace the Synology/Asustor units I've got.

 

[ehorn]

osnexus quantastor is a slick zol distribution. Not sure I would pay for it, as its just puts a web ui on the surface, but in a lab or small storage environment (<= 10TB raw) it is free and pretty polished.

 

[HellDiverUK]

Napp-It purely for the ZoL interface, and webmin for everything else. Two free and easy to install GUIs for a X-less linux box.