Finally finished all the testing and ready to make then change to zfsonLinux

[vangoose]

Being using ZFS since it's out and using Solaris/OpenSolaris/OmniOS for my centralized storage server for a few years. The storage server provides VMware datastore through NFS mainly, some iSCSI (used to do FC) for specific applications, then NFS/SMB for other servers and workstations at home.

Generally I'm very happy with the setup but lately as the flash/nvme disks take off, the setup is not providing the performance I'm supposed to get even I upgraded to 10G network, SAS SSD for Vmware datastore, Optane for slog, etc. No VAAI and lack of support for VMFS space reclaim from Comstar are other disadvantage on Solaris platform.

I started to look at other options, tried vSAN, Nutanix CE, FreeNAS, but not happy with any of them. The ICX switch post got me looking at RDMA option as I was almost out of port on my 10G-T switch and my HP procurve switch showed its age.

Long story short, I've completed the POC and migrated all the data from my Solaris 11.4 server to 2 temporary CentOS 8 servers so I can rebuild my Solaris server to CentOs 8.

Here is the brief of the new infrastructure for my home lab to be built.
Switchs
- 2 ICX-7250/P in stack for frontend traffic in full layer 3.
- 2 ICX-7250P in layer 2 for backend storage traffic.
- ICX 6450-24 will be used to extend connection to other room if needed
- 1 Netgear XS716T for 10G-T

Storage Server
- SuperMicro 743TQ-SQ case
- XEON E5-2620 on SuperMicro X9SRL (thinking to upgrade CPU)
- Intel x550-T2/HP 530SFP+ for frontend traffic
- ConnectX-3 for backend traffic
- HGST SN260 6.4TB HHHL nvme for VMware storage
- 8*HGST DC510 10TB SAS in mirror for NFS/SMB
- CentOS 8.0
- ZFSonLinux 0.82
- SCST for iSER/iSCSI target (tried LIO but SCST performs better)
- SAMBA 4.1, NFS 4.1
- Backup - NetBackup (another reason FreeNAS won't work)


A few issues during POC.
- HGST SN260 doesn't work on my old ASUS P8B WS with XEON E3 board, Intel P3700 doesn't post either, but Intel SSD 750 works fine. Had to use older SuperMicro X8 board, thus 2 temp storage server, performance is lower than SSD 750 on P8B WS.
- Had issues where some ZFS volumes not mounted on the system
- ZFS broken after kernel update (using kmod). No kmod for latest kernel update, had to switch to DKMS and exclude future kernel update.

If enough people are interested, I can write a build guide when I rebuild the server.

 

[BeTeP]

Any specific reason for choosing CentOS? Personally I switched to Proxmox even for generic non VM related use. I think of it as Debian + zfs.

 

[vangoose]

Any specific reason for choosing CentOS? Personally I switched to Proxmox even for generic non VM related use. I think of it as Debian + zfs.

NetBackup

 

[Rand__]

My primary concern is long term reliability.
O/c its simple (more or less) to build a ZoL filer, but will it run without major hickups for 5 years? Even if I update OS and zfs?
Not really keen on loosing pools due to upgrade issues (even if they can be recovered/fixed), I am too old for that. O/c backups will cover most of that too, but it would bee quite annoying to loose all VMs to a zfs update and then having to restore them from a hopefully recent backup (since almost all my client/servers run on VM that would mean no workstation, no Internet, no PDC until I can bring up a backup with the emergency hardware pc )

Hm means I need nightly backups to an easily reachable (slower) datastore - maybe vsan...

A build guide can't hurt

 

[Rain]

My primary concern is long term reliability. ... Not really keen on loosing pools due to upgrade issues (even if they can be recovered/fixed), I am too old for that.

ZFSonLinux has proven to be quite stable but has had very occasional hiccups with updates. The "right" way, in my opinion, to handle this is simply not to jump to the latest updates right away. Wait and see if other big users report bugs/issues.

 

[Rand__]

Hm ok, been doing that for FreeNas too.
So whats the verdict on versions and distris? Whats a well performing combo?

 

[vangoose]

Hm ok, been doing that for FreeNas too.
So whats the verdict on versions and distris? Whats a well performing combo?

I'm freezing kernel and ZFS until I fully test the new versions as with Solaris 11.4 SRU as well. Learned it during POC, a mismatch kernel will bring zpool down. Even Solaris 11.4 SRU broke my storage server.

Good thing is my critical data is only abount 2-3TB, I have daily backup and another storage server that I can storage vmotion to before upgrade plus some local ssd datastores.

I'm still debating if I should use XFS for NAS data, ZFS for zvol or ZFS for both.

 

[Rand__]

well the value of zfs holds true also for non VMs... not much harm on video but pics start to look funny at some point

Probably going to wait for 0.8.3 which might be relatively soon and then need to run some tests .
My initial attempt was messed, didnt load modules on boot and with rdma and pmem drivers thats probably going to take a couple of attempts

 

[vangoose]

well the value of zfs holds true also for non VMs... not much harm on video but pics start to look funny at some point

Probably going to wait for 0.8.3 which might be relatively soon and then need to run some tests .
My initial attempt was messed, didnt load modules on boot and with rdma and pmem drivers thats probably going to take a couple of attempts

I have zfs, scst, iser, samba and nfs all worked out, smb permission isn't a big deal in my case, so far everything is good.

 

[Rand__]

glad to hear that - looking forward to your guide

Your vm storage is a single/pool of the SN260's?
How many VMs are you running on that?

 

[vangoose]

glad to hear that - looking forward to your guide

Your vm storage is a single/pool of the SN260's?
How many VMs are you running on that?

My SN260 is AIC that requires x8 slot and I have 3 of them. I still have one x8 slot free on my board but using 2 is a bit of waste for my lab as I have daily backup and other storage server in case the primary one fails.

X9 only has 4*x8 slots and 2*x4 slots from cpu, one pcie v2 x4 slot.
I have
- LSI 9207-8i in x8 for 8*10TB HGST He10 SAS
- HGST SN260 6.4TB in x8
- ConnectX-3 dual port in x8 for RDMA
- Intel X550-T2 in x4 for SMB/NFS with LACP
- Optane 900P in x4 (not used now)

I have one shelf with 24*3TB HGST Ultrastar disks in case I need to expand the storage, I need slot for external HBA.

30-40 VMs for now.

 

[Rand__]

Didnt even know there were x8 nvme cards, nice.
Could you do me a favor and do a fio run on that (locally and or vm) with a single job, single qd, seq write 64K blocksize

 

[vangoose]

Didnt even know there were x8 nvme cards, nice.
Could you do me a favor and do a fio run on that (locally and or vm) with a single job, single qd, seq write 64K blocksize

Yes once I finish some basic config.
The SN260 6.4TB is rated for 6000MB+/s + read and 2200MB+/s for write. random 4K is is 1.2M for read, 200000 for write, a lot faster than the 7.68TB version. Endurance is 3dwpd

 

[Rand__]

yeah just looking at it

 

[vangoose]

yeah just looking at it

SN 260 doesn't work on my Asus workstation board and Gigabyte desktop board. It works on every of my Supermicro X8/X9 board.

 

[Rand__]

Well havent found a MM version anyway (at any realistic price point), just an MR. Not sure thats worth spending so much on at this point in time.
Still hope rdma will help out with remote access performance, that should be enough then even with the scaling issues (single thread performance scaling on multiple drives is basically non-existent) (got a pool of SS300's ready for that box)

 

[vangoose]

Well havent found a MM version anyway (at any realistic price point), just an MR. Not sure thats worth spending so much on at this point in time.
Still hope rdma will help out with remote access performance, that should be enough then even with the scaling issues (single thread performance scaling on multiple drives is basically non-existent) (got a pool of SS300's ready for that box)

I tested single Intel 750 during POC, iSER maxed out the performance of 750 with 2.2GB/s read and 1.2GB/s write, faster than my SAS3 HGST SSD zpool on Solaris with NFS/iSCSI. 100% faster than another ESX host with iSCSI protocol.

iSER is 50-100% faster than iSCSI on same hardware if configured properly.

 

[Rand__]

On how many threads/jobs and qd?
But nice and glad to hear that.

Makes me wonder though why so few jump on the bandwagon.
At least iwarp should be simple enough to implement since it doesn't need the switch to support it...

 

[vangoose]

On how many threads/jobs and qd?
But nice and glad to hear that.

Makes me wonder though why so few jump on the bandwagon.
At least iwarp should be simple enough to implement since it doesn't need the switch to support it...

zol 0.8.3 is released.

 

[Rand__]

yay, thanks for the heads up, need to check it out