Hi!
After spending a lot of time reading valuable sources such as STH, I spent the last 3 months slowly incubating my new file server. I am unfortunately quite disappointed by the performance of the resulting system.
I apologize for the long post, but I will try to give all relevant info that experts around might need to help me tweak my setup.
Server configuration
Use case and expectations
Overkill home office server, able to fully take advantage of the 10GbE LAN (+ 1Gb Internet)
Performance tuning so far
I don't think my expectations are unrealistic and I will be gratefull if someone could point me towards the correct direction (to 10GbE Nirvana ;-)
Thank you for the read and for your most welcomed insights.
Additional information
After spending a lot of time reading valuable sources such as STH, I spent the last 3 months slowly incubating my new file server. I am unfortunately quite disappointed by the performance of the resulting system.
I apologize for the long post, but I will try to give all relevant info that experts around might need to help me tweak my setup.
Server configuration
- OS: OpenMediaVault 5.3.5-1 with Proxmox Kernel (Debian GNU/Linux, with Linux 5.3.18-2-pve), booting from a USB 3.1 stick (Samsung MUF-128AB), using the OMV flashmemory plugin.
- Motherboard: ASRockRack X470D4U2-2T (BIOS 3.30, BMC FW 1.70) custom fitted into a Rackable S3012 chassis
- CPU: AMD Ryzen 7 3700X
- RAM: 2 x 32GB ECC Samsung M391A4G43MB1-CTD DDR4-2666@3200MT/s
- HBA: LSI 9201-16i using 3 SAS connections to the 3 backplanes of the S3012. The HBA is mounted on PCIE slot #6 of the MB (PCIE 3.0 x 16) with an old PCIE 2.0 riser and flashed to IT mode with FW 20.00.07.00.
- ZFS pool: 12 x 3TB HDD (WD Red) in 2 vdevs of 6x RaidZ2 (ZFS version 0.8.3-pve1)
- NIC: 10GbE embedded Intel X550 (server) -> Mikrotik CR305-1G-4S+IN / S+RJ10 SFP+ modules -> Sonnet Solo10G thunderbolt 3 (to iMac or PC, using Cat6a S/FTP cabling)
- Power Supply: Corsair SF750, 750 Watt, 80Plus Platinum
Use case and expectations
Overkill home office server, able to fully take advantage of the 10GbE LAN (+ 1Gb Internet)
- Documents & Media file server : over SMB for Mac / Windows clients and over NFS for Kody/Emby
- Direct editing of huge RAW photos / movies from iMac over SMB
- Time Machine backups for Mac clients over SMB
- Various docker containers : portainer, unifi-controller, embyserver (with software transcoding), logitech-media-server, heimdall, rutorrent (through Wireguard), letsencrypt, etc.
- (TBD: RSync or ZFS send to a local OpenMediaVault backup server + encrypted cloud syncing to Dropbox / Google Drive + homelab virtualization with Cockpit)
Performance tuning so far
ZFS
main pool (/tank):
dataset used for performance tests (/tank/media):
/etc/modprobe.d/zfs.conf:
SMB options (just to solve some issues with MacOS Catalina)
Code:
ashift = 12
recordsize = 128K
compression = lz4
atime = off
xattr = sa
sync = disable (the server is protected with a UPS, monitored with NUT)
Code:
recordsize = 1M
other attributes are inherited from /tank
Code:
options zfs zfs_arc_max=53687091200 (I don’t need much RAM for the non ZFS tasks)
Code:
min protocol = SMB2
usershare path =
NFS options (local share for Kodi / Emby, used to stream UHD content to an Android TV set)
Speed tests
Code:
ro,subtree_check,insecure,crossmnt
Bonnie++ (with compression = off on /tank/media)
gets 820MB/s write, 320MB/s rewrite, 677MB/s read:
Note 1: the write performance is quite good here due to the use of sync=disable and increased arc size.
Note 2: with compression = lz4, I get 1.7GB/s write, 1.5GB/s rewrite, 3.4GB/s read with 100% CPU, but this is not a realistic scenario with my mostly incompressible data.
Sample output of # zpool iostat -v 10 during bonnie++ read phase:
Note: during bonnie++ write phase, I get between 84 to 107MB/s write speed on each drive.
I think I should get 25 to 50% more read speed than this...
Code:
# bonnie++ -u root -r 1024 -s 64G -d /tank/media -f -b -n 1 -c 4
Code:
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
gringotts 64G::4 820m 68 320m 29 677m 35 656.5 49
Latency 43974us 975ms 296ms 450ms
Version 1.98 ------Sequential Create------ --------Random Create--------
gringotts -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 59us 9us 428us 35us 71us 72us
1.98,1.98,gringotts,4,1583180288,64G,,8192,5,,,839945,68,327742,29,,,693055,35,656.5,49,1,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,43974us,975ms,,296ms,450ms,59us,9us,428us,35us,71us,72us
Note 2: with compression = lz4, I get 1.7GB/s write, 1.5GB/s rewrite, 3.4GB/s read with 100% CPU, but this is not a realistic scenario with my mostly incompressible data.
Sample output of # zpool iostat -v 10 during bonnie++ read phase:
Code:
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------------- ----- ----- ----- ----- ----- -----
tank 15.6T 16.9T 2.44K 144 654M 1.14M
raidz2 7.80T 8.45T 1.22K 67 326M 576K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1NV74PJ - - 0 11 5.20K 98.0K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4NNRF1H - - 312 10 81.5M 98.4K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKLVF6 - - 314 10 81.5M 91.6K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7PJANV7 - - 313 10 81.5M 93.2K
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K6YCCJS3 - - 304 11 81.5M 95.6K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6AT49D5 - - 0 12 1.60K 99.2K
raidz2 7.77T 8.48T 1.22K 76 328M 589K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1XDHAKL - - 0 12 3.60K 97.2K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKL3SA - - 313 12 82.1M 101K
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K5PKN5R0 - - 310 13 82.1M 102K
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K1SHENDA - - 310 12 82.1M 99.2K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5VPU63H - - 313 12 82.1M 94.8K
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6SJP7R9 - - 0 12 5.20K 94.0K
-------------------------------------------- ----- ----- ----- ----- ----- -----
I think I should get 25 to 50% more read speed than this...
iperf3 from client to server over 10GbE
Note 1: I see here that I have some retries from server to client, which is probably due to a cabling issue. I will investigate this but I believe it is marginal.
Note 2: MTU size is kept to default 1500 as it seems the Mikrotik SFP+ modules I have don’t support jumbo frames (S+RJ10 rev. 1). This may explain the gap between local and SMB speeds... I will try a direct link with jumbo frames enabled to see the difference.
Code:
% iperf3 -c 10.0.0.16 -t 30 -V
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-30.00 sec 32.8 GBytes 9.40 Gbits/sec sender
[ 5] 0.00-30.00 sec 32.8 GBytes 9.40 Gbits/sec receiver
CPU Utilization: local/sender 98.7% (0.6%u/98.1%s), remote/receiver 31.4% (0.7%u/30.7%s)
rcv_tcp_congestion cubic
Code:
% iperf3 -c 10.0.0.16 -t 30 -V -R
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 31.9 GBytes 9.12 Gbits/sec 869 sender
[ 5] 0.00-30.00 sec 31.9 GBytes 9.12 Gbits/sec receiver
snd_tcp_congestion cubic
Note 2: MTU size is kept to default 1500 as it seems the Mikrotik SFP+ modules I have don’t support jumbo frames (S+RJ10 rev. 1). This may explain the gap between local and SMB speeds... I will try a direct link with jumbo frames enabled to see the difference.
Speed Test over SMB / 10GbE (target is /tank/media)
I don't think my expectations are unrealistic and I will be gratefull if someone could point me towards the correct direction (to 10GbE Nirvana ;-)
Thank you for the read and for your most welcomed insights.
Additional information
Code:
Pool status (zpool status):
pool: tank
state: ONLINE
scan: scrub repaired 0B in 0 days 06:11:44 with 0 errors on Tue Feb 11 21:56:46 2020
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1NV74PJ ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4NNRF1H ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKLVF6 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7PJANV7 ONLINE 0 0 0
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K6YCCJS3 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6AT49D5 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1XDHAKL ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKL3SA ONLINE 0 0 0
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K5PKN5R0 ONLINE 0 0 0
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K1SHENDA ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5VPU63H ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6SJP7R9 ONLINE 0 0 0
errors: No known data errors
Pool details (zpool get all):
NAME PROPERTY VALUE SOURCE
tank size 32.5T -
tank capacity 47% -
tank altroot - default
tank health ONLINE -
tank guid 7470709504985343392 -
tank version - default
tank bootfs - default
tank delegation on default
tank autoreplace off default
tank cachefile - default
tank failmode wait default
tank listsnapshots off default
tank autoexpand off default
tank dedupditto 0 default
tank dedupratio 1.00x -
tank free 17.0T -
tank allocated 15.5T -
tank readonly off -
tank ashift 12 local
tank comment - default
tank expandsize - -
tank freeing 0 -
tank fragmentation 2% -
tank leaked 0 -
tank multihost off default
tank checkpoint - -
tank load_guid 11594792279076986393 -
tank autotrim off default
tank feature@async_destroy enabled local
tank feature@empty_bpobj active local
tank feature@lz4_compress active local
tank feature@multi_vdev_crash_dump enabled local
tank feature@spacemap_histogram active local
tank feature@enabled_txg active local
tank feature@hole_birth active local
tank feature@extensible_dataset active local
tank feature@embedded_data active local
tank feature@bookmarks enabled local
tank feature@filesystem_limits enabled local
tank feature@large_blocks active local
tank feature@large_dnode enabled local
tank feature@sha512 enabled local
tank feature@skein enabled local
tank feature@edonr enabled local
tank feature@userobj_accounting active local
tank feature@encryption enabled local
tank feature@project_quota active local
tank feature@device_removal enabled local
tank feature@obsolete_counts enabled local
tank feature@zpool_checkpoint enabled local
tank feature@spacemap_v2 active local
tank feature@allocation_classes enabled local
tank feature@resilver_defer enabled local
tank feature@bookmark_v2 enabled local
Pool filesystem details (zfs get all):
NAME PROPERTY VALUE SOURCE
tank type filesystem -
tank creation Sun Nov 17 12:18 2019 -
tank used 10.3T -
tank available 10.7T -
tank referenced 37.1G -
tank compressratio 1.01x -
tank mounted yes -
tank quota none default
tank reservation none default
tank recordsize 128K default
tank mountpoint /tank default
tank sharenfs off default
tank checksum on default
tank compression lz4 local
tank atime off local
tank devices on default
tank exec on default
tank setuid on default
tank readonly off default
tank zoned off default
tank snapdir hidden default
tank aclinherit restricted default
tank createtxg 1 -
tank canmount on default
tank xattr sa local
tank copies 1 default
tank version 5 -
tank utf8only off -
tank normalization none -
tank casesensitivity sensitive -
tank vscan off default
tank nbmand off default
tank sharesmb off default
tank refquota none default
tank refreservation none default
tank guid 6829761628355240197 -
tank primarycache all default
tank secondarycache all default
tank usedbysnapshots 0B -
tank usedbydataset 37.1G -
tank usedbychildren 10.3T -
tank usedbyrefreservation 0B -
tank logbias latency default
tank objsetid 51 -
tank dedup off default
tank mlslabel none default
tank sync disabled local
tank dnodesize legacy default
tank refcompressratio 1.02x -
tank written 37.1G -
tank logicalused 10.5T -
tank logicalreferenced 37.9G -
tank volmode default default
tank filesystem_limit none default
tank snapshot_limit none default
tank filesystem_count none default
tank snapshot_count none default
tank snapdev hidden default
tank acltype off default
tank context none default
tank fscontext none default
tank defcontext none default
tank rootcontext none default
tank relatime off default
tank redundant_metadata all default
tank overlay off default
tank encryption off default
tank keylocation none default
tank keyformat none default
tank pbkdf2iters 0 default
tank special_small_blocks 0 default
tank omvzfsplugin:uuid 753ded0e-697f-4d52-aec2-274b5c3f852d local