Nexentastor based build with ZFS raidz2 - slow write performance, need help

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

teq

New Member
Mar 22, 2012
9
0
1
Hi,

I have built my new media server over the last couple of weeks, and while I have thrown every piece of hardware at it that I was able to afford, I am not very satisfied with its performance - when I copy DVDs to this server, write performance still seems very slow. I would appreciate your thoughts on my build and what you would suggest that I should do to improve my write speeds.

HW:
- MB: Supermicro MBD-X9SCM-F-O
- CPU: Intel Xeon E3-1220
- RAM: 32 GB
- Host adapters: 2x LSI Logic LSI00194 Host Bus Adapter
- 1x system drive: Western Digital RE4 WD1003FBYX (connected to MB)
- 8x data drives: SEAGATE Constellation ES 7200 2TB SAS (connected to 1st host adapter)
- 2x logfile SSDs: Intel SSDSA2CW080G310 320 Series SATA (connected to 2nd host adapter)
- 1x cache SSD: Samsung MZ-5PA064/EU 64GB (connected to 2nd host adapter)

SW:
- VMware ESXi 5.1
- Nexentastor 3.1 as VM (with 20GB RAM)
- WHS2011 as VM (wit 8 GB RAM), connected to the Nexentastor storage pool through iSCSI

In Nexentastor, I have configured a raidz2 pool consisting of two volumes (4x 2TB drives each) and using the Intel SSDs for the logfile (mirrored) and the Samsung SSD as a cache device.

I had started this setup without the logfile SSDs, lower RAM (16GB) and without the cache SSD. However, upgrading with this (costly) hardware hasn't really improved my performance.

To be honest, I haven't accurately measured my write performance (I'm not really sure how to do that). So it's really just my "impression" how long it takes to copy a DVD rip from a client PC to the media server over the (1.000MB) network connection.

So I would really welcome any suggestions how to improve my performance.

Cheers,
teq
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Here is a quick test. Take a big file from your old machine and copy it to the server. Note how fast it gets transferred. Also what is the source disk?
 

teq

New Member
Mar 22, 2012
9
0
1
Here is a quick test. Take a big file from your old machine and copy it to the server. Note how fast it gets transferred. Also what is the source disk?
OK, I did that. Copying a directory with 6.7GB of video files from my Windows 7 PC to the media server took approximately 3 minutes 25 seconds. That calculates to roughly 21 MB/sec., right? Assuming my calculation is correct that's pretty terrible...

The source disk is a WD 1 TB EARS green drive - not very fast, but definitely not that slow.

*** Update: OK, I tried two additional tests:

1. To rule out the slow EARS drive, I copied the directory to my Windows 7 PC's system drive (SSD) first, and then repeated copying it to the media server. This time the copy only took approximately 1 minute and 48 seconds, which should calculate to roughly 70MB/sec. Much better. But still not great, right?

2. I also tried copying the same directory on the media server itself, i.e., copying it from the media servers system drive to the data drives. This took only 1 minute and 21 seconds, which would mean roughly 83MB/sec. (Not sure how relevant this test is, however.)

So if I read these results correctly, my Windows 7 PC's 1 TB EARS drives seems to be the main culprit for the poor performance when copying files to my media server? Does this result seem realistic?

And assuming these results are correct, the 70MB/sec. or even the 83MB/sec. write performance still doesn't seem great for the kind of hardware I am using?

Cheers,
teq
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE
There are too many options but think about:

- WD Ears are slow
- NexentaStor3 is based on a real old OpenSolaris 134 codebase, expect better values with V.4 (same Illumos codebase like OI)
- If you use a virtualized SAN, did you pass-through your Nic or did you use the ESXi VMXnet3 for best performance?
- If you share ZFS/NFS storage, have you compared the performance when you disable sync on the ZFS dataset?
- Intel 320 is slow for a ZIL and two ZIL's are useless (no improvement over one ZIL), disable sync to check if its worth at all.
- Did you use copy tools like Teracopy? They slow down transfers with ZFS
- Did you use higher MTU's or trunk network adapters: may cause problems
- Performance (I/O vs sequential) depends on vdevs (mirror vs raid-Z)
 
Last edited:

teq

New Member
Mar 22, 2012
9
0
1
There are too many options but think about:

- WD Ears are slow
- NexentaStor3 is based on a real old OpenSolaris 134 codebase, expect better values with V.4 (same Illumos codebase like OI)
- If you use a virtualized SAN, did you pass-through your Nic or did you use the ESXi VMXnet3 for best performance?
- If you share ZFS/NFS storage, have you compared the performance when you disable sync on the ZFS dataset?
- Intel 320 is slow for a ZIL and two ZIL's are useless (no improvement over one ZIL), disable sync to check if its worth at all.
- Did you use copy tools like Teracopy? They slow down transfers with ZFS
- Did you use higher MTU's or trunk network adapters: may cause problems
- Performance (I/O vs sequential) depends on vdevs (mirror vs raid-Z)
Hi gea,

sorry for my late reply. Yes, I am aware that there are many too options - we can rule out the WD EARS topic because I don't use this drive in my server (only in one client PC). Also, I'm not using any copy tools. Regarding some of your other points:
  • Nexentastor 3 old code base: I tried OI, but since I wasn't able to reconfigure the OI server version to use a fixed IP address (btw: I googled extensively on this topic, and it's amazing how difficult that seems to be in OI), so I used the OI GUI version with napp-it with VERY poor results.
  • NIC: No, I don't pass-through the NIC. * Update: I have installed the VMXnet3 drivers for both the WHS 2011 and the Nexentastor VMs (Nexentastore required an update to 3.1.3.5 for VMXnet3 to work). No change in write speeds, though. *
  • Disabling sync on the ZFS dataset - how do I do that? Note: I am using iSCSI to connect the WHS 2011 VM to the ZFS volume. The WHS 2011 handles the shares in the network.
  • Intel 320 for ZIL: My reason for using this SSD was that I read horror stories regarding data corruption in case of power failures when using SSDs for ZIL. The Intel 320 seems to be one of the few consumer priced SSDs that use supercapcitors to prevent data corruption in case of power failures. BTW, I am not using two ZILs - I am mirroring the two SSDs. Again, this was recommended in a number of ZFS related blogs (quote: "Since the ZIL is so important for ensuring data integrity, it should therefore be mirrored.")
Thank you very much,
teq
 
Last edited: