Performance Issues (Throughput)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
K

kronik

New Member
Jan 19, 2013
6
0
0
Posted this over on [H] as well..

So I finally managed to purchase all of the gear I needed and build my fileserver.

As expected, it's not behaving as expected.

Pertinent info:
ESXi 5.1
Gigabyte Z77N-Wifi
Realtek (sadly) GigE
Solaris 11
Intel 3770
7x HGST 7k4000
16GB of RAM
Direct passthrough of a M1015
ESXi running on Intel X25

Currently assigned to Napp-it box is 8GB of memory, 4 vCPU and a VMXNET3 NIC. E1000 did not resolve the issue. RAIDZ2 with all 7 disks. Samsung Force GT 64GB as L2ARC. I'll probably pare down the virtual resources after resolving the issues I'm having of which there are a couple:

First and foremost, what type of performance should I be expecting out of this configuration? Currently seeing approximately 40MB/s sequential reads and 70MB/s sequential writes in CrystalDiskMark. This seems abysmally low. These benchmarks are being run, also over GigE from varying vendors (Intel, Realtek, Broadcom).

In my initial testing tonight, I tried d/ling a torrent (legally) that resulted in Disk Overload at 100%? This was over SMB. I increased my uTorrent cache but that only helped temporarily. Subsequently, download speeds dropped to 7kb/s. I suspect this ties back into the preceding issue.

Since I'm running Solaris 11, I can't use Bonnie so here's DD bench:

Memory size: 8192 Megabytes

write 16.777216 GB via dd, please wait...
time dd if=/dev/zero of=/vdev1/dd.tst bs=2048000 count=8192

8192+0 records in
8192+0 records out

real 1:21.8
user 0.0
sys 5.3

16.777216 GB in 81.8s = 205.10 MB/s Write

wait 40 s
read 16.777216 GB via dd, please wait...
time dd if=/vdev1/dd.tst of=/dev/null bs=2048000

8192+0 records in
8192+0 records out

real 22.6
user 0.0
sys 4.3

16.777216 GB in 22.6s = 742.35 MB/s Read
 
S

sotech

Member
Jul 13, 2011
305
1
18
Australia
Any possibility of chucking in an Intel NIC? I've seen non-Intel NICs with worse max performance than that before. Even if you had to take out the M1015 and create a test separate Solaris 11 VM on the SSD just for the purpose of testing...

Curious choice of board - why that over something like a S1200KP?

Edit: Also, any difference between accessing the server via NFS vs. SMB?
 
Last edited:
K

kronik

New Member
Jan 19, 2013
6
0
0
sotech said:
Any possibility of chucking in an Intel NIC? I've seen non-Intel NICs with worse max performance than that before. Even if you had to take out the M1015 and create a test separate Solaris 11 VM on the SSD just for the purpose of testing...

Curious choice of board - why that over something like a S1200KP?

Edit: Also, any difference between accessing the server via NFS vs. SMB?
Click to expand...
I suppose that's a possibility though I'm not sure what it would accomplish given my footprint is what it is.

The S1200KP doesn't allow for VT-D, correct? If it had, it's definitely the board I would have purchased.

I can test over NFS/AFP later today.
 
S

sotech

Member
Jul 13, 2011
305
1
18
Australia
kronik said:
I suppose that's a possibility though I'm not sure what it would accomplish given my footprint is what it is.

The S1200KP doesn't allow for VT-D, correct? If it had, it's definitely the board I would have purchased.

I can test over NFS/AFP later today.
Click to expand...
Ahh yes, you are right - no VT-D there, which is an irritating omission for that board. Probably overkill to create a separate VM for testing with such a new install I suppose... I habitually use a spare VM to test with so I don't worry about trashing something useful but with a fresh install it probably doesn't matter so much.
 
K

kronik

New Member
Jan 19, 2013
6
0
0
sotech said:
Ahh yes, you are right - no VT-D there, which is an irritating omission for that board. Probably overkill to create a separate VM for testing with such a new install I suppose... I habitually use a spare VM to test with so I don't worry about trashing something useful but with a fresh install it probably doesn't matter so much.
Click to expand...
Any other ideas?
 
K

kronik

New Member
Jan 19, 2013
6
0
0
NFS/AFP seemed better but before I really got a chance to dive into that testing..


The hits just keep on coming with this thing.

Things that I've observed - if I run CrystalDiskMark from a VM also on the same ESXi host, the results are fine. dd bench is still showing around 200MB/s write/500MB/s read.
I determined that uTorrent is a POS but the ZIL + increasing the disk cache largely remedied the Disk Overloading problem though now that I'm having this other issue, it seems to have resurfaced.. and that is:

The server just goes unstable and essentially needs to be rebooted. The only thing that I can see that looks troublesome is the following in the system log:
Code: 
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:33:34 napp-it-box      /scsi_vhci/disk@g5000cca22bc29860 (sd4): Command Timeout on path mpt_sas2/disk@w5000cca22bc29860,0
Jan 26 06:33:34 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      Disconnected command timeout for Target 12
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:33:34 napp-it-box      /scsi_vhci/disk@g5000cca22bc15e82 (sd3): Command Timeout on path mpt_sas3/disk@w5000cca22bc15e82,0
Jan 26 06:33:34 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      Disconnected command timeout for Target 11
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:33:34 napp-it-box      /scsi_vhci/disk@g5000cca22bc1fab5 (sd9): Command Timeout on path mpt_sas5/disk@w5000cca22bc1fab5,0
Jan 26 06:33:34 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      Disconnected command timeout for Target 15
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:33:34 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:33:34 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:33:34 napp-it-box      /scsi_vhci/disk@g5000cca22bc2a3ac (sd6): Command Timeout on path mpt_sas8/disk@w5000cca22bc2a3ac,0
Jan 26 06:34:44 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      Disconnected command timeout for Target 16
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:34:44 napp-it-box      /scsi_vhci/disk@g5000cca22bc29860 (sd4): Command Timeout on path mpt_sas2/disk@w5000cca22bc29860,0
Jan 26 06:34:44 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      Disconnected command timeout for Target 12
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:34:44 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      Disconnected command timeout for Target 13
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:34:44 napp-it-box      /scsi_vhci/disk@g5000cca22bc201fa (sd5): Command Timeout on path mpt_sas6/disk@w5000cca22bc201fa,0
Jan 26 06:34:44 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      Disconnected command timeout for Target 15
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:44 napp-it-box      mptsas_check_scsi_io: IOCStatus=0x48 IOCLogInfo=0x31130000
Jan 26 06:34:44 napp-it-box scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jan 26 06:34:44 napp-it-box      /scsi_vhci/disk@g5000cca22bc2a3ac (sd6): Command Timeout on path mpt_sas8/disk@w5000cca22bc2a3ac,0
Jan 26 06:34:54 napp-it-box scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:54 napp-it-box      MPTSAS Firmware Fault, code: 265d
Jan 26 06:34:55 napp-it-box scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
Jan 26 06:34:55 napp-it-box      mptsas0 Firmware version v9.0.0.0 (?)
Jan 26 06:34:55 napp-it-box scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3040@0 (mpt_sas0):
I checked all of the connections.. ugh. Just ordered some new SFF8087 breakout cables an external drive to migrate all of my stuff off of..
 
You must log in or register to reply here.
Top Bottom