Slow FC R/W Comstar

[[RXP]Andy]

Afternoon Chaps,

Ive been a bit of a long time lurker on these forms. However, I have a bit of a interesting problem with Comstar specifically using FCs. I seem to be that I am hitting a limit with the GC HBA of around 195MB/s, which is itself isn't bad but as you can see from the DD and Bonnie benchmarks are capable of much more. In theory I should be expecting around 350 MB/s ~ from a 4GB HBA, but as it currenly stands is a little bit short of that. Both of the volumes are running in RAID Z

  pool: vol_data
 state: ONLINE
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM     CAP            Product
	vol_data                   ONLINE       0     0     0
	  raidz1-0                 ONLINE       0     0     0
	    c3t5000C5004E31AC95d0  ONLINE       0     0     0     2 TB           ST2000DM001-1CH1
	    c3t5000C50051D8D402d0  ONLINE       0     0     0     2 TB           ST2000DM001-9YN1
	    c3t5000C50051E4E37Ad0  ONLINE       0     0     0     2 TB           ST2000DM001-9YN1
	    c3t5000C50051E51DD0d0  ONLINE       0     0     0     2 TB           ST2000DM001-9YN1
	    c3t5000C5005CB67BACd0  ONLINE       0     0     0     2 TB           ST2000DM001-9YN1

errors: No known data errors

  pool: vol_ssd
 state: ONLINE
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM     CAP            Product
	vol_ssd                    ONLINE       0     0     0
	  raidz1-0                 ONLINE       0     0     0
	    c3t50025388A01D3DF1d0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840
	    c3t50025388A01D411Dd0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840
	    c3t50025388A01D417Fd0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840
	    c3t50025388A01D4181d0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840
	    c3t50025388A01D4182d0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840
	    c3t50025388A01D707Ed0  ONLINE       0     0     0     250.1 GB       Samsung SSD 840

Ive been testing OmniOS and OpenInidana, which I have been testing and they have shown to be very close as you can see from the benchmarks shown below. Has anyone else experienced a FC Comstar issue like this?

System Spec:

SuperMicro X9SCM-F
Intel Xeon E3 1220V2
Qlogic 2464
2 X LSI 9207-8i
1 x Cruical M500 120 GB - Boot Drive
5 X Seagate 2TB (VOL_DATA)
6 X Samsung Evo 250GB SSD (VOL_SSD)

DD Bench

vol_data - SATA - 40.96 GB in 73.9s = 554.26 MB/s Write / 40.96 GB in 73.4s = 558.04 MB/s Read
vol_SSD - SSD - 40.96 GB in 24.4s = 1678.69 MB/s Write / 40.96 GB in 18.8s = 2178.72 MB/s Read

Bonnie

vol_data - SATA

NAME SIZE Bonnie Date(y.m.d) File Seq-Wr-Chr %CPU Seq-Write %CPU Seq-Rewr %CPU Seq-Rd-Chr %CPU Seq-Read %CPU Rnd Seeks %CPU Files Seq-Create Rnd-Create

vol_data 9.06T start 2014.02.24 48G 182 MB/s 98 587 MB/s 97 202 MB/s 25 136 MB/s 89 493 MB/s 21 1165.1/s 3 16 +++++/s +++++/s

vol_SSD - SSD - TBC

Windows 2012 Host (MPIO Installed) - Qlogic QLE 2462

OmniOS

-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   195.023 MB/s
          Sequential Write :   194.361 MB/s
         Random Read 512KB :   181.632 MB/s
        Random Write 512KB :   145.491 MB/s
    Random Read 4KB (QD=1) :    26.070 MB/s [  6364.8 IOPS]
   Random Write 4KB (QD=1) :    20.265 MB/s [  4947.6 IOPS]
   Random Read 4KB (QD=32) :   190.597 MB/s [ 46532.6 IOPS]
  Random Write 4KB (QD=32) :   187.496 MB/s [ 45775.5 IOPS]

  Test : 100 MB [D: 0.0% (0.2/2047.9 GB)] (x5)
  Date : 2014/02/24 14:09:54
    OS : Windows NT 6.2 Datacenter Edition (Full installation) [6.2 Build 9200] (x64)

Open Inidana

-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   193.673 MB/s
          Sequential Write :   192.930 MB/s
         Random Read 512KB :   179.408 MB/s
        Random Write 512KB :   145.251 MB/s
    Random Read 4KB (QD=1) :    24.180 MB/s [  5903.3 IOPS]
   Random Write 4KB (QD=1) :    20.334 MB/s [  4964.3 IOPS]
   Random Read 4KB (QD=32) :   189.240 MB/s [ 46201.2 IOPS]
  Random Write 4KB (QD=32) :   193.320 MB/s [ 47197.3 IOPS]

  Test : 100 MB [D: 0.0% (0.2/2047.9 GB)] (x5)
  Date : 2014/02/24 15:31:26
    OS : Windows NT 6.2 Datacenter Edition (Full installation) [6.2 Build 9200] (x64)

Thanks

 

[[RXP]Andy]

I think I have found the culprit, I have a FC HBA installed in a X1 slot on the Windows Server. Ill test this out later tonight.

From the Qlogic Tools

PCIe Attribute Name PCIe Attribute Value 
Max Bus Width:  x4 
Negotiated Width:  x1 
Max Bus Speed:  2.5 Gbit/s 
Negotiated Speed:  2.5 Gbit/s

Moving the card to another slot cured the issuse, but I seem to be hitting the limits of the card now.

-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 616.205 MB/s
Sequential Write : 540.863 MB/s
Random Read 512KB : 329.204 MB/s
Random Write 512KB : 303.809 MB/s
Random Read 4KB (QD=1) : 31.695 MB/s [ 7738.1 IOPS]
Random Write 4KB (QD=1) : 23.102 MB/s [ 5640.2 IOPS]
Random Read 4KB (QD=32) : 318.264 MB/s [ 77701.0 IOPS]
Random Write 4KB (QD=32) : 148.701 MB/s [ 36304.0 IOPS]

Test : 100 MB [E: 0.0% (0.2/749.9 GB)] (x5)
Date : 2014/02/25 9:57:30
OS : Windows NT 6.2 Datacenter Edition (Full installation) [6.2 Build 9200] (x64)