SAS2008 HBA in IR mode for RAID 1, expected performance?

[logan893]

Does anyone have experience with SAS2008 based HBA in IR mode with at least two HDDs in RAID 1? Two SSDs in RAID 1? What performance do you have?

I have an LSI 9212-4i4e (SAS2008) in IR mode (P20 FW), currently with 2x Western Digital RE 320 GB drives (first gen I believe) I had left over from an old build.

I first tested them as a VMware datastore (ESXi 6.0), several months ago. Even after they seemed to have finished all RAID 1 initialization (after initialization they were silent during idle, and I let them sit for hours, even days), I was still getting shitty sequential writes (4.5 MB/s writes with dd bs=1m from /dev/zero; 37 MB/s read), and they were audibly (and constant) thrashing during writes.

That was a few months ago. I haven't touched it since, except to change it to pass-through mode. I don't recall if and what performance I saw in VT-d based pass-through.

I did another test tonight, with the HBA in pass-through mode to a FreeBSD VM, and it behaved somewhat better. I reach 12 MB/s sequential write speed with dd (both from /dev/zero and /dev/random) to the device in FreeBSD (unbuffered writes), and 64 MB/s read, from the beginning of the disk. No thrashing for sequential writes. This is more acceptable.

Is the difference down to something that VMware does?

Maybe tomorrow I'll test the VMware datastore performance via an ESXi 6.0 VM, as I don't want to restart the server right now, only to switch the HBA out of pass-through mode.

These are pretty old drives, purchased over 8 years ago, so I don't expect stellar performance. As a reference, a newer, single Western Digital RE4 as a datastore can achieve approximately 18 MB/s sequential writes with dd, even now when it's half full (of non-important data.)

Now I'm looking at options to get it working somewhat better for RAID 1 VM stores, or replace the HBA with a reasonably priced flash cache+supercap RAID controller.

HP P410 512MB FBWC looks interesting, and very cost effective, even if it lacks SATA 3 (and PCIe 3.0) support.

 

[BackupProphet]

That sounds really bad. The Dell H310 that I have flashed to IT-mode runs full speed on seq read/write (150MB/s). I do remember some old drives that I had that performed really bad, like 10MB/s for seq write. Turns out that there was something wrong with them and I got them replaced.

 

[logan893]

Yes, this truly is horrible performance. For me, it appears that an HBA in IR mode for HDDs in RAID 1 in VMware ESXi will at best be useful for reads of mostly-static data.

These drives are not the most amazing, for sure. With VMware basically all write caching switched off in the OS. I can only assume that the drive thrashing is due to writing of data, followed by writing of metadata, and repeat.

I loaded up an ESXi 6.0U2 test VM, and again I see up to a maximum of 4.5 MB/s writes (dd to file), and the drives are thrashing. Why performance is somewhat better in FreeBSD is possibly due to soft updates of metadata.

Writing a 150 MB to a file with dd takes roughly 35 seconds. Reading the same amount takes only 4-5 seconds.
Block size for dd is mostly irrelevant, I ran the 150 MB write and read tests with block sizes from 4k to 10M.

 

[Jannis Jacobsen]

My experience is that hba for raid needs cache + bbu to have decent performance.
non-cache hba is only good for passthrough for zfs, storage spaces etc.

-Jannis

 

[T_Minus]

Just a note... IR mode is not "crippling" your performance of single disks down to those speeds itself.
I have IR mode currently on my 3008 and have tested it with HGST 12Gb/s SSD and achieve over 100,000 IOPs and around 900MB/s in ZFS, slightly faster of course just passed through to VM direct or used direct in ESXI.

 

[logan893]

An SSD is going to perform much better than an HDD in any situation, and for sure with this kind of simple HBA RAID in-between. Your 3008 in IR mode and HGST 12 Gb/s SSD are also likely performing much better than my old SAS2008 and older WD RE HDDs.

I haven't found any way to confirm what settings are used, but I believe the HBA in IR mode will, when using RAID 1, disable the HDD's own cache, just as a MegaRAID controller will by default even with write cache and BBU, to prevent inconsistency and data loss. An SSD will perform much better in this situation than an HDD. And some high-end server disks may even ignore the disabling of its own cache, if they have built-in protection in the form of supercaps.

Over at the FreeNAS forum, I found that jgreco has had a similar experience as myself with IR mode and RAID 1 using spinning disk media, around three years ago.

forums.freenas.org: LSI SAS2008 IT vs IR Firmware

Flashing to IT or IR mode gets rid of lots of crap. In IR mode, the most readily apparent difference (between IR and IT) is that in the BIOS setup utility, a new line item appears to allow for setting up basic RAID functionality. I just tossed an IR controller in a FreeNAS box to confirm that drives show up under the mps driver, but a RAID 1 volume does show up as "LSI Logical Volume 3000". Oh and yikes about a 3.5MB/sec write rate (the individual disks are around 60MB/sec). Yow, even the RAID1 read sucks, 33MB/sec (disks around 70MB/sec).

Always fun getting a slow little PowerPC core involved in the data path.

I bought an HP P410 with 1GB FBWC via eBay the other day, which should arrive tomorrow or early next week. I'll rebuild my RAID 1 with this card once it's in my hands, and do some additional tests.

 

[logan893]

SAS 9212-4i4e Host Bus Adapter Documentation, SAS-2 IR Solution User Guide (Direct link: SAS-2 Integrated RAID Solution User Guide (975 KB))

2.4.6 Disk Write Caching
By default, the Integrated RAID firmware disables disk write caching for mirrored volumes to make sure that the write journal entry stored in nonvolatile static RAM (NVSRAM) is always valid. If you enable disk write caching (not recommended), you might cause the disk write log to be invalid.

 

[T_Minus]

An SSD is going to perform much better than an HDD in any situation, and for sure with this kind of simple HBA RAID in-between. Your 3008 in IR mode and HGST 12 Gb/s SSD are also likely performing much better than my old SAS2008 and older WD RE HDDs.

I haven't found any way to confirm what settings are used, but I believe the HBA in IR mode will, when using RAID 1, disable the HDD's own cache, just as a MegaRAID controller will by default even with write cache and BBU, to prevent inconsistency and data loss. An SSD will perform much better in this situation than an HDD. And some high-end server disks may even ignore the disabling of its own cache, if they have built-in protection in the form of supercaps.

Over at the FreeNAS forum, I found that jgreco has had a similar experience as myself with IR mode and RAID 1 using spinning disk media, around three years ago.

forums.freenas.org: LSI SAS2008 IT vs IR Firmware


I bought an HP P410 with 1GB FBWC via eBay the other day, which should arrive tomorrow or early next week. I'll rebuild my RAID 1 with this card once it's in my hands, and do some additional tests.

I didn't post to tell you a SSD was better, that's obvious.
I was only trying to let you know that IR mode vs IT mode only took a very minor perf. hit... and was letting you know what hardware was utilized.

Do with the info what you will, but i'm not trying to convince you to go one way or another.

 

[logan893]

I didn't post to tell you a SSD was better, that's obvious.
I was only trying to let you know that IR mode vs IT mode only took a very minor perf. hit... and was letting you know what hardware was utilized.

Do with the info what you will, but i'm not trying to convince you to go one way or another.

You are right, and I realize you were addressing single drive performance. In which case IR and IT mode are likely very similar, perhaps even identical, performance wise. Please don't get me wrong, your input is appreciated!

I was specifically interested in the IR mode RAID 1, and after some research I believe I found the main reason for why performance is so poor (disk cache disabled).

What I've also discovered is that it should be possible to re-enable the disk cache. Which may be of interest to me if/when I get my hands on some supercap SSDs, for example a pair of Intel DC S3700, for use with this controller. It would be a waste to put them together with the HP P410, as the SSDs are SATA 6Gbps while the P410 controller is limited to 3Gbps, and neither read-ahead nor write-back would be necessary. And I also do not yet know if I can disable read-ahead and write-back cache, and enable disk cache, on one RAID 1 volume with SSDs, while keeping it reversed for another RAID 1 volume with HDDs.

 

[aero]

I was specifically interested in the IR mode RAID 1, and after some research I believe I found the main reason for why performance is so poor (disk cache disabled).

I am guessing you won't see any significant improvement in large sequential workloads with the local disk cache enabled. It should certainly help for random io though. Very interested to see your results.

 

[jgreco]

You are right, and I realize you were addressing single drive performance. In which case IR and IT mode are likely very similar, perhaps even identical, performance wise. Please don't get me wrong, your input is appreciated!

For single drive performance, IR is somewhat slower than IT, which is the reason it is suggested to flash these things to IT for ZFS rather than put them in the more flexible IR mode.

Observed about 10-15% speed penalty for IR vs IT here in the past, but that was years ago, and I don't know if we've tested on more recent firmware phases.

I was specifically interested in the IR mode RAID 1, and after some research I believe I found the main reason for why performance is so poor (disk cache disabled).

What I've also discovered is that it should be possible to re-enable the disk cache. Which may be of interest to me if/when I get my hands on some supercap SSDs, for example a pair of Intel DC S3700, for use with this controller. It would be a waste to put them together with the HP P410, as the SSDs are SATA 6Gbps while the P410 controller is limited to 3Gbps, and neither read-ahead nor write-back would be necessary. And I also do not yet know if I can disable read-ahead and write-back cache, and enable disk cache, on one RAID 1 volume with SSDs, while keeping it reversed for another RAID 1 volume with HDDs.

An IR mode HBA under ESXi is going to have poor performance primarily due to ESXi's sync write handling, but other factors can impact this as well, as you've discovered. Your 4.5MB/sec is slightly poorer than I would expect, usually more around 10-12MB/sec for HDD writes and around 80-120MB/sec for SSD writes to RAID 1 virtual disks seems to be in the ballpark for us here.

If you are dependent on the low cost/high capacity of HDD, there are lots of options on eBay for used LSI RAID controllers that include BBU or supercap at a relatively reasonable cost, and these will accelerate your HDD datastore very noticeably.

 

[logan893]

I am on a budget for this system, and I do want to use the equipment I have, where possible. Such as these two WD RE drives. It's also as much a learning experience, as it is a useful tool to host e-mail, media, games, etc.

I've received the HP P410 with 1GB FBWC (912 MB usable cache it says) and I'm testing it a bit now in a Windows 10 VM (VT-d pass-through of the device).

Hurdles
Booting with the Option ROM setting enabled in BIOS results in boot loop. I reach the HP controller "initializing" screen, and after a few seconds the system reboots. Unclear if I can find a way to have Option ROM enabled, however as I am not booting from the RAID controller, with ESXi booting from USB, I do not need this.

I have a Fujitsu branded server. The boot loop may be impacted by this fact. Controller firmware version seem to also impact this. This P410 firmware version is 6.40.

Having a Fujitsu branded server also makes loading the administration software in ESXi impossible, without modification of the VIB file. I'll rebuild the VIB and try later.

I first configured the controller for pass-through and hooked it up to a FreeBSD 10.3 VM. During boot of FreeBSD, the ciss device driver was stuck for a long time, and eventually threw an error.
ciss0: error sending BMIC ID_CTLR command (35)

It finally reached the OS login but the device did not seem to be working properly. I manually added "hw.ciss.force_interrupt=1" to /boot/loader.conf (as suggested by a 3 year old DragonFly BSD bug report Bug #2585: Dfly 3.4.3 on ESXi 5.1, HP Smart Array P410 passthrough recognised, but not functioning - DragonFlyBSD - DragonFlyBSD bugtracker) and rebooted the VM. This did not help, it was still stuck during driver initialization, and finally I opted to power-off the VM, which caused a kernel panic on the host VMware ESXi system. Not sure if it's related to manually adding "hw.ciss.force_interrupt=1" to /boot/loader.conf, the power-off action, the failure to initialize the device twice in a row, or a combination.

I also hooked it up to a Ubuntu VM, where it also seemed to have some issues, and also screwed up the volume group info and mounting of the boot drive, so I abandoned those two for now.

Pass-through to Windows 10 works just fine.

I added the two drives to a RAID 1 array, and it even showed me the GPT partition I had previously created on the RAID 1 array from when the drives were hooked up to the SAS2008 HBA in IR mode. The array seem to be compatible between the two, but I could not verify any data on the partition, as I had left it empty. No additional reinitialization seem to have been triggered either. Tests below were performed straight away.

Performance
I set the cache ratio to 15/85. I'm not quite sure if I really need any read cache, but neither should I really need 775 MB write cache during normal operation.

I enabled "wait for cache room" (pauses writes when write-back cache is full). Drive write cache I left at disabled, and the setting is only on controller level, so I cannot mix a disabled-for-HDD with enabled-for-SSD-with-supercaps (and as of yet unclear if the SSDs even benefit from it).

When the write cache is not full, write performance is maxed out at 1.1 GB/s as tested with ATTO (overlapped IO, QD4, 256 MB total length, Direct IO active), for blocks 64 kB and up. A quick analysis of sub-64kB block sizes gives me some crude buffered write iops estimates for this HP P410. From what I've read this P410 controller should handle 50k IOPS max for random 4kB writes. I reach over 45k iops for 4kB writes at QD4, which is not too shabby.

The card is mounted in a PCIe 3.0 x8 slot (mechanical and electrical), is itself PCIe 2.0 x8, but the HP data sheet specifies 2 GB/s, and I reach just over half of that.

Steady-state sequential read speeds during high write load seem to be 18-36 MB/s on the low end, and with a low write load they can reach 60-90 MB/s on the high end.

 

[logan893]

I rebuilt the VIB and installed hpacucli from version 1.4 of the HP utils.

Output from command /opt/hp/hpacucli/bin/hpacucli :

HP Array Configuration Utility CLI 9.40.12.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

=> ctrl all show config detail

Smart Array P410 in Slot 2
   Bus Interface: PCI
   Slot: 2
   Serial Number: PACCR9SZ4AL0
   Cache Serial Number: PBCDF0CRH0VAXO
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 6.40
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Surface Scan Mode: Idle
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Enabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 15% Read / 85% Write
   Drive Write Cache: Disabled
   Total Cache Size: 1024 MB
   Total Cache Memory Available: 912 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SATA
      Unused Space: 0  MB
      Status: OK
      Array Type: Data



      Logical Drive: 1
         Size: 298.1 GB
         Fault Tolerance: 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C8493CC219CF57A88CA1B
         Disk Name: vmhba1:C0:T0:L1
         Mount Points: None
         Logical Drive Label: A8F57BCBPACCR9SZ4AL0B7E2
         Mirror Group 0:
            physicaldrive 1I:0:1 (port 1I:box 0:bay 1, SATA, 320.0 GB, OK)
         Mirror Group 1:
            physicaldrive 1I:0:2 (port 1I:box 0:bay 2, SATA, 320.0 GB, OK)
         Drive Type: Data

      physicaldrive 1I:0:1
         Port: 1I
         Box: 0
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 320.0 GB
         Firmware Revision: 21.00M21
         Serial Number:      WD-WCAPD4945791
         Model: ATA     WDC WD3200YS-01P
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 39
         Maximum Temperature (C): 48
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps

      physicaldrive 1I:0:2
         Port: 1I
         Box: 0
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 320.0 GB
         Firmware Revision: 21.00M21
         Serial Number:      WD-WCAPD4520338
         Model: ATA     WDC WD3200YS-01P
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 37
         Maximum Temperature (C): 53
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps


   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250
      Device Number: 250
      Firmware Version: RevC
      WWID: 500143800975961F
      Vendor ID: PMCSIERA
      Model:  SRC 8x6G

=>

 

[logan893]

I ran some quick tests with a pair of Intel SSD DC S3700 400GB drives in RAID 1, using HP P410 and LSI SAS2008 9212-4i4e. Arrays were local to ESXi, with vmfs datastore and regular vmdk files. Benchmarked using diskspd from a Windows 10 VM.

HP P410 - stripe size 8 kB (also tested 256 kB (default), with similar performance):
IOPS: Caps at around 22k IOPS for reads, and latency is a bit higher than I would like (both average and highest). Writes max out at 10-11k when write buffer is full. Turning off write cache completely bumps this to up to 14-15k during 100% writes of both 4 and 8 kB. Enabling disk write cache does nothing, as I would expect of these Intel SSDs.
Throughput: Doesn't seem to read from both drives in the RAID 1 array simultaneously. Maxes out around 300 MB/s (SATA II speed of single drive).

LSI SAS2008 9212-4i4e - stripe size 64 kB:
IOPS: Close to 100k IOPS (95-97k) during 100% reads of 4 kB benchmark. 75k IOPS at 8kB. Writes up to 35-40k IOPS. No option to enable disk write cache, but it should make no difference.
Throughput: Scales up to around 900 MB/s for reads, fully utilizing both drives and almost saturating the SATA III interface. Writes are quite a bit slower, as expected.


LSI SAS2008 9212-4i4e is the way to go for SSDs, if only I can figure out my other issues with seemingly invisible background tasks and high latency during certain controller tasks.
LSI SAS2008, IR mode, RAID 1 with two S3700; poor performance for a while after reboot