Intel RES2SV240 performance bottleneck

[Kelon]

I recently built a new server based on Supermicro's X11SSH-F motherboard, SC846TQ chassis, and an IBM M1015 (flashed to 9211 IT) connected to an Intel RES2SV240 expander. I am running Server 2012 Essentials R2. The Intel expander seems to be causing performance issues with the drives that are connected to it.

The expander is connected to the M1015 via a single SAS cable; 20 of the drive bays are connected to the expander, and the remaining 4 directly to the second port on the M1015. The drives that are directly connected to the controller have no issues, however all of the drives connected to the expander are showing 153 event IDs, corresponding with SRB_STATUS_COMMAND_TIMEOUT. During normal usage, the error doesn't come up very often, but when running a sync with SnapRAID, there are a significant amount of these errors. Reduced performance in certain circumstances is something I can live with, but I am concerned this could cause other issues down the line.

v12 of the RES2SV240 firmware indicates a fix for timeouts using SATA drives, but was released at least 4 years ago, so I am assuming my card with a pack date of 10/2015 is up to date. I am unclear how to identify the firmware version, however, or if I am even able to update it using an M1015 rather than the Intel controllers recommended for the update.

The fact that I need support for all 24 drives limits me somewhat, but I could pick up another M1015 and run 2 cables to the expander, or pick up a different expander that allows 24 ports with a dual link.

 

[rubylaser]

I would definitely try to update the firmware as my card (bought secondhand) had a PH9 firmware on it. Updating the firmware to PH13 almost eliminated the timeout errors. As a sidenote, I completely eliminated my m1015 + Intel RES2SV240 in my Norco 4224 and moved to (3) Dell H310's in IT mode (this is in Ubuntu 16.04 with SnapRAID + mergerFS). This has been both faster and completely eliminated the timeout errors.

 

[pricklypunter]

I agree, the way to go is to remove the expander from the equation altogether if possible and roll with 3 HBA's, not only will it alleviate most if not all the little niggles, but you should see a performance boost as well. Either way, short term, you should definitely look into a firmware upgrade

 

[PigLover]

What firmware rev are you running on the M1015?

Sent from my SM-G925V using Tapatalk

 

[rubylaser]

What firmware rev are you running on the M1015?

Sent from my SM-G925V using Tapatalk

I was having issues with both P19 and P20 on mine until updated the expander firmware on mine.

 

[Kelon]

Thanks for the feedback guys. I figured there would be some degree of a performance hit with the expander, but didn't expect so many IO timeouts during heavy access. I am running P20 on the M1015, since the Server 2012 drivers seem to be designed for that version. I will definitely try updating the firmware, hopefully that will alleviate some of the issues I am having in the short-term.

If a firmware update doesn't significantly reduce or eliminate the problem, I am just going to replace the expander altogether, possibly with something like the 9305-24i, since it seems like it would solve my issues as well as leave a couple slots free for future expansion.

 

[PigLover]

There have been several revs of P20. Make sure you have the most recent.

Surprisingly, you might find downgrading to P19 could actually help. The exact symptom you describe (timeouts when connected to an expander under Windows) has been reported with P20 but not P19. I don't know if the later versions of P20 fixed this one or not.

Sent from my SM-G925V using Tapatalk

 

[BackupProphet]

Could it be the hard drive firmware causing these timeouts?

 

[Kelon]

There have been several revs of P20. Make sure you have the most recent.

Surprisingly, you might find downgrading to P19 could actually help. The exact symptom you describe (timeouts when connected to an expander under Windows) has been reported with P20 but not P19. I don't know if the later versions of P20 fixed this one or not.

I do have the latest version of P20, but I will definitely try downgrading to the P19 firmware and driver if updating the expander's firmware doesn't resolve anything. Thanks for the suggestion.

2nd what everyone else said, better to run 3 m1015's by themselves than use the expander. My personal prefernce is 3x dell h310 too. I wonder why didnt you have both sff ports going from the m1015 to the expander for double the bandwidth?

I have a second chassis I plan to expand to at some point, so 3 M1015's would take up too many slots, but if I can't resolve the errors with the current hardware, I am going to look into a single card with more ports, such as the 9305-24i, and just forego the expander. I'm comfortable with a performance hit (as long as it's not night and day), but not so much with timeout errors. The only reason I didn't run two cables to the expander is that I would need a second M1015 to get back to 24 drives, but if nothing else helps, I will try dual link to see if it makes any difference.

Could it be the hard drive firmware causing these timeouts?

It's possible, they're all 4TB HGST SATA drives. Maybe something with regard to how the expander communicates with the M1015, combined with the drives being SATA?

 

[RyC]

I'm currently running an M1015 on P19 IT firmware with the latest RES2SV240 firmware with SnapRAID on Windows 7 and am not encountering any issues (in the log or otherwise). I have a mishmash of SATA WD and Toshiba drives

 

[pricklypunter]

If they are genuine HGST 4TB disks with HGST firmware on them, I seriously doubt that they are causing your issue, well, not unless you have a faulty one in the mix of course. HGST disks are pretty much as reliable as the day is long

 

[Kelon]

I'm currently running an M1015 on P19 IT firmware with the latest RES2SV240 firmware with SnapRAID on Windows 7 and am not encountering any issues (in the log or otherwise). I have a mishmash of SATA WD and Toshiba drives

Good to know - hopefully I will have similar experience later today or tomorrow morning when I have time to sit down and work on this some more.

If they are genuine HGST 4TB disks with HGST firmware on them, I seriously doubt that they are causing your issue, well, not unless you have a faulty one in the mix of course. HGST disks are pretty much as reliable as the day is long

Yep, they're all retail boxed 4TB HGST drives. I am still psychologically traumatized from the 3TB Seagates in my last drive pool, so short of buying enterprise class drives, I tried to cut as few corners as possible on the hard drives this time around. It's definitely not a faulty drive, as the issue only occurs with the drives that are connected through the expander. I've swapped cables around to see if I could isolate either a cabling issue or a problematic drive, and everything is fine on all drives under heavy activity, as long as the expander isn't involved.

 

[Kelon]

I just wanted to thank everyone in the thread for their advice - everyone here has been extremely helpful, and I've definitely picked up a few suggestions on hardware that I will take into account for future upgrades.

I am happy to report that the IO errors have been resolved, although the end result is a bit puzzling. It turned out to be the M1015 itself. Updating the motherboard and expander BIOS yielded no better results. My next step was to downgrade to P19, but since the 2nd M1015 I had ordered to expand to a 2nd chassis in the future showed up this morning, I decided to try it first. I flashed it with the same P20 9211-8i firmware I had used on the other card, installed it, and have been able to sync and run burst tests on the drive with no errors. My sync speed in SnapRAID has also doubled by switching to the other card, still only using a single cable to the expander.

The funny thing is, the older card only generated errors when connected to the expander, regardless of which port I used, and with multiple cables - never when direct connected to 4 drives via breakout cable. There's either something odd with the older M1015, or the x16 slot I had installed it in (the newer card is in the adjacent x8 slot). I'm going to move the newer card to the x16 slot later just to rule that out, but my plan for now is just to relegate the M1015/RES2SV240 to a secondary array in the future and upgrade the primary controller at that point.

A quick ebay check shows the 24i going for over $600, while the 12gb hp expander goes for about $350. At least you can save a couple dollars my friend.

LSI Avago SAS 9305-24I 05-25699-00 12Gb/s SAS / SATA Host Bus Adapter (NEW)

HP 769635-B21 12Gb SAS Expander Card for ML350 Gen9

Hope you get your setup working great again!

Thanks for the advice. My main reservation on the 24i was the cost, and this looks to be a much better value at slightly better than half the cost.


Update: I tested the newer M1015 in the x16 slot, again no IO errors. Looks like it's 100% the older card.