SAS2 expanders $60 (IBM, LSI chip, Intel alternative)

[BLinux]

Just received this expander with 510A firmware and its attached to a Dell H730 card, I want to update the firmware on it through windows if its possible, downloaded firmware from ibm site, extracted and executed install.bat from "image" folder and I get the following error

Flash:Failed rc=99h
Error:Flash Level Mismatch: Drive Ver:510A DBase Ver:634A!

where did you download the firmware?

 

[SIlviu]

where did you download the firmware?

IBM Site

IBM 6Gb PCI-E SAS Expander Daughter Card Firmware Update
ibm_fw_exp_6gb-sas-634a_windows_32-64.exe (1.35 MB)

 

[BLinux]

IBM Site

IBM 6Gb PCI-E SAS Expander Daughter Card Firmware Update
ibm_fw_exp_6gb-sas-634a_windows_32-64.exe (1.35 MB)

thanks for the link, but how did you find it? i can't seem to navigate their website to find it...

 

[SIlviu]

Simple Google "IBM SAS Expander Firmware"
First link

 

[BLinux]

hmmm.... i downloaded the Linux firmware update utility, but it won't run. Says "this update is not for this system". Do i need to be in an IBM server to run these updates?

 

[SIlviu]

Search this thread, you unzip the .bin to get the .rd2 firmware and you send it with sg_write_buffer.
I still don't know if this will work without an IBM HBA/RAID card. I know it will bypass "this update is not for this system"

 

[TheBloke]

For anyone confused as to how to flash the FW:

I then used the sg_write_buffer method described in this ServeFault post to upload the 634A firmware which I extracted from the download package. (Which I've since realised can be done simply with unzip <filename.bin> - no need to hex-edit the ZIP part out, unzip will just scan through until it finds a valid ZIP.)

This worked perfectly, giving me FW 634A. And even better, the new FW 634A did, as hoped, fix the problem I had where SATA disks weren't properly detected on first expander power-on.

And here's a new guide for those still having trouble:

1. When you follow this guide, make sure there are no drives connected to the IBM Expander.
   1. I'd also recommend having no other expanders or drives connected, as this makes it easier to find the expander and reduces the risk of trying to flash the wrong device.
      
   2. When I did this, I put a single LSI HBA into my spare desktop machine, and connected the IBM Expander to it. I unplugged all SATA drives and then booted from USB stick.
2. You need a Linux install, or just a Linux Live CD (eg Ubuntu) on USB or DVD.
   1. When I did this back in 2017, I used the Live CD of Ubuntu 16.04 LTS, written to a USB flash drive.
   2. Link to download ubuntu-16.04.5-desktop-amd64.iso.
3. You need sg_write_buffer version 1.15 or later. If your Linux doesn't have that already (and Ubuntu 16.04 LTS definitely won't), download and install it with:
   curl -O http://sg.danny.cz/sg/p/libsgutils2-2_1.39-0.1_i386.deb &&
   curl -O http://sg.danny.cz/sg/p/sg3-utils_1.39-0.1_i386.deb &&
   dpkg -i *_1.39-0.1_i386.deb
4. Download the Linux FW package from IBM, or from the end of this post.
   1. You'll get a file called: ibm_fw_exp_6gb-sas-634a_linux_32-64.bin.
      
   2. Unzip this like it were a ZIP file:
      tomj@magrathea:~/IBM $ unzip ibm_fw_exp_6gb-sas-634a_linux_32-64.bin
      Archive: ibm_fw_exp_6gb-sas-634a_linux_32-64.bin
      warning [ibm_fw_exp_6gb-sas-634a_linux_32-64.bin]: 74943 extra bytes at beginning or within zipfile
      (attempting to process anyway)
      inflating: Linux_verAny_UP0_xAny.wrapcfg
      inflating: image/IBM_RMLib.so
      inflating: image/IBM_RMLib.so.64
      inflating: image/IBM_StoreLib.so
      inflating: image/IBM_StoreLib.so.64
      inflating: image/IBM_StoreLibIR.so
      inflating: image/IBM_StoreLibIR.so.64
      inflating: image/IBM_StoreLibIT.so
      inflating: image/IBM_StoreLibIT.so.64
      inflating: image/IBM_StoreLibIT2.so
      inflating: image/IBM_StoreLibIT2.so.64
      inflating: image/fdrvwl
      inflating: image/fdrvwl.64
      inflating: image/files/dl-634a.rd2
      inflating: image/flashdrv.bin
      inflating: image/install.sh
      inflating: image/libsysfs.so
      inflating: image/libsysfs.so.64
      inflating: image/readme.txt
      inflating: image/storelibconf.ini
      inflating: image/storelibconf.ini.64
      inflating: linwrap
5. The only file you need is the FW file, which is: image/files/dl-634a.rd2
6. Following the ServerFault post again: find the device name (eg /dev/sg1) for your expander, using: lsscsi
7. Now write the FW with sg_write_buffer:
   sudo sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=dl-634a.rd2 /dev/XXX
   sudo sg_write_buffer --mode=activate_mc /dev/XXX
   # replacing /dev/XXX with the correct /dev name for your Expander, as shown with lsscsi
   
8. Reboot back into Linux again, and running lsscsi again should show the updated FW version.

Now, here's a ZIP I prepared containing the FW downloaded from IBM, so you don't need to navigate IBM's awful site:

IBM-6Gb-SAS-Expander-FW-634a-Linux-and-Win.zip

As the name suggests, this contains both Linux and Windows packages. If you're going to flash using sg_write_buffer, use the Linux package. The Windows package is only for those who want to flash on Windows, meaning you must have a compatible IBM HBA in order to run the flash software. I have done this myself and found it did nothing different compared to using sg_write_buffer. But it does provide a less technical way to flash the FW, if you happen to have a supported IBM system / IBM HBA.

I'm pretty sure I already uploaded that earlier in this thread, because I already made the ZIP and uploaded it to my Google Drive. But there it is again

Good luck!

 

[nthu9280]

Nice write-up! I think this need be in guides / resources section.

 

[TheBloke]

PS. In the above guide I say you need a Linux system. I saw an earlier post in this thread, by @SIlviu , indicating that they had sg_write_buffer on Windows and were running the flash on Windows.

So I guess it may also be possible to do this on Windows, because the same sg_* tools are available.

However, user @SIlviu didn't actually get it working on Windows - he/she got errors. So I'd still highly recommend trying it on Linux, which can be easily achieved by rebooting into a USB live 'CD' of Ubuntu, or any other Linux distro.

 

[BLinux]

@TheBloke I saw earlier in this thread that you said the latest firmware show performance degradation? Also, it prevents using the input ports as output to backplane? If so, that seems more negatives than positives from upgrading the firmware? Why upgrade FW then?

 

[TheBloke]

@TheBloke I saw earlier in this thread that you said the latest firmware show performance degradation? Also, it prevents using the input ports as output to backplane? If so, that seems more negatives than positives from upgrading the firmware? Why upgrade FW then?

Yes, sadly all FW after 510A appears to have a performance issue in any setup where two cables are used to connect the expander to HBA. Two cables provide around 1.1x the performance of one cable, versus the expected 1.8 - 2x performance gain.

So why upgrade? Because double-sadly, 510A also has a major bug : SATA drives are not detected unless the drives / back planes are turned on after the expander has already initialised. In practice this means one must separate the powering of expander and drives/back planes, and only turn the drives on once the server and HBA has been powered on for 60 seconds or so (I wait until after I hear the server's POST beep.)

So unfortunately this expander is a case of a rock and hard place. We need to choose between two different sets of bugs. Or at least, those of us who want to use both of the ports marked for HBA/input do. Anyone who both doesn't need dual HBA-cable performance, and only plans to connect a max of 16 drives, should definitely upgrade to the latest, 634a.

Also, it prevents using the input ports as output to backplane?

Yes this seems to be the case, though I didn't verify this fully myself.

What I confirmed myself is that this issue exists on FW 602A and 633A: I was unable to use either of the two ports marked "INPUT" (or "HBA" or "CONTROLLER" or whatever it is) for drives. So I was only able to connect four drive cables = 16 maximum disks, and not 20 maximum disks as I had thought would be possible (and is possible on other expanders.)

Another user then said they did not experience the issue on 510A, confirmed on two different HBAs. And I believe they also confirmed that 634A, the latest, also experiences the problem, like I confirmed for 602A and 633A.

So it appears the overall situation is:

1. FW 510A:
   1. When connected with one HBA cable, allows connecting 20 drives;
   2. When connected with two HBA cables, provides full 2 x cable performance;
   3. But has the SATA power-on detection issue, requiring an annoying staggered power-on routine
2. FW 634A (and 633A, 602A, and seemingly every FW >510A):
   1. Is maximum 16 drives, as the two ports marked for HBA won't work with drives;
   2. Gets very little extra performance from using two HBA cables (only 1.1x faster than 1 cable);
   3. But resolves the SATA power-on issue.
   4. (And I guess all the other issues mentioned in the cumulative changelogs, though I don't know of any other reported issues affecting people.)

So this is a rather unfortunate combination of problems. A rock and a hard place, as I say.

Personally I'm still sticking with FW 510A, because I want the extra performance. It's not that often that I need to power-up my server, and I plugged all my server gear into remote-controlled power sockets to make it a bit easier. When I do have to power on the serve, I: a) Press one remote button to turn on the server, b) wait approx 2 minutes until I hear the POST beep, c) Press another remote button to turn on the external enclosure, spinning up the disks. Then I can leave it to boot. The problem doesn't happen on reboots, only when everything has powered down (eg a power cut.)

My IBM expander is physically located in my external drive enclosure, so I also had to do some funky wiring - the expander is sitting in an PCIe slot adapter, and that's connected to my server PSU via a very long Molex cable that runs between server chassis and external drive enclosure. This wouldn't be necessary if I'd had room inside my server chassis to power the expander, but it's only a 2U case and has no free PCIe slots. The key issue is that I can't power the expander from the same PSU as the drives as I originally planned, because that doesn't allow for staggered power-on.

 

[TheBloke]

Regarding workarounds to the staggered-power-on-issue:

It wouldn't be hard to come up with a nicer, automatic solution using a timer. Such that turning on the server would then turn on the disk enclosure automatically, X seconds later.

Someone earlier in the post mentioned a device called Add2PSU, which is a small board that allows linking multiple PSUs together. When one turns on, the second one turns on as well, and you can power additional equipment from the second. They also have a device with a timer, Add2PSU-D. But unfortunately the timer seems to only work for turning off the second PSU. You can set it so that PSU2 turns off X seconds/minutes after PSU1. That's the opposite of what we need - we need a way for PSU2 to turn on some time after you press the switch to turn on PSU1.

I'm sure someone could come up with a simple way to do that. In fact I could myself, using a Raspberry Pi - I have a little board for the Pi that sends RF signals to control Energenie RF power sockets (I use them for amateur theatre productions, to provide on/off lights that can be triggered during a show.) So I could write a little Python program that turns on socket 1, waits 2 minutes, then turns on socket 2. Then I could trigger that Python script over SSH from my workstation. I've just not bothered yet Though I might think of doing so if I keep having power outages as I have had a few times in the last couple of weeks.

I'd also consider changing out the expander. But last time I looked, other expanders that could do 16+ SATA3 drives with 2 x 6Gb connections to the HBA were far more expensive than the IBM. Like £150+ versus the £20 the IBM now costs on eBay. They're so cheap that when I was selling unused/spare HW a few months ago, I didn't even bother putting my second 46M0997 on eBay because getting £15 for it just didn't seem worth the hassle.

 

[BLinux]

@TheBloke thanks for taking the time to provide a lengthy response very much appreciated.

However, my experience has been slightly different and I'm wondering if you might be able to shed some light on them:

1) I have firmware 510A on 2 of these expanders. I've been testing them in a Supermicro 836 on my workbench, with a Dell H200 flashed to LSI IT mode P20. The 16-bay backplane is a direct attached via 4x SFF-8087->4xSATA. The H200 has 2x SFF-8087 cables to the 2x "input" ports of the expander. One thing I've noticed now a couple of times, and it happens intermittently, is that only 3 of the 4 HDDs on BP3 port show up. The first unit I got I thought was defective because of this and returned it. I've since bought 2 more of them from other sellers, and at first the problem didn't manifest, so it reinforced my thought that the 1st one I got was defective, but now, I'm getting the same problem with the other 2, but not always. I don't know what it is... I've tried both SAS and SATA drives, and same issue. i've swapped drives, and it is always the same port that seems to have problems (HDD in that port works fine if swapped to another slot, while working HDD put into that slot, doesn't work). could this be related to the issue you mentioned? doesn't exactly sound like it to me... and why 1 out of 4 SAS lanes on BP3 port?

2) on that note, other than the 1 out 4 SAS lanes not working on BP3, when it does work, I don't have a problem detecting HDDs even when the backplane powers on at the same time as the rest of the system. I don't have the delay the backplane power for it to work.

3) one more question about your experience: I think I read that you had the performance problem using Solaris? Have you since been able to confirm the same problem using other OSes to exclude the problem being the LSI driver in Solaris?

Any thoughts?

 

[TheBloke]

Hmm, interesting.

Assuming you've confirmed that all your backplanes always work when connected directly to your HBA, I'm wondering if your issue 1 is not only related to mine, but is in fact the exact same issue as my "staggered power-on" - just with far less severe and intermittent symptoms.

Way back on page 1 I first described the issue I was having. Specifically, when I powered up the expander at the same time as the drives/backplanes, I would only see two or three (forget which) drives out of the 15 I had connected at the time. But it was always the same drives. Or rather, it was always the same lanes. So just like you, if I swapped drives around, which drives were detected would change, according to which expander port they were connected to.

Therefore I wonder if we're seeing the same thing, just for me it affected 13 out of 16 lanes, and for you it only sometimes affects 1 out of 16 lanes.

If I'm right, then you will *never* see the issue if you do the same staggered power on (or staggered cabling) that I do. The way I found out about staggered power-on was testing in the LSI BIOS: I was doing the sort of cable swapping I describe above, thinking that I must have bad ports on the expander or something. I was in the process of unplugging cables from the expander, and then happened to start up the server before I plugged them back in. I then plugged them back in with the system on, sitting in the LSI BIOS - therefore after the drives had powered-up - and lo' and behold, all drives were detected.

So I wonder if you can do the same tests. First, power everything on at the same time with all cables connected, and drop into the LSI BIOS rather than boot the OS. Check your available drives. If you can consistently do this and get the full 16 drives, then that's already different to me. But try this a few times (powering down between each test.) See if you can get 15 drives a few times. We know your loss of 1 lane is intermittent, and perhaps it's often enough to repeatedly confirm. Below, I theorise it could be timing related.

If you can prove that you can sometimes lose 1 lane in the above test, then follow by duplicating what I did: boot everything (server, BPs and disks) with no drive/BP cables connected to the expander. So the LSI HBA is cabled to the expander, but the expander is not cabled to the BPs. Drop into the LSI BIOS, then connect up the backplane cables while watching the BIOS. See if this method always gives you 16 drives.

If your issue is intermittent you might have to repeat these tests a number of times. But if I'm right, you should notice that the missing lane only happens in the first test, and never happens when the drives are allowed to power up without being connected to the expander. In which case, it should be solved by a later FW.

Maybe your issue is different, but it certainly sounds like it could be the same or similar, despite the seeming difference in symptoms. The problem could well be related to the hardware attached - like maybe your BPs or disks initialise faster than mine. Or even that your BP/drives initialise slower than mine. Either way, a timing issue could explain why the issue always happens severely to me, but only happens intermittently, and much less severely, for you.

In my tests, the problem goes away both with a staggered-power-on, and with staggered cabling. This indicates that the issue only occurs when the expander can see the drives while the expander is first initialising. To me this implies that the expander isn't ready to connect to the drives, but then for some reason also stops polling for them; as if they 'half detect', and therefore it stops looking. Maybe it's erroneously checking the lanes before it's ready to connect to them, and because it's not ready, those lanes go into a hung/limbo state until either it restarts or the cables are re-plugged.

If it is anything like that, then the symptoms could definitely be dependent on the attached equipment. For example, if my drives/BPs initialise faster than yours - ie they're ready to start communicating with the expander sooner after power-on than yours are - that could put them into the window during which the expander is not ready. Whereas in your case, perhaps the drives and/or BPs taking longer to get ready would be similar in effect to a staggered connection: by the time they are ready, the expander is also ready. This might also explain why you sometimes get all drives, and sometimes miss one: the spin up time of drives can vary a little, so perhaps a small fluctuation in the start up time of the drives decides whether you get all, or miss one.

The fact that I do get 3 drives working also implies that the lanes are checked sequentially. If so, that means that the 3 lanes that work for me are the last three it checks, and by the time it gets to them, it's completed its initialisation and is ready to actually connect. The other 13 are checked too soon and always hang. I recall that the three drives that worked were always on a single BP port, which fits that theory.

As you can tell I'm completely guessing on all this. But a timing-based problem does sound plausible to me given the resolution I've found, and could describe both our sets of symptoms.

Regarding your third point - no, I've never verified it outside Solaris. Nor do I think I can, as to do so needs enough drives to perf test 2 x 6Gb connections versus 1 x 6Gb connections, and I just don't have enough spare drives. In fact I barely have any (working) spare drives left.

I suppose it should be possible to USB boot my server into Linux and run a raw read-only test - eg using dd to read raw data from every drive - that could verify the speed difference using the same drives I use in Solaris. But that would also mean re-flashing an expander to 634A, knowing that even if it did work, I couldn't benefit So I never bothered to test that. If it becomes a big issue for someone I could try it out. But it's easy enough to test yourself.

And anyway, I still find it hard to imagine there being an OS-driver-specific issue that could be impacted by the FW of an expander. And not even in a "only one port works" kind of way, but in a way where both ports connect, and bandwidth is slightly higher with two, just not high enough. Just seems like it must be purely related to the expander FW alone, or at most expander FW + HBA FW. But not OS driver as well.

 

[TrevInCarlton]

For anyone confused as to how to flash the FW:


And here's a new guide for those still having trouble:

1. When you follow this guide, make sure there are no drives connected to the IBM Expander.
   1. I'd also recommend having no other expanders or drives connected, as this makes it easier to find the expander and reduces the risk of trying to flash the wrong device.
      
   2. When I did this, I put a single LSI HBA into my spare desktop machine, and connected the IBM Expander to it. I unplugged all SATA drives and then booted from USB stick.
2. You need a Linux install, or just a Linux Live CD (eg Ubuntu) on USB or DVD.
   1. When I did this back in 2017, I used the Live CD of Ubuntu 16.04 LTS, written to a USB flash drive.
   2. Link to download ubuntu-16.04.5-desktop-amd64.iso.
3. You need sg_write_buffer version 1.15 or later. If your Linux doesn't have that already (and Ubuntu 16.04 LTS definitely won't), download and install it with:
   curl -O http://sg.danny.cz/sg/p/libsgutils2-2_1.39-0.1_i386.deb &&
   curl -O http://sg.danny.cz/sg/p/sg3-utils_1.39-0.1_i386.deb &&
   dpkg -i *_1.39-0.1_i386.deb
4. Download the Linux FW package from IBM, or from the end of this post.
   1. You'll get a file called: ibm_fw_exp_6gb-sas-634a_linux_32-64.bin.
      
   2. Unzip this like it were a ZIP file:
      tomj@magrathea:~/IBM $ unzip ibm_fw_exp_6gb-sas-634a_linux_32-64.bin
      Archive: ibm_fw_exp_6gb-sas-634a_linux_32-64.bin
      warning [ibm_fw_exp_6gb-sas-634a_linux_32-64.bin]: 74943 extra bytes at beginning or within zipfile
      (attempting to process anyway)
      inflating: Linux_verAny_UP0_xAny.wrapcfg
      inflating: image/IBM_RMLib.so
      inflating: image/IBM_RMLib.so.64
      inflating: image/IBM_StoreLib.so
      inflating: image/IBM_StoreLib.so.64
      inflating: image/IBM_StoreLibIR.so
      inflating: image/IBM_StoreLibIR.so.64
      inflating: image/IBM_StoreLibIT.so
      inflating: image/IBM_StoreLibIT.so.64
      inflating: image/IBM_StoreLibIT2.so
      inflating: image/IBM_StoreLibIT2.so.64
      inflating: image/fdrvwl
      inflating: image/fdrvwl.64
      inflating: image/files/dl-634a.rd2
      inflating: image/flashdrv.bin
      inflating: image/install.sh
      inflating: image/libsysfs.so
      inflating: image/libsysfs.so.64
      inflating: image/readme.txt
      inflating: image/storelibconf.ini
      inflating: image/storelibconf.ini.64
      inflating: linwrap
5. The only file you need is the FW file, which is: image/files/dl-634a.rd2
6. Following the ServerFault post again: find the device name (eg /dev/sg1) for your expander, using: lsscsi
7. Now write the FW with sg_write_buffer:
   sudo sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=dl-634a.rd2 /dev/XXX
   sudo sg_write_buffer --mode=activate_mc /dev/XXX
   # replacing /dev/XXX with the correct /dev name for your Expander, as shown with lsscsi
   
8. Reboot back into Linux again, and running lsscsi again should show the updated FW version.

Now, here's a ZIP I prepared containing the FW downloaded from IBM, so you don't need to navigate IBM's awful site:

IBM-6Gb-SAS-Expander-FW-634a-Linux-and-Win.zip

As the name suggests, this contains both Linux and Windows packages. If you're going to flash using sg_write_buffer, use the Linux package. The Windows package is only for those who want to flash on Windows, meaning you must have a compatible IBM HBA in order to run the flash software. I have done this myself and found it did nothing different compared to using sg_write_buffer. But it does provide a less technical way to flash the FW, if you happen to have a supported IBM system / IBM HBA.

I'm pretty sure I already uploaded that earlier in this thread, because I already made the ZIP and uploaded it to my Google Drive. But there it is again

Good luck!

Well, slowly getting there but for some reason the sg_write_buffer package won't install. Tried both from the supplied links, tried double clicking the files as well as right click and picking "Open with software install" which opens up an installer, click install but when I try to run the fw patch it just returns with "sudo sg_write_buffer : command not found" . I have no knowledge of Linux so there must be something I am doing wrong ? So close !!!

 

[TrevInCarlton]

@TheBloke thanks for taking the time to provide a lengthy response very much appreciated.

However, my experience has been slightly different and I'm wondering if you might be able to shed some light on them:

1) I have firmware 510A on 2 of these expanders. I've been testing them in a Supermicro 836 on my workbench, with a Dell H200 flashed to LSI IT mode P20. The 16-bay backplane is a direct attached via 4x SFF-8087->4xSATA. The H200 has 2x SFF-8087 cables to the 2x "input" ports of the expander. One thing I've noticed now a couple of times, and it happens intermittently, is that only 3 of the 4 HDDs on BP3 port show up. The first unit I got I thought was defective because of this and returned it. I've since bought 2 more of them from other sellers, and at first the problem didn't manifest, so it reinforced my thought that the 1st one I got was defective, but now, I'm getting the same problem with the other 2, but not always. I don't know what it is... I've tried both SAS and SATA drives, and same issue. i've swapped drives, and it is always the same port that seems to have problems (HDD in that port works fine if swapped to another slot, while working HDD put into that slot, doesn't work). could this be related to the issue you mentioned? doesn't exactly sound like it to me... and why 1 out of 4 SAS lanes on BP3 port?

2) on that note, other than the 1 out 4 SAS lanes not working on BP3, when it does work, I don't have a problem detecting HDDs even when the backplane powers on at the same time as the rest of the system. I don't have the delay the backplane power for it to work.

3) one more question about your experience: I think I read that you had the performance problem using Solaris? Have you since been able to confirm the same problem using other OSes to exclude the problem being the LSI driver in Solaris?

Any thoughts?

Just to throw in my own experience with the 510a firmware. I too would only see 3 drives if the whole system was powered at the same time but I had other problems with my SATA drives. On pulling out the drives and reinserting them to force the expander to detect them some of the attached 8 drives would start to make very unnatural noises and this was reflected on the Freenas boot up, drives would disappear and sometimes reappear at random. It was so bad that I was convinced that something had trashed one ore more drives. When I replaced the expander with one that I suspect does have later firmware (not been able to confirm version) Freenas did boot and my 28tb volume was intact and after a scrub a few checksum errors were repaired. So for me using Freenas the 510a is most definitely not an option

 

[TrevInCarlton]

Well, slowly getting there but for some reason the sg_write_buffer package won't install. Tried both from the supplied links, tried double clicking the files as well as right click and picking "Open with software install" which opens up an installer, click install but when I try to run the fw patch it just returns with "sudo sg_write_buffer : command not found" . I have no knowledge of Linux so there must be something I am doing wrong ? So close !!!

Panic over, all installed and expander successfully updated to 634a. Just need to test it now.

Many thanks TOM, you really are "THE BLOKE"

 

[TheBloke]

Great, glad to hear you've got the FW updated OK @TrevInCarlton !

And thanks for the details with 510A. Good to know I'm not the only one experiencing that issue - I was starting to wonder if I was leading everyone astray, describing an issue that actually no-one but me had

I can't say I experienced the second symptom, the weird drive noises. However I spent very little time playing about in the "only three drives detected" scenario. I quickly stumbled upon the fact that it could be resolved with staggered power-on, or staggered cabling, so I didn't do much further testing in situations where only a few drives were detected from boot.

I'm sure you'll find 634A will solve that problem. And if you only need to connect 16 drives total, and weren't looking for dual-cable performance between HBA and expander/disks, you can stay on 634A with no downsides.

 

[TheBloke]

I'd also consider changing out the expander. But last time I looked, other expanders that could do 16+ SATA3 drives with 2 x 6Gb connections to the HBA were far more expensive than the IBM.

I've now done 10 minutes of research on this.

There's an Intel 24-port SAS2/SATA3 expander, the RES2SV240, which I guess that may be the "Intel" mentioned in the title of this thread, which our IBM is the 'alternative' to. Although there's also a 36-port Intel which is much more capable, but also much more expensive (36 x 12Gb/s I think.)

I can get the 24-port RES2SV240 on eBay for between £65 (from China) and £80 (from UK.) That's 4 - 5 times the cost of the dirt-cheap IBM. But it's also not an unaffordable price in and of itself. And a quick check on eBay.com seems to indicate it can be got for as little as $55 in the US ($50 + shipping.)

There's a rather old STH forum thread on the expander here.

Not much info, and last post is 2013, but throughout the thread it does seem to be confirmed that:

1. It supports up to 20 drives with one HBA cable, or 16 drives with two;
2. Both SAS2 and SATA3 drives are supported up to 6Gb/s;
3. Two HBA links should provide 2 x 6Gb/s performance, and one user confirmed eight-lane connectivity with lsiutil. No benchmark results though.
4. It has a 4-pin Molex power connector, and therefore does not require a PCIe slot or PCIe adapter. It's also half-height, meaning if you do want to stick it in a real slot you can do so in a 2U case.

This would therefore seem to be a good replacement for the IBM, for those of us who either want full dual-link performance, or else want 20 drives (single-HBA-link), and are a bit tired of the staggered-power-on issue

I may splash out and get one and see what happens. From all I can tell pre-purchase, it seems like it should be a drop-in replacement for my existing IBM 46M0997, minus the problems.

If I do take the plunge I'll put any further findings in that thread I linked above. But I thought it'd be worth mentioning it here, given it does appear to be an affordable alternative for those of us who are stuck deciding between two unpleasant sets of bugs in the IBM.

 

[TrevInCarlton]

I've now done 10 minutes of research on this.

There's an Intel 24-port SAS2/SATA3 expander, the RES2SV240, which I guess that may be the "Intel" mentioned in the title of this thread, which our IBM is the 'alternative' to. Although there's also a 36-port Intel which is much more capable, but also much more expensive (36 x 12Gb/s I think.)

I can get the 24-port RES2SV240 on eBay for between £65 (from China) and £80 (from UK.) That's 4 - 5 times the cost of the dirt-cheap IBM. But it's also not an unaffordable price in and of itself. And a quick check on eBay.com seems to indicate it can be got for as little as $55 in the US ($50 + shipping.)

There's a rather old STH forum thread on the expander here.

Not much info, and last post is 2013, but throughout the thread it does seem to be confirmed that:

1. It supports up to 20 drives with one HBA cable, or 16 drives with two;
2. Both SAS2 and SATA3 drives are supported up to 6Gb/s;
3. Two HBA links should provide 2 x 6Gb/s performance, and one user confirmed eight-lane connectivity with lsiutil. No benchmark results though.
4. It has a 4-pin Molex power connector, and therefore does not require a PCIe slot or PCIe adapter. It's also half-height, meaning if you do want to stick it in a real slot you can do so in a 2U case.

This would therefore seem to be a good replacement for the IBM, for those of us who either want full dual-link performance, or else want 20 drives (single-HBA-link), and are a bit tired of the staggered-power-on issue

I may splash out and get one and see what happens. From all I can tell pre-purchase, it seems like it should be a drop-in replacement for my existing IBM 46M0997, minus the problems.

If I do take the plunge I'll put any further findings in that thread I linked above. But I thought it'd be worth mentioning it here, given it does appear to be an affordable alternative for those of us who are stuck deciding between two unpleasant sets of bugs in the IBM.

Yes, do let one and all if you do change expander. I have my own problems with performance on the IBM expander which I will update. As for the staggered power situation I have done some tests and the expander can be powered from a small 12v PSU. I tried running the expander from a Seagate Expansion drive PSU rated at 1.5a. On power up the expander peaked at 790ma so it would be possible to plug in the expander and just leave it powered up. If I read you right, this might be a simpler solution. This is what I have the expander plugged into one of these
USB 3.0 PCI-E Express 1x To 16x GPU Extender Riser Card Adapter Power BTC Cables 647746589637 | eBay

On the subject of the FW do you have any links to the 510A, 602A, 632A, and 633A. According to IBM the SATA drive issue was fixed with 633A although I have one expander with 602A which does not have the SATA problems. The following quote is from the IBM update history. 632A is interesting and might be the cause of the slow performance. On the face of it by the time we get to 634A all ? of the problems have been resolved.

" Version 633A - Moderate Impact/Low Probability of Occurrence
---------------
Fix(es):
- Modified OOB PHY settings in Expander. Addresses issue in which SATA drives would not
link up on power up

Version 632A - Suggested
------------
Expander | Version |
-----------------------------
Rev B | 602A |
Fix(es):
- Fixed a condition in which Phy 35 of the expander sometimes links up at a
data rate of 1.5G instead of 6.0G. "