If you have always wanted an Intel (LSI) SAS2 expander, and can deal with a full height SAS expander card and no molex power connector, the IBM ones use the same LSI chip, seem to have finally appeared at really good prices:
Yes, PCIe power, and it is intended for 8 channels in, 16 channels out (2 ports in, 4 out) but like all SAS2 expanders, you can use it for any combination of inputs/outputs you want to. Used as 1 port in, 5 ports out works nicely for 24 total drives on a single 8-port controller with 20 drives on one half and 4 on the other half, so you can use the 4 dedicated bays for SSDs if you want to, with the rest for spinners for instance.
that makes more sense. If I have to plug in 2 ports to get 4 ports via an expander then I may as well buy another M1015 to get a total of 4 ports (use up the same 2 PCI-e slots but provide more bandwidth and flexibility).
Yes, PCIe power, and it is intended for 8 channels in, 16 channels out (2 ports in, 4 out) but like all SAS2 expanders, you can use it for any combination of inputs/outputs you want to. Used as 1 port in, 5 ports out works nicely for 24 total drives on a single 8-port controller with 20 drives on one half and 4 on the other half, so you can use the 4 dedicated bays for SSDs if you want to, with the rest for spinners for instance.
your quote above may be true 99% of the time, but i can assure you that its not true for ALL expanders (i so wish it were true though, and totally should be)). see my thread from last week below, where i just recently bumped (more like rammed or crashed) into this exact issue, with an unfortunate conclusion:
TLDR: the Intel 36 por RES3TV360 SAS3 expander is called a 36 port expander, it has 9 x 4 lane SFF HD ports... however the ONLY mention of the fact that it will only support 24 HDDs (and i mean ONLY, i cant even find this on the net anywhere) is a single , very short sentence with no explanation or reasoning (see screen shot in my thread above). 2 weeks ago i tried everything to bring up 28 (or even 25 ) drives through the expander, NO LUCK at all... (im not very happy with intel right now). i had to end up tossing an extra 8405 card into the server to run 28 drives , outside the expander (so now i have 25 drive on my adaptec 8805 going to the expander, and 4 drives on a adaptec 8405 card) ... its a waste of a good 8405 card i constantly used for testing and such.
Im actually going to open a support case on this with Intel and see what their official reasoning is.
I posted a more detailed reply in your thread based on this info - have you tried updating the expander's firmware?
As for the deal, the first auction based in the US is sold out, only the others from China are still going, both for about $72 a unit, and the cheapest one I see in the US otherwise is $150.
I just ordered one of these expanders from China, and was wondering if anyone is still using them and has any feedback? I won't receive mine for at least 10 days, maybe longer.
A 6-port 6GB expander sounds awesome, so I'm quite hopeful. But I can't really find much info on this at all. Just a single page of installation documentation; no full specifications, firmware downloads/versions, anything like that. Far less info than is available for the famous HP 3G expander.
My main concern is how well it works with SATA. I already have that 9-port HP 3GB expander (the one everyone seems to use) and that does work OK, however it only does SATA at 3GB; 6GB only works with SAS. And for some reason I don't seem to be able to get the full 3GB speed expected from 2 x controller links.
This has meant that although it has those awesome 9 ports, I can only actually use it with 3 outbound ports, and doing so requires 2 ports to my controller. Any other config results in me not getting the full performance my drives and array can do (150-160MB/s sequential writes per drive, totalling about 3GB/s for the whole pool).
Therefore the HP has only enabled me to do 2 ports -> 3 ports, a net gain of one, which is rather less than I hoped. If I connect only one port from the LSI to the expander performance tanks to half, or if I run 4 ports outbound from the expander it drops by about 25%.
So if this IBM at the very least enables 2 ports -> 4 ports, that will already be a benefit for me. But I'm rather hoping it will enable me to have 1 controller port to 4 outbound, which would be a really useful upgrade if it doesn't lower my overall performance. 16 drives at 150MB/s each is only 2400MB/s so I should theoretically be able to get that from 4 x 6G lanes. But then my HP expander can't do 2400MB/s from 8 x 3G lanes (it seems to top out around 1750MB/s), so apparently this doesn't always work like it should!
Any further info anyone can provide on whether this works well with SATA, does proper 6G speeds, and whether 1 port to 4 port would work would be much appreciated. Likewise any link to firmware upgrades or better documentation.
The only issue I ever has with one of these is first one of these IBM SAS expanders I bought was DOA, but I didn't figure that out until I couldn't return it anymore, that was over 2 years ago.
Otherwise, I have been using one of these IBM LSI SAS2 expanders in my drive testing rig with no problems for a long time with no issues so far, but all I'm doing is a lot of dd, badblocks, smartctl through an LSI SAS2008 based 9210-8i HBA. It seems to work equivalently to the Intel SAS2 expander which uses the same LSI chip.
When I get around to it, I am planning to try to upgrade the firmware with the Intel SAS2 expander firmware update packages, but it's relatively low on my long todo list.
One caveat with SAS expanders - SAS moves to higher speeds than SATA does, so SATA drives often connect to SAS expanders at half the speed of SAS drives, especially if the firmware you are running on your SAS expander is old, and it varies by drive model and firmware as well. Also, it seems that link speeds are often forced down to the speed of the slowest device in part or all of the SAS "network" connected to a SAS expander, definitely with SAS1, and also with SAS2. SAS3 supposedly deals with this, allowing you to connect a SAS3 expander to slower SAS2 and SAS1 expanders at their native speeds, but send the data back to the SAS3 HBA at full SAS3 link speeds, however I can't confirm how well it works since I have been sticking with SAS2 due to it's maturity, stability, and wide support.
Also, for drives and link speeds and SAS expander compatibility, I have seen really old SATA 1 drives which even a SAS2 expander would not recognize, probably due to some spec variance.
There are also bandwidth conversion limitations - go through old STH articles for detailed bandwidth test results - PCIe 2.0 x8 cards may have a theoretical max bandwidth of 40Gbps, but the effective throughput to a SAS2 card loaded with fast SSDs is only roughly 75% of that - 75% is still very respectable since you don't expect to get theoretical ideal maximum throughput all the time.
Another thing to consider is that while your drives may have peak write speeds of 150-160MB/s at the beginning of the disk under ideal conditions, drive throughput for spinning disks always gradually falls to a little over half the initial speed by the time you get to the end of your disks - unless you massively under-provision your disks, your average serial write throughput is probably closer to 100MB/s for the disks you are using, which means that your real world 1750MB/s result should easily be enough for 16 drives under most conditions.
Back to firmware, I have noticed very significant increases in effective bandwidth from a firmware update on controllers before. Also, did you ever upgrade you HP SAS Expander's firmware? There are huge link speed and compatibility issues with older firmware revisions on that particular SAS expander.
To theorize further, I need to know more about your setup - please post more details of your test conditions, procedures, and results:
Motherboard and BIOS revision
CPU(s)
RAM type and amount
What OS are you running?
HP SAS expander firmware revision
SAS controller model(s) and firmware revision(s)
If you are using a hardware RAID controller, do you have a BBU installed, and is the BBU working properly?
Which PCIe slot(s) are you using for your SAS controller(s)? Remember, some slots are electrically only x4 or even slower, and many motherboards have different PCIe versions for different slots
Drive model(s) and firmware revision(s)
Drive configuration - please describe your hardware/software raid topology
Bandwidth test tools and test procedures you are using
What real software workloads are you running against these disks?
I ran 20 Hitachi 4TB coolspin drives through an Intel RES2SV240 SAS2 expander from one of two ports on an LSI SAS2008 card on P16 IT mode firmware in a PCIe 2.0 x8 slot on an X9SCM with a Xeon e3-1260 and 32GB of ECC DDR3 1333 running ZFS on Linux in Centos 6 for years with no issues. My initial bandwidth test for the array via dd of 100GB of /dev/urandom data cached on an SSD yielded 1.7 GB/s write to a stripe only at the beginning of the disks, which was plenty for me at the time. Years later, I found that upgrading the controlelr firmwar to P19 gave me a roughly 10% bandwidth increase to drives in a similar situation.
Thanks so much for the very detailed response, @sfbayzfs ! I'm really pleased to hear the IBM SAS expander is working well.
Those speeds you report from the Intel expander sound awesome - 1.7GB/s from a single port is exactly double the speed I get to the HP. So you must be getting 6GB to the controller even while using SATA. I get around 1.7GB/s from two ports.
If those speeds are indicative of what I should get from the IBM SAS2 then I'm really excited by that prospect! I might even have to buy a second one and stop using the HP at all (they're seriously cheap from China, only a few $ more than an HP.) I suppose I should wait to receive my first one to be sure, though!
You mentioned firmware for the IBM SAS2 - do you happen to have a link available? I'll definitely want to upgrade to latest firmware when I get mine, and to look at what's available in the meantime. I can find precious little such info on this expander so far, although I've relied only on Google searches; I haven't yet tried searching on IBM's site directly, and maybe that's what's required.
Below are the answers to your questions. Before that, I wanted to re-iterate my objective: I want to run 27 drives using no more than 2 x PCI-E 2.0 x8 slots + 1 x PCI-E 1.1 x4 slots.
I want this so that I can ensure leaving another 2 x PCI-E 2.0 x8 slots empty, for use by a dual-port 10GBe NIC and a PCI-E SSD. In total I have 4 x PCI-E 2.0 x8 slots and 1 x PCI-E 1.1 x4 (the latter has turned out to be the bane of my life - when I first read the manual I thought it was 2.0 x4 which would have been fine; now I find that not only is it 1.1 x4, but it also has conflicts with my onboard SATA2 ports. More on that later!)
And I want to do this without lowering the full performance I know my disks can do.
My performance comparison point is the speeds I get when all drives are connected direct to LSI cards (max 8 drives per LSI), which requires I use 4 x LSI cards and therefore 4 x PCI-E slots. When I do this, I get the 150-160MB/s writes per drive I mentioned, giving a total speed of 3GB/s. Read speeds are slower, as detailed below.
I should point out that once I get the IBM expander I've ordered, and assuming it works as expected, everything should then be fine even if the HP SAS Expander continues to only be usable as 2 ports in => 3 ports out as it is now. So my problem is short-term (10-20 days). Though of course I am also very keen to learn more about SATA/SAS and find out why things are as they are!
On to your questions. The following describes the config that I could use immediately which would free up the PCI-E slots I need without using any of my motherboard's onboard SATA2 ports (which I can't use at the same time as the PCI-E x4 slot as they share the same ICH10 and there's not enough bandwidth to use both at once.)
Motherboard and BIOS revision: Tyan S7012GM4NR, LGA1366, latest BIOS (2012 or so)
CPU(s) : 2 x Intel x5670 hex-core 2.93ghz (12 total cores, 24 total threads)
RAM type and amount : 120GB ECC DDR3 running at 800mhz
6 x 8GB + 6 x 8GB + 6 x 4GB, all 1333mhz Dual-Rank (2Rx4) PC3-10600R DIMMS
RAM is only running at 800mhz in this config for some reason; the manual suggested I should expect 1066mhz when using all 18 DIMM slots.
I got 1333mhz before I recently added an additional 6 x 8GB and filled all the slots. I've not yet investigated further.
What OS are you running? : Solaris 11.3
HP SAS expander firmware revision : The latest, 2.10c (also tested with 2.02)
SAS controller model(s) and firmware revision(s) : LSI 9207-8e (SAS2308), LSI 9201-8i and LSI 9211-4i (SAS2008), all running v20 IT firmware.
I also have available a 9205-8e (2308) and another 9201-8i (2008). I have tested both of those in place of the 9211-4i at various times.
If you are using a hardware RAID controller, do you have a BBU installed, and is the BBU working properly?
Which PCIe slot(s) are you using for your SAS controller(s)? 2 x PCI-E Rev 2.0 x8 slot + 1 x PCI-E Rev 1.1 x4 slot
Drive model(s) and firmware revision(s) : 27 drives; all SATA3 7.2k 64mb cache, 4K Advanced Format drives; all but four purchased used:
11 x Seagate ST2000DM001, revision 1ER164, firmware CC25 (only one available for this revision), dated 2014 - 2015
16 x Toshiba DT01ACA200, revisions ABB0 (newer) and AAB0 (older), dated 2012 - 2016
Can't immediately check or remember the Toshiba firmware, but it's not update-able anyway.
The oldest Toshiba drives (the AAB0 revision) show up on the controller as Hitachi HDS72302. It appears that all these drives are rebadged HGST drives, then in around 2013 with the ABB0 revision they start showing up as Toshiba. I haven't observed much difference between the older and newer revisions besides the software labelling.
Drive configuration - please describe your hardware/software raid topology:
Hardware:
7 x Chenbro 80H10321516A1 6G SAS/SATA 4-drive backplane with 1 x SFF-8087 port per-backplane:
3 x backplanes are in server chassis, connected to 12 drives: 11 x Seagate + 1 x Toshiba.
4 x backplanes are in external JBOD chassis, connected to 15 drives: all Toshiba.
4 x SFF-8087 from 4 x JBOD backplanes to HP SAS Expander: 15 drives, negotiated at 3G (the max the HP offers for SATA.)
2 x SFF-8087 from HP SAS Expander to SFF-8087-to-SFF-8088 adapters, then 2 x SFF-8088 into LSI 9207-8e card in PCI-E 2.0 x8 slot. (Later I will use the HP's external 8088 port for one of these links.)
The LSI reports these two links as negotiated at 6G, although the speeds I get are certainly not 6G.
2 x SFF-8087 from 2 x internal backplanes to LSI 9201-8i in PCI-E 2.0 x8 slot: 8 drives, negotiated at 6G
1 x SFF-8087 from 1 x internal backplane to LSI 9211-4i in PCI-E 1.1 x4 slot: 4 drives, negotiated at 6G
Software:
ZFS: 3 x 9-drive RAIDZ2 VDEVs; 6 total parity disks, 21 total data disks.
Pool capacity is around 32TB, of which 6TB is currently used; ~80% free space.
All VDEVs are ashift=12 (4K sectors).
Bandwidth test tools and test procedures you are using:
iozone run against an idle pool with all filesystems unmounted except those used for benchmarking. Compression is disabled on the benchmark filesystems. Command:
This test uses 2 concurrent processes to each write 128GB (one output file per process) using a recordsize of 128k.
It first does a write test, then a re-write test (which I ignore as its results are mostly cached with this data size), then a read test, then a re-read test (which again I ignore, because I don't understand its purpose and the result is 100% identical to read)
CPU impact:
On a full-performance (3GB/s) write test, iozone reports 170% CPU utilisation (relative to one core - ie. two cores each at 85% utilisation.) The OS as a whole is at about 50% of capacity, with the balance being used by the zpool-data process (not sure if this per-pool visible process is unique to Solaris or is also in other ZFS implementations)
I need to use two iozone processes else writes bottleneck on CPU. I get no additional performance from increasing concurrency beyond 2.
I consider the write speed as my benchmark value, as it's from this that I get the highest per-disk performance - up to 160MB/s. Read is consistently slower, maxing around 100-110MB/s per disk. I have theorised that it's slower in reads because of the concurrent processes, which will make it not quite a sequential test (unlike writes where ZFS ensures all writes are sequential to new blocks), but I haven't recently tested a single reading thread. In any case I am less concerned about reads because I expect real-life read performance to be significantly aided by RAM and SSD L2ARC caching.
What real software workloads are you running against these disks?:
None yet; for the purposes of these tests I'm just comparing the difference with iozone.
I will be doing more real world benchmarking once I get my 1.2TB PCI-E SSD (LSI/Seagate Nytro Warp Drive) installed, as this will provide a large amount of read cache (L2ARC).
Before I get going with that I wanted to ensure I still had the maximum 'best case' performance from my disks after freeing up the necessary PCI-E slots.
The config I described above does work, but the end performance is an array write speed of around 2.3GB/s instead of the 3GB/s.
Once I get my new IBM expander I hope I can end up running all 27 drives from only two cards - one LSI 9207-8e and an LSI 9205-8e (both 2308 chipset), in the following config:
12 internal drives via HP SAS expander, connecting to 2 x ports of LSI 9205-8e (in PCI-E 2.0 x8 slot)
15 external drives via IBM SAS2 expander, connecting to 2 x ports of LSI 9207-8e (in PCI-E 2.0 x8 slot)
So I'd run the HP SAS as 2 ports in => 3 ports out as I do now, but the IBM I hope I can do 2 ports in => 4 ports out. That would be a nice neat config using only two PCI-E slots and enabling me to sell several unused LSI cards.
Worst case, if the IBM turns out to be like the HP in that it can only do 2 ports -> 3 ports, that will still work:
12 internal drives via HP SAS expander, connecting to 2 x ports of LSI 9205-8e (in PCI-E 2.0 x8 slot)
12 external drives via IBM SAS2 expander, connecting to 2 x ports of LSI 9207-8e (in PCI-E 2.0 x8 slot)
3 external drives connected to LSI 9211-4i (in PCI-E 1.1 x4 slot)
Phew! Thanks again so much for your very detailed and very helpful response, and for offering to help further. I hope the above is all clear and not too horribly long.
Finally I will just say again that the problems I have now are hopefully only temporary for the next 10-20 days, so please don't feel they're vital. Of course, if you do have a moment to read and help further I'd definitely love if there was a way to get things running at full speed sooner, and even if there isn't, I definitely always love to learn as much as possible on any subject whenever possible.
My day of full-disk-bandwidth-from-two-x8-PCIe slots has finally arrived, thanks to our friends in China!
I got my IBM expander this morning, around 10 days after I ordered it on eBay; not bad, and probably would have been quicker if I'd not ordered it on a Saturday.
In short, it's done exactly what I hoped: I'm now running the 15 x SATA3 disks in my external JBOD from the IBM, with 2 cables to my 9207-8e. All drives are negotiated at 6GB, as are the two links from LSI to IBM.
So the IBM is functioning as 2 ports in => 4 ports out, which is what I hoped for.
I did test 1 port in => 4 ports out, but this reduced bandwidth. One port gave me about 1.7GB/s total bandwidth, a figure I note matches what @sfbayzfs reported from one link, and is the same as I got from 2 x 3GB links on the HP. I still don't quite understand this figure, as I thought the limit for a 6GB link was 2400MB/s, not ~1750MB/s.
Anyway, I am now able to get the magical 3GB/s write speeds to the pool, with all drives averaging around 160MB/s. I now have a three-LSI solution, with my 12 internal disks connected to a 9201-8i and a 9211-4i, the latter being in my PCIe 1.1 x4 slot (which works fine with a max of four drives.) This leaves me two free 2.0 x8 slots, which was the whole point of this exercise.
I've did briefly test having the 12 internal disks on the HP expander, allowing me to use only two LSI cards in total, but I found this reduced my max bandwidth by 200MB/s or so.
I've now ordered a second IBM from China and ultimately hope to run all 27 drives from just two LSI 2308 cards. I'll be selling my two HP expanders (I bought another at the same time as the IBM, in case the IBM didn't work.)
Annoying issue with SATA drives: usually not detected on power-on
Unfortunately I do have one irritating problem with the IBM expander. When I first cabled it up and checked the disks in the LSI BIOS, it showed only two out of 15 disks, which made no sense at all.
I pulled all the cables and re-inserted one by one; no change. Then I pulled and re-powered the IBM with no cables connected, then again re-inserted cables one by one, and then all disks came up properly.
This then happened again later after I had powered down the JBOD. So at the moment it seems like I might have the rather tedious procedure of needing to always turn the IBM on with no cables attached and then attach them one-by-one. This wouldn't be a terrible problem once the server is fully up and running, at which point it won't be turned off for weeks at a time, but is of course far from ideal.
Later firmware versions show a fix that may well solve this, but I am not yet able to apply any FW, as explained later.
46M0997 = "IBM 6Gb PCI-E SAS Expander Daughter Card". Mine is a "Revision B", 38 port expander
I've now learnt that the full proper name for this IBM 46M0997 is "IBM 6Gb PCI-E SAS Expander Daughter Card", and the card I received is a "Revision B", which is a 38-port expander. The original Rev A was 24 port.
I had always thought I was getting a 24-port expander, matching the number of physical ports it has: 6 SFF-8087 ports x 4 lanes each = 24 ports. But the one I have is definitely 38 port, both because it shows as 38 lanes in lsiutil, and because it has firmware 510A which is the first revision of the Rev B card (Rev A stopped at FW 440A).
I don't yet know if every 46M0997 out there is a Revision B - I don't know if the product code changes between Rev A and Rev B.
Nor do I quite understand the implication of this being a 38-port, given it only has 6 physical ports and therefore 24 lanes. I guess it means that if I connect other expanders to these ports, I can have up to 34 drives + 1 x 4-lane HBA connection?
Regardless of that the Rev B is definitely preferable as it has much later FW - assuming at least that this can be applied..
Firmware
Knowing that my card had firmware 510A finally enabled me to find the Google-fu required to locate the firmware updates for these cards.
The latest firmware is 634A, and it has at least one important-sounding fixes versus the 510A I have. Specifically:
Version 634A - Added support for Macronix MX25L3235EM2I-10G device
- Fixed an issue where running concurrent configuration commands would reset the expander
Version 633A - Modified OOB PHY settings in Expander. Addresses issue in which SATA drives would not link up on power up
Version 632A-1 - Fixed an issue where running the update package could initiate a system reboot.
- Fixed a condition in which Phy 35 of the expander sometimes links up at a data rate of 1.5G instead of 6.0G.
Version 602A - Fixed intermittent issue of Expander not responding to an receive diagnostic command.
I rather hope that "Modified OOB PHY settings in Expander. Addresses issue in which SATA drives would not link up on power up" fixes the annoying power-on issue I am currently having.
However, while I can download the latest FW OK (details below), I then unfortunately hit the not-unexpected issue of it refusing to install on both Linux and Windows, because "This update is not meant for this system." As seen in this screenshot from the Windows version.
So in the same way as the HP SAS expander FW won't normally apply except on an HP-branded HBA, this IBM FW is presumably locked to running only from an IBM server (such as x3650 or x3500.) Or perhaps it's locked to the IBM ServeRAID HBA specifically, as the HP expander is to the HP HBA.
Thoughts on how to install the FW (without an IBM server/HBA)
I did manage to get the HP FW applied from a normal LSI HBA, thanks to someone on ServerFault who published a method which uses sg_write_buffer from sg3_utils to simply dump the firmware to the expander. (Method described in this ServerFault post.)
Unfortunately that will be harder for the IBM, because where the HP firmware was distributed as multiple files including a separate firmware.bin, the IBM FW is distributed as a single executable on both Windows and Linux.
It must be possible to extract the actual expander FW from that executable, but that's immediately more complicated than for the HP. And even if/when the FW code can be isolated, there's the potentially much bigger problem that maybe the sg_write_buffer dump method won't work in the same way, with the risk of perhaps bricking the card.
I suppose it will be possible to validate if it should work. sg3_utils has sg_read_buffer, a read equivalent of sg_write_buffer, which presumably can be used to extract the current firmware from the card. That could then be compared against the new FW once it has been extracted from the updater executable.
Presuming it was possible to read the current FW, and that it looked similar enough to the updated FW, one could probably assume that it would work to overwrite current FW with new using sg_write_buffer. And in fact this method could also help extract the new FW from the executable, as having a copy of the current FW would show what to look for in the updater executable.
There is one alternative I can think of: 'crack' the firmware updater. Load it up in a debugger and bypass the system/HBA check, jumping straight to the FW update point. And hope that this then works normally with a standard LSI HBA. Which I suppose it should, as the IBM ServeRAID is just a re-branded LSI. If there's an HBA check it's probably just on Vendor and/or Model ID, and that could also be bypassed with a crack.
I can't profess any skill in such cracking but I am willing to have a go at least!
However I definitely won't be attempting any FW updating trickery until I get my second IBM SAS expander, for fear of bricking this one. As annoying as it is, I can definitely work around the SATA linkup issue until then. At least I don't have any performance problems or link-speed negotiation to worry about, which was my main concern.
How to get firmware for the IBM SAS expander
In case anyone does have access to an IBM system or just wants to play with the FW themselves, I have created a ZIP file containing the files for the latest 634a firmware for both Linux and Windows. It can be downloaded from this Google Drive link.
And here's the instructions for getting it from IBM the official way:
Click the SAS link to jump down to the SAS section
Look for the latest "IBM 6Gb PCI-E SAS Expander Daughter Card Firmware Update", which at the time of writing is dated 2016/09/13
There's two - one for Windows, one for Linux
When you click the link you want, you'll be prompted for your IBM ID login. Fortunately you can just sign up for this immediately (with email verification); it doesn't require a support contract or anything.
After creating an ID and/or signing in, you'll have access to the FW download pages.
Made what I hope is some good progress on the FW already.
The Windows updater executable is half executable, half ZIP file. The executable is a basic GUI part which, so far as I can tell, is purely for checking the system eligibility. It doesn't apply the FW itself.
I used a basic hex editor to copy out the ZIP file part and then extracted it, and found it contains a few files including another readme, an install.bat that runs a small command-line utility to apply the FW, and of course the raw FW file itself.
It's therefore possible that if I put an LSI card into a Windows machine, attach the expander to it, and run the install.bat I got out of this zip, it will apply the firmware. But not certain, as the extracted ZIP also contains a few dll files, named things like IBM_StoreLib.dll. These may mean that the patch utility is also expecting IBM-specific hardware/infrastructure.
I can almost certainly perform the same get-the-ZIP procedure on the Linux binary to get the Linux version.
So I'm not yet sure if I have a way to 'properly' apply the firmware. But I am now certain I have the actual FW file itself, which could hopefully be applied with sg_write_buffer as the HP one was.
I ran the same procedure on Linux to extract Linux's version of the FW ZIP
On a Ubuntu Live CD I tried running the resulting FW flashing utility, and sadly it is also locked to an IBM HBA, with an error of "No Adapter(s) found!" which the install script expands to "No applicable controllers were found. Flash not performed."
(I've confirmed that the expander is visible through my LSI controller using lsscsi)
I played around with sg_read_buffer to see if it is possible to download FW from the expander in the same way that sg_write_buffer can upload it.
But I couldn't find any --mode flag that would dump data. They all hung having only output a few bytes of data like the following: read buffer cdb: 3c 03 00 00 00 00 00 10 00 00
After which sg_read_buffer always reports the following (I am already using -v for verbose): read buffer: transport: Host_status=0x03 [DID_TIME_OUT]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
Read buffer failed: Sense category: -1, try '-v' option for more information
Maybe there's another mode that would give more data. There's other settings to sg_read_buffer like "buffer id field" and "buffer offset field". But of course I have no idea what values might go there, there's no obvious info on Google, and in any case I imagine it's quite likely that the expander simply doesn't support reading data in this way.
And perhaps it's not even meant to be possible to do this. The write utility, sg_write_buffer, has a whole bunch of extra modes relating to "downloading microcode" that sg_read_buffer doesn't mention. So maybe it's always intended to be write-only.
As a result I have no desire to rush into trying sg_write_buffer to send FW. Maybe I'll go out on a limb once I get my second one. But for now, having only just received the expander I was eagerly awaiting, I'm not going to risk bricking it!
I suppose a possible next step is to go back to debugging, and see if the command-line FW flash tool can be massaged into bypassing/ignoring whatever HBA check it makes.
Edit: thanks prickly! Sadly got a brick wall for now. Maybe when I get bored I'll try the debugger idea
Damn it I'm itching to just try this upload I'm an idiot.
Some more googling on sg_write_buffer tells me that:
It's fairly often mentioned as a generic way to upload FW on SAS/SATA devices.
I've found at least one IBM documentation page mentioning it for updating FW on a SAS enclosure (using basically the same command line as I used to update my HP)
Not quite yet I'm going to go configure my server and enjoy the bandwidth for at least a little bit before I risk breaking it all..
Nah, I'm gonna live with the SATA connection power-on issue for the 7-9 days it'll take for the second one to arrive. Or at least, I'm not going to try sg_write_buffer. I may still dabble with the IBM FW updater a bit more to see if it can be convinced to run on a standard LSI.
The SATA power-on issue only happens after the JBOD has been powered down, so I can reboot and power cycle the server as much as I like so long as the JBOD remains on. And even when I do need to mess about, it only takes two minutes a time to resolve.
I forgot the obligatory money shot:
It's a daughter card so it doesn't come with a bracket. However it does have screw holes for one, and I was surprised to find that the holes are in the right place for the HP expander's full-height bracket.
I briefly tried it affixed to that bracket so I could firmly screw it down into my JBOD (an old 16-drive, 3U server chassis.)
However unfortunately two of the 8087 ports are right at the top, so on a bracket in a 3U case I found it a little too high to comfortably connect those top two 8087s with the lid on. I've already had to carefully bend those cables as much as possible. Attached to the bracket the card was a few mm higher and I was worried about damaging the cables by bending them too much. Right-angle cables might be better, assuming two will fit side by side.
Instead I've gone for my standard solution: some combination of Blu Tack, cable ties and/or duck tape. There's a cable tie in each of the bracket screw holes, looping through the expansion holes in the chassis and holding it upright and pretty tight. I also put a little Blu Tack between the edge of the expander and the inside of the case, which the cable ties then pull the card tightly onto.
Under the expander is one of those "mining" PCIe power adapters, providing a Molex port for power.
I've got a small, medium-RPM fan pointed straight at it. I've not yet checked its temperature under load, but the HP expander I first used in that position got surprisingly, worryingly hot until I put that fan in.
Finally, there's two further small issues worthy of a quick note:
Unlike the HP expander or a normal LSI card, I don't get bay numbers with the IBM. In the LSI BIOS, and in the OS, all drives attached to the IBM report as being in Bay 255. I don't think this matters much, or at all, unless it's affecting:
Strangely, having this IBM expander in my system causes the LSI MegaRAID SNMP Agent to shutdown after boot. (I'm on Solaris 11.3)
No idea why, and I can't find any logs. I just see messages on console after boot indicating it went into its exit routine, and that it had an "error in removing Shared memory"
Maybe it doesn't like all the drives being on bay 255.
No big deal for me. Out of completeness I installed the SNMP agent when I first installed an LSI, but I've never used it since and don't expect I will. If I do monitoring on this server it'll e something like Nagios executing sas2ircu, lsiutil, and the various OS tools like iostat and zpool iostat.
Oh, screwing it down onto the base of the chassis? That's an interesting idea, I didn't think of that!
I'll have to see if there are standoffs anywhere near that area. Good thought, thanks!
EDIT: There aren't any right in that area, but the nearest aren't too far away. But the spacing is such that I could only screw down one hole of the PCIe adapter at a time, unless, as you say, I make some kind of plate.
My main concern is that it again raises the whole thing up by 2-3mm, creating the contention between closing the lid of the chassis and the top two 8087 cables. Either risking bending them too much and damaging them, or else not being able to close the lid.
I think I'll leave it as-is for now, but if I need to adjust it for any reason I will test it out sometime It's definitely a better plan than using an HP expander's back plate, as I want to sell those complete; not missing a back plate.
I'm already thinking I will keep an eye out for affordable used right-angle 8087 cables, as I have a couple of places where it would be more convenient and/or would reduce wear and tear on cables that otherwise need to be bent quite sharply.
Got my second IBM expander, and I have.. mixed news.
The good news is that I have successfully updated the FW on my second expander.
It came with the same old 510A version as the first; not surprising as I bought it from the same Chinese eBay store: taishan1980.
I then used the sg_write_buffer method described in this ServeFault post to upload the 634A firmware which I extracted from the download package. (Which I've since realised can be done simply with unzip <filename.bin> - no need to hex-edit the ZIP part out, unzip will just scan through until it finds a valid ZIP.)
This worked perfectly, giving me FW 634A. And even better, the new FW 634A did, as hoped, fix the problem I had where SATA disks weren't properly detected on first expander power-on.
As soon as I saw the new expander updated OK, I immediately reached over to grab my original one to upgrade that too.. but thankfully I stopped myself, saying "no, do it properly! Test the new one first."
Glad I did. For reasons I can't understand, the new FW has significantly reduced performance. Instead of the ~160MB/s write speeds my disks are capable of, FW 634A gives me only around 120MB/s, with a total speed across two controller->expander links of around 1750MB/s. Which is a figure that should be familiar to anyone reading this thread (assuming anyone actually is!)
This means I am getting only around 3GB/s speeds across the two links to the controller. Performance is therefore basically the same as I got on the HP expander, and not enough to get the full performance from my disks when I have four cables out of the IBM, accessing 15 drives.
So it would appear that somewhere between FW 510A and 634A, proper 6GB speeds for SATA disks has been broken. Or I suppose it could be that dual-link operation is broken. Either way, this is very disappointing, as it means I currently have to choose between full performance or the annoying power-on non-detected issue. Which isn't a choice at all, really: I'll go for full perf, and if need be put up with the power-on detection issue.
I have tried pulling and re-attaching cables to the 634A expander while powered on, in the hope that maybe it would be a similar issue to the power-on detection such that pulling and re-attaching cables would eventually connect properly and give full speed. But so far, no joy - I always hit the same performance bottleneck on the 634A expander. I've also swapped back in the original, version 510A expander to confirm that that does give full speed. So it's not an issue with cabling or anything else.
I have checked with lsiutil and confirmed my controller does see 2 x 6G links (ie. a total of 8 x 6G lanes), and it does. Exactly the same as with the original FW - and indeed with the HP, which definitely can only do 3G despite connecting at 6G. The difference on the IBM is that all drives show connected at 6G, and that remains the case on the perf-broken 634A FW.
So I suppose my next step is to progressively try each older FW version to see if I can find one that both has the full performance and doesn't have the power-on detection issue. Very annoying! Although now I look again I see there's only about three of them. Which also means I don't hold out much hope at all of finding one with both issues fixed.
In fact it looks like there's only one FW to try, 633A, which is the one before the latest. That's the one that introduced the fix for the SATA power-on detection. I'm going to go try that now..
Welp this just gets stranger and stranger - and more and more annoying.
I've spent the evening flashing every single available FW to the new expander. Meaning I've now tried it with:
634A
633A
632A
602A
Every single one performs identically. And so far, I haven't noticed the power-on connection issue with any of them, either.
Which seems to imply that the initial FW, 510A, is both the only one with that issue, and the only one that can sustain 2 x 6G links. Which just seems bizarre as it doesn't in any way match the changelog.
I've identified the exact nature of the performance problem:
With one cable from controller to expander, I get around 113MB/s per disk to 15 disks, totalling around 1700MB/s
Tested twice, using two different ports on the IBM expander, with identical results
With two cables from controller to expander, I get around 122MB/s per disk to 15 disks, totalling around 1830MB/s
So it is using two ports, but either:
the second port contributes only a ridiculously low ~150MB/s to the total bandwidth;
or having two ports connected cause both ports to drop to 3G
however, the total of 1830MB/s is slightly higher than I get from 1 x 6G port on the IBM, or from 2 x 3G on the HP. So it does really seem like one port is going full 6G speed, and the other is crawling along.
Then I get back to the weirdness of this happening with every FW version, and not even seeing the not-connected-on-poweron issue I see with 510A. Making me wonder if the symptoms I see are not related to FW.
Maybe there's something wrong with this second expander related to dual-port performance? And/or, maybe the power-on connection issue is related to the first card? Or maybe my non-official FW update method is not working properly? Given I basically can't see any difference, positive or negative, between any of four FW versions, from 602A through 634A.
Though it would seem weird for the update method to cause the version number to change but not actually change the FW. And if that's the case, it would either mean that the new card's 510A has both the low performance and no problem with connection - very different to the first card - or it would mean perhaps that only the first FW update (634A) applied, and all subsequent attempts to downgrade it have failed, despite changing the version number each time.
Unfortunately I didn't performance test the second expander when it arrived, so I can't verify if it was identical to my original card before I applied any FW to it. I just confirmed it had the same 510A FW version number, then went straight to updating it.
EDIT: I just found the original 510A firmware, so I'm now going to try flashing that to see if I can in fact return the second card to the same behaviour as the first.
I could try and test whether there's a hardware difference by also flashing the FW on my first card. But then that seems to carry a high risk of leaving me with no expander capable of doing 2 x 6G and getting full perf from my disks. So I really don't want to do that.
Nor do I want to buy yet a third expander! Or more precisely, my fifth, as I now have two HPs and two IBMs! (I do plan to sell the HPs soon..)
So I'm stuck and annoyed. Looking on the bright side, my original 510A expander can do full performance on my 15 external drives, and am hopeful that the new expander will provide just enough bandwidth to run my other 12 drives without significant performance drop, which was the primary reason I bought it. So I can still achieve 27 drives from two LSI cards.
But that will still leave me with the annoyance of the power-on issue on the original 510A expander. Not the end of the world, but it does require physical cable manipulation every time I power-on the expander which is a pain.
Ho hum. If anyone is reading this, I'd be grateful for any thoughts. Maybe this symptom of getting tiny amounts of bandwidth from the second port is familiar to someone. Or there's some kind of diagnostics I can run? I'd love to be able to see the bandwidth going through each of the ports - maybe there's some way to do that with lsiutil?
Besides that I guess I will do a bit more fiddling around - perhaps try the new expander on a different LSI card, just in case it's some weird conflict between specific cards. But ultimately I expect I'll just have to live with it.
Shame. I was so pleased earlier today when I saw that FW version successfully update to 634A after sg_write_buffer, and thought I'd be coming here to write a glowing recommendation of this expander as the #1 cheap expander: 2 x full 6G performance for 16 or more external drives, for only £30.
Well, I suppose it's still possible it is that and I just have a bum card and/or that the FW update method isn't working right. But unless someone else out there tries one, we probably won't know for sure
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
