SAS2 expanders $60 (IBM, LSI chip, Intel alternative)

[TheBloke]

OK, I finally got this to make some sense. Having found the original FW package for version 510A I flashed that to the new card, and it finally behaved identically to my first card: it had the issue with not connecting drives on first power on, and it had full 2 x 6G performance.

So either every FW version after 510A has fixed the former issue but broken the second or, far more likely, it's related to how I installed the FW - in reverse order, downgrading every time.

With regard to that, I can't help notice that the original FW package, 510A, is significantly larger than all of the subsequent ones.

So I'm wondering if all of these were intended to only be installed in ascending order, and perhaps the later files weren't even the full FW! Which makes it amazing it worked at all of course..

Whatever the case, I can't trust any of my results so far. Maybe 634A really did have a problem, but then my downgrades to all the other intermediate versions didn't change anything. Or maybe there's no problem with 634A, and the problems came from me going all the way from 510A to 634A in a single jump.

Maybe if I had been able to use the proper IBM installer, it would have checked versions and told me to install each FW update in order.

Anyway, now I need to go apply every FW update again, for the second time this evening, but this time in ascending order

I'll edit and clear up these two posts with a once I finally know what's going on. Thanks for listening

 

[pricklypunter]

Certainly makes sense given your experiences that the later ones could simply be code patches to the base firmware, in that they don't actually overwrite the firmware at all but rather just patch in code at certain addresses. Pretty bizarre though

 

[TheBloke]

Yup, certainly made sense. Wasn't correct, of course. All it achieved was wasting another hour or two of updating and testing

I applied the first updated FW, 602A, and immediately this caused both the power-on connection issue to resolve, and the performance to degrade. I therefore immediately feared the worst, but I went ahead and did further FW upgrades from 602A -> 632A -> 633A (I didn't bother perf testing 632A.)

Now on 633A, having done all the upgrades incrementally in the correct order, I still have the perf issue. Exactly the same as when I went straight to 634A.

So, two possibilities:

• The FW really is broken for dual-connection SATA performance as of 602A onwards, and the SATA connection issue I had was also fixed by that first update
  • And therefore apparently must be related to the following changelog entry in 602A: "Fixed intermittent issue of Expander not responding to an receive diagnostic command."
  • Further meaning that the issues described in later FW, that I was sure must be the fix for the power-on issue, were actually yet some other issue that presumably I don't have.
• Or, the unofficial sg_write_buffer method doesn't work properly except for applying the initial FW, 510A, which as noted is a much larger package than all the others (1.4MB versus ~860k for the others)
  • In which case, it means it's able to apply the FW in some way that changes the version number, and fixes at least one issue, but causes another..?
  • In an attempt to prove or deny this, I looked again at the files that come with the official updater.
  • There's a config file that tells the command-line patcher what FW file to apply. I looked in there to see if there was maybe some data that could be an offset or something, but no, there's nothing. Just the name of the FW and a couple of fields like the version number and the product code.
  • So if there is some special offset or way of applying the FW, it must be baked into the FW updater itself. But I note from Google that's also used for FW upgrading of various IBM SSDs, backplanes, etc. So this also seems pretty unlikely. And doesn't answer the issue of how the FW upgrade works well enough not only to change the version number, but to also fix a bug.

So I don't know. I have to assume it's the first, that the later FW simply is broken for 2 x 6G connections, at least in my application. Which seems pretty incredible given that the FW that introduces this issue for me, 602A, is dated 2011 or 2012, and the latest FW is 2016.

But then the issue may well depend on LSI controller version, or the backplanes I'm using, or even the drives I have. None of which would be found in an official, tested IBM system.

Oh well, I tried. A lot.

Actually there is one final, desperate measure: apply the FW to my first expander, in the vain hope that this perf issue is somehow specific to the second. Which it surely can't be given I know its performance is fine on the initial FW 510A. But I guess I may as well give it a go tomorrow, now that I know I can revert back to a working (from a performance point of view) FW version when it doesn't work.

Oh, and I suppose I should test with a different LSI controller. That's definitely the last avenue.

Tomorrow, though It's gone 3.30am.

 

[pricklypunter]

Now a cynic might draw the conclusion, given how small the other files are, that their purpose is to change the firmware revision number and nothing else. Of course no manufacturer has ever pulled that stunt before

As you said, really the only course of action is rule out the backplane/ controller hardware, but given your results, I'm not confident

 

[TheBloke]

Alrighty, probably my final update.

Workaround for FW 510A's SATA no-connect on power-on issue (and description of the issue)

The only good news: I've found a way to resolve this issue in my setup.

Firstly, I identified the exact problem: when the expander is powered on when already connected to powered-on backplanes and drives, drive detection is messed up.

I have four 8087 cables out of the IBM expander, each going to one Chenbro 4-drive SAS/SATA backplane. Connected to those four backplanes I have a total of 15 x SATA3 drives, all Toshiba DT01ACA200 drives (though the older ones show up as HGST, as Toshiba rebranded them.)

When I power the IBM on with those cables in place and with the backplanes and drives already powered, the controller sees only two drives. Which makes no sense, as one backplane has three drives connected, the others have four. I believe the two it shows are always on a single backplane. Clearly something gets screwy in the IBM's drive detection and it just quits on the job.

So to resolve this, the IBM needs to be powered on before it gets a chance to see the backplanes. Either the cables need to be connected after the IBM powers up (as I was doing manually before) or, better, the IBM needs to be powered on while the backplanes are turned off. As long as the IBM can power up before the backplanes/drives are visible, everything is OK.

My expander is physically sitting in my JBOD case in a PCIe power adapter and was originally powered via 4-pin molex from the JBOD PSU. I've now changed this so that I am instead running a long molex cable out of the back of my server case into the JBOD to power the expander directly from the server.

Now the expander comes on when the server does. I then power the JBOD a few seconds later, and everything works. It does mean that I need to remember that if I power down the server, I must also power down the JBOD so I can stagger its power-on.

Other people using the expander might well already have it powered from the server because they're using a real PCIE slot for power - the way it's designed to be installed. That would achieve the same result. Of course it may well be that this isn't anyway an issue for everyone, as it might be specific to my backplanes or drives.

Another, better option I am considering is to use a second, independent power supply for the PCIe adapter, inside the JBOD case. Just a little 20W sort of thing. That would remove the extra cable between server and JBOD and would also mean that a server power-down wouldn't require a JBOD power down, as I would be able to keep the expander powered independent of the server.

Final failed firmware tests

I've confirmed the following regarding the performance issue found on FW versions 602A and later:

• The problem persists when tested with a different LSI controller
  • I had originally tested with a 9207-8e / SAS2308 controller, I then additionally tried with a 9201-8i / SAS2008.
• The problem exists identically on both of my two expanders
  • I went ahead and upgraded my original expander to 602A, confirming that it then immediately hit the same performance issue
  • So it's surely solely related to the FW version.

Learning: The IBM expander has specific output ports - not any port for any purpose

Finally, I had wondered if there might be any difference using different ports on the expander. But this failed immediately and completely, as it appears not to be possible on the IBM expander to use any port for any purpose.

Specifically, while any port can be used to talk to the expander - I can use any of the six ports for a connection to the LSI controller - the same is not true for backplanes/disks. The expander only sees disks if they are connected to one of the four ports marked 'backplane'; when they are connected to one of the two 'input' ports, their drives do not show in the LSI BIOS or in the OS.

So as I want to do a 2 input => 4 output setup, this means I have to have the controller connected to the two ports labelled 'input' otherwise I can't use all four output ports.

This surprised me, as when researching the HP expander I had read several times that expanders are "like network switches" in that any port can talk to any other port; there aren't specific "controller/input" ports or "backplane/output" ports, even if the card is physically labelled as such.

That proved correct for the HP but does not appear to be for the IBM. I tested it on two FW versions (602A and 633A) across both of my IBM expanders without success.

Still a great expander!

So that's it. It's been a frustrating couple of days. But I still have an expander capable of turning two LSI ports into four backplane ports achieving 6G with SATA, for only £30. Overall it's still a great improvement over the HP in my setup.

It does have the power-on issue, but the symptoms of that can be almost completely eliminated with some alternate wiring that ensures the expander powers-on before the backplanes and disks. That is assuming others even experience the issue, which may not be a general problem. And won't be a problem at all if you only want to use one cable from controller to expander, in which case you can upgrade to the latest FW.

And indeed, I'd certainly recommend trying the latest FW even if you do want full 2 x 6G performance, as again maybe the issue I have is not general and others might not see it.

Either way, I'd definitely highly recommend the IBM to anyone who needs good performance out of a cheap expander. If you don't need high performance from expander to controller, or if you only plan to use SAS, then the HP is a little cheaper (£25 instead of £30) and has more ports (nine versus six).

But if like me you really want to have 2 x 6G links between expander and controller with SATA disks, the IBM seems in a class of its own at this used price.

 

[pricklypunter]

Oh well, at least you have a working solution, if not ideal

I still can't help but think there's a piece of the puzzle missing, but I think the only way of finding out for sure would be if the right IBM tech was to let it slip...

 

[TheBloke]

Oh well, at least you have a working solution, if not ideal

I still can't help but think there's a piece of the puzzle missing, but I think the only way of finding out for sure would be if the right IBM tech was to let it slip...

Yeah, I do find it hard to believe there's such a major performance issue that persists through literally years of multiple firmware updates. Maybe it's specific to my hardware, but then again I am using proper SAS/SATA backplanes with good quality drives, so it's not like I'm doing anything completely off the wall.

Not being able to apply the FW the official way, it's impossible to rule out the method of updating the expander as a cause of problems. I wish there was someone else here using these cards so I could get another datapoint.

But there is one option left to me, which I've just taken. I checked the FW readme for the list of compatible IBM ServeRAID HBAs and then cross-checked eBay, and surprisingly found a very cheap used IBM ServeRaid M5014 in the UK - only £39 delivered. So I've bought it.

I figure it's worth getting as even if it's no help at all, I'm certain to be able to sell it for at least the price I paid, and quite possibly more. They usually sell for £55+ from China, so I reckon I am very likely to be able to get at least £50 selling it in the UK if I wait long enough. Or I know I can sell an existing LSI HBA for that much, so I could keep the ServeRAID and sell another and still make a (very) small profit. In fact as the IBM is a SAS2108 it would seem to make sense to keep it in all scenarios to use in place of an existing SAS2008 (not sure if the 2108's faster processor has any benefit over 2008 with IT firmware, but it certainly can't hurt.)

Having a ServeRAID HBA should allow me to run the official command-line updater - I'm very confident that it only checks for HBA. It's the executable wrapper that checks for server and that's basically just a glorified ZIP file that is easily bypassed.

So I think having a supported IBM-branded HBA will open two possibilities:

• that using the real updater makes a difference to the resulting firmware, resolving the performance problem;
• even if not, there's the chance that the performance problem is actually related to controller, and will go away when the expander is used with a supported IBM HBA.
  • Who knows, perhaps later FW patched in a check for specific HBAs, crippling dual-port perf with a non-IBM-branded controller!

So I guess this saga isn't quite over yet

Thanks for responding and keeping this thread going Hopefully all this info will be useful to someone eventually, even if no-one else seems to be using these expanders at the moment

 

[pricklypunter]

I knew you had another ace up yer sleeve

I'm rooting for ya, but I can't say I'm confident of success, I still feel like the firmware is the issue, however, in for a penny an all that...

 

[TheBloke]

I knew you had another ace up yer sleeve

Sadly, more like a 2 of clubs

I'm rooting for ya, but I can't say I'm confident of success, I still feel like the firmware is the issue, however, in for a penny an all that...

Yeah seems so.

Got my ServeRAID 5014 this morning. It did indeed enable the official IBM FW patch tool, but unsurprisingly this has made no noticeable difference.

Using the IBM updater I tried going from 602A -> 634A on the 2nd expander, then 510A -> 634A on the 1st, then on the 2nd I re-installed the original 510A and from there upgraded first to 602A and finally to 634A.

By default the official IBM flasher won't downgrade FW. It checks the current FW and if it's the same or lower than the requested file, it tells you so and aborts. But I found it was possible to add 'FORCE' to the command line to apply any version. Not that this helped, of course.

Though I suppose it is good that this has verified that that a raw sg_write_buffer appears to be a viable firmware applicator for expanders in general.

The only option still remaining is to see if the dual-port performance problem might magically resolve when using the expander with its intended, supported ServeRAID controller. Of course I hold little hope, and my initial investigations are indicating that even trying this might be more trouble than it's worth , let alone actually using the config long-term.

At first I thought the ServeRAID wasn't even detected in Solaris, because I couldn't see any of the disks connected to it and it didn't show up in lsiutil or sas2ircu. Then I remembered that a MegaRAID is a rather different beast to the basic LSI HBAs. It's been so long since I used HW RAID that I've forgotten how these things work.

The card is detected in the OS and visible in MegaCLI, but if I want to see my drives it looks like I will need to configure it in JBOD mode and then configure each drive as a JBOD drive. What I'm so far unsure of is whether this is a non-destructive process which would allow my existing drives to become visible on the ServeRAID without losing any data. I know that making a volume out of each disk will definitely destroy the data, but I'm hoping JBOD mode will provide HBA-like operation and make the raw disks accessible without touching them.

I do have a full backup of my pool, and presently it only contains a few TB of data, but nonetheless I really can't be bothered to perform any operation that requires rebuilding the array. So this will only be viable if JBOD mode is non-destructive.

It seems almost certain that this idea, like all the others, will make no difference to the dual-port performance issue. But if it is possible to non-destructively get all my external disks visible to the OS through the ServeRAID, I will have one go at seeing if using this controller with the expander makes any difference versus a plain LSI.

That said, even if it did miraculously help, I'm not sure I'd actually want to run it long-term in that config. I'd be swapping out a lean IT-mode HBA for a full-fat HW RAID card. It'll be annoying requiring different management tools and the unnecessary RAID layer could even prove to be a performance or latency hit, despite the 2108's faster processor. I remember reading someone benchmarking LSI IT vs IR firmware and recording a 5% difference, which if true would imply MegaRAID could be even more taxing. And I have no idea if JBOD mode really is like a transparent HBA, or whether the MegaRAID interposes itself in any way, perhaps changing how the OS and ZFS sees the disks in terms of sector sizes and the like. Again I know that would happen with a volume, but I'm hoping the point of JBOD mode is to avoid that.

If there was anything like that it would make it not at all worth it to use MegaRAID vs a normal LSI. But I've come this far so I will at least try to see if it makes any positive difference so I can fill in the last pages of this tedious tale.

I also started to wonder, when pondering the question of "how could this major dual-port performance issue survive through so many FW updates", as to whether it could be specific to Solaris. That would certainly explain the issue persisting for so long, as Solaris is not an IBM-supported OS, and likewise Oracle do not provide support for the IBM servers that use this expander (though several of them are listed as 'Reported to work' in the Oracle 11 HCL.)

However I have no idea if it's even possible for OS to be an impacting factor to an expander. It's not like the OS has drivers for it, or even really accesses it directly. So perhaps there's no chance it's the OS. Regardless, it's academic: trying another OS is most definitely out of the question, as I'm not using an OpenZFS-compatible pool version. Likewise I have no intention of changing any other HW, such as backplanes or disks.

So while we're still not quite yet at the end of the line, we're definitely now gathering our personal belongings and making sure our tickets are ready for inspection at the barriers

 

[TheBloke]

OK looks like we're finally at the end of the line. I tried to set the ServeRAID into JBOD mode but kept getting an error.

Some Googling shows some discussion of that error in the context of Dell PERC cards (also re-badged LSI MegaRaid), indicating that they simply don't support JBOD. So the same must apply to the ServeRAID.

The only way to use these cards for ZFS is to create a RAID 0 volume for every individual drive, which would both destroy all the existing data, and be an extra unwanted layer between the OS and the actual drives. So I'm not going to even consider that.

I could flash the ServeRAID to be a generic LSI 9260-8i which would then support JBOD, but then that surely also removes the already tiny probability of the dual-port perf issue not existing with a supported IBM controller.

So, we're done

 

[Falloutboy]

OK looks like we're finally at the end of the line. I tried to set the ServeRAID into JBOD mode but kept getting an error.

Some Googling shows some discussion of that error in the context of Dell PERC cards (also re-badged LSI MegaRaid), indicating that they simply don't support JBOD. So the same must apply to the ServeRAID.

The only way to use these cards for ZFS is to create a RAID 0 volume for every individual drive, which would both destroy all the existing data, and be an extra unwanted layer between the OS and the actual drives. So I'm not going to even consider that.

I could flash the ServeRAID to be a generic LSI 9260-8i which would then support JBOD, but then that surely also removes the already tiny probability of the dual-port perf issue not existing with a supported IBM controller.

So, we're done

I have a sneaking suspicion I know what might be going on with this power on issue while researching the 46m0997 as I was given one I can across a document showing the M5015 at the back of a rack mounted case in a PCIe slot whereas the SAS expander was mounted at the front of the rack case near the bays, my suspicion is being that this is proprietary hardware that the machine powers up without power applied to the expander and then after initialization of some on board firmware that the sas expander is allowed to power up - being that the initialization is probably out of sequence on a desktop PC this could create a problem, it also mentioned that both the SAS card and the SAS expander HAD TO BE WHERE THEY WERE PLUGGED IN for things to work, changing the sas cards position would bodge things and putting the sas expander in a pcie slot in the rear would bodge things... Maybe there are some answers here... I have a few M1015s a 3108 a 9631-8i and the 46m0997, I am more than happy once I've got some personal stuff sorted on my end to run some tests on the M1015 with the 46M0997 attached if you would like and see if that still has the issue, I don't believe however that it is an issue with the different model of sas hardware I believe it is more likely to be something to do with the way that the propitiatory hardware initializes on those systems, perhaps instead of a separate power supply you could use a timed relay circuit so that when the system is powered on, after a certain amount of time or perhaps a signal generated from the sas controller by one of the led connections available as jumpers on the card it could kick off allowing the sas expander to power up, if my understanding is correct all the sas expander needs from the machine is juice and it does not use any of the other PCIe signal lines - though you might want to confirm this as it is just something I have read.

Good luck

for reference heres the link i was referring to
dW:Eserver:System x Server:SAS M5015 with Expansion Kit 58Y3825, x3650 M3 7945 - System x Server Forum

 

[pricklypunter]

That's an interesting twist to the tale, if it turns out to be true. And the game is on again...

 

[rampage666]

Is there any reason to favor this one instead of the cheaper famous 36 Port 6G 'HP SAS Expander'?
Compatibility reason?

 

[Jenz]

Hello,

got the same expander and take the Firmware from 510 Up to 634A and have no performance issues with my lsi 2008 SAS controller.
Operate fine with 16 bays! The price is realy cheap on ebay and the cheapest method to expand the home storage system without loud server equipment.

regards

Jens

 

[squidman]

Thanks for the posts all! Have the serveraid m1115 hba and the same sas expander in my jbod, was a bit nervous about getting it to work, but here are my experiences/2cents worth.

For me, (this is post fall creators update and many reinstalls..well 3 after updates killed every pc in the house!), actually got the sas expander to work in jbod DAS by powering up AFTER the server.

Setup: main workstation asrock rack ep2c612 ws with serveraid m1115/lsi 2008 as is with no firmware udates etc..no idea what bios or firmware, would have to take a pic on boot..not game to mess with that!

2 sff/8088 or whatever cables out to the DAS box (an old antec bought for 500 kroner/80bucks

DAS box: an old asus ddr2 mobo sans memory/cpu, attached to 8 various old hdd's, and an old corsair vx550w PSU. So far only 2x sff to 4xsata cables attached to the old hdd's.

Finally working (after I removed hdd's from a startech hba in the internal server for ssd caching), and figured I could try powering on the DAS AFTER the workstation/main pc!

Only problem at this point, is that the hba and sas expander are not seeing my last 8th hdd which is formatted in REFS (part of my storage spaces volume), and as I'm out of sata ports on the mobo, and would like to use the DAS box (old antec case) for future expansion..weell have a problem.

BUT! at least the system works and I can see all the old hdd's I didn't have space for in my corsair 900D! So was despairing of that ever happening, but maybe the stars were aligned correctly (live in Bergen, Norway, so don't know! always cloudy and raining here!), or some voodoo was going on, but amazingly was able to at least see 7 of 8 hdd's in my ghetto DAS. I probably shouldn't congratulate myself prematurely as it will probably go to hell in a handbasket, after 2 days reinstalling win 10 and only a third of my apps (thanks a bunch M$!!)..but there ya go!

The m1115 (worked fine for a year before this internally in my rig) and serveraid sas expander thing appears to work! In ghetto mod no less! Without replugging in all the hdd's or convoluted firmware upgrades! Hallelujah!

 

[squidman]

Addendum: using storage spaces with REFS after my mobo/software raid 5 died on my previous mobo (asus z9pe-d8ws) after upgrading to win 10 from win 7. Had luckily backed up pretty everything to an unlimited cloud account (jottacloud here in Norway), but still took a couple of weeks on (150/20 internet) to recreate lost data. Anyway after that, NEVER again raid5, never mobo raid, never hardware raid with/without bbsu, data on drives needs to be recoverable on system change and independent of other drives on array. Note this is mostly for home use, but 100k photos and 4k vids since the old hdv days...not to mention scans of slides...just can't lose it! Not to mention every doc since pretty much I discovered the internet back in around 1987?

Ok I have a problem, I admit it....will come out of the closet...am a data hoarder! There..I said it! Happy now?

 

[squidman]

Addendum ii: well not quite painless, the m115 suffers catastrophic failure and needs to be reset now everytime I power on the main workstation...according to the megaraid software gui..(well just gives an error message..no biggie!) but after that all is well! How about that? Have to admit reading all these trials and tribulations never expecting this ghetto server to work ast all, and yes have looked up the firmware update links and lenovo spec sheets (thanks shout out to TheBloke for all the research! and links)..but damn! Well 7 out of 8 ain't too bad! Just wondering why it won't see my last crucial drive in REFS (out of sata spaces on mobo, and need that one for the storage spaces!). Was not a problem pre-installing (before I installed) sas expander card, with just the m1115 to various drives INSIDE my rig...pre-fall creators update of course!

Actually getting windows to boot at all is a major victory! Took 2 days tho! Thanks M$oft!

 

[squidman]

Update serveraid m1115 and ibm sas expander pretty stable (apart from serveraid crashing on windows start and restarting), ALMOST all drives seen everytime, but had to give up on REFS formatted drive...sas expander and m1115 just wouldn't see it despite some switching of sata power cables and sata data cables (sas->4sata). So bit of a bummer there. Otherwise seems pretty solid for ntfs anyway (haven't tried exfat etc).
Key was to power on the DAS (with sas expander and drives) after Workstation with m1115 booted up.

 

[kapone]

I've been playing with a few these expanders (got back home early from trip, one of the kids got sick) for my "big ass 84 bay file server", and a couple of notes.

- My HBA is the IBM M1015 in IT mode.
- Motherboard/CPU/RAM should be immaterial for this discussion.
- The expander to be tested is plugged into an actual pic-e slot for power. The HBA is plugged into the "rearmost" of the top two 8087 connectors.
- Test HDD plugged into one of: either one of the Backplane ports, OR one of the HBA ports (top ones)

I have 10 of these and the first one I fired up came with the 602A firmware. On this one, I have no issues with drive detection (SATA), when the system is cold booted. Link speed is negotiated correctly, no performance issues. However...if the drive is plugged into one of the top 'HBA" ports, it'll show up in Windows Disk Management "momentarily" and then drop off...and then reappear....and then drop off. I concluded that "TheBloke" was correct that you can only add 16 drives to this.

Then I removed the first expander and plugged in a second one. This one had firmware 510A. Everything worked correctly, like the first card, except....the drive works correctly in the top HBA port as well! Weird! That means this actually CAN support 20 drives...

More testing to be done.

 

[TheBloke]

I've been playing with a few these expanders (got back home early from trip, one of the kids got sick) for my "big ass 84 bay file server", and a couple of notes.

- My HBA is the IBM M1015 in IT mode.
- Motherboard/CPU/RAM should be immaterial for this discussion.
- The expander to be tested is plugged into an actual pic-e slot for power. The HBA is plugged into the "rearmost" of the top two 8087 connectors.
- Test HDD plugged into one of: either one of the Backplane ports, OR one of the HBA ports (top ones)

I have 10 of these and the first one I fired up came with the 602A firmware. On this one, I have no issues with drive detection (SATA), when the system is cold booted. Link speed is negotiated correctly, no performance issues. However...if the drive is plugged into one of the top 'HBA" ports, it'll show up in Windows Disk Management "momentarily" and then drop off...and then reappear....and then drop off. I concluded that "TheBloke" was correct that you can only add 16 drives to this.

Then I removed the first expander and plugged in a second one. This one had firmware 510A. Everything worked correctly, like the first card, except....the drive works correctly in the top HBA port as well! Weird! That means this actually CAN support 20 drives...

More testing to be done.

Interesting, thanks for the new information.

I've not been playing with my server and HBAs for some months, due to being working away from home for a while. Mine is still in the state I described in my previous post - expander on 510A FW, 15 drives, with 2 x 6G connections to HBA. Which requires the correct power on cycle to ensure the expander is switched on before the HBA, else most of the SATA drives are not detected. This is an annoyance, but not a show stopper.

So you're saying that on 510A firmware, like me you could only connect max 16 drives. But on 602A firmware, you could use more ports for drives and connect 20 drives? That's good to know. I'm pretty sure I never tested that on later FW, because it was only on the 510A firmware that I could get 2 x 6G connections, and there was no way I wanted more drives if I could only use a single lane to the HBA. So I think I only tested >16 drives with 510A FW.

Are you able to see what total bandwidth you are able to get with 602 or later FW? Have you done any performance test with two cables from HBA to expander, and if so are you able to confirm if the total bandwidth is as expected for 2 x 6G links?

When I tested, the links to HBA always negotiated at 6G, but then the resulting bandwidth was drastically crippled on all FW except 510A. So for example when using 510A with 15 x SATA3 drives, I would see a total write performance of 3+GB/s; each drive around 160MB/s. But when performing the same test with any later FW, this would drop to 1750MB/s - performance per drive of about 115MB/s.

This 1750MB/s figure was pretty weird, because it was a little higher than 1 x 6G link to HBA; just a tiny bit higher. I think I got 1600MB/s with 1 x 6G link, then 1750MB/s with two. So it wasn't that it was simply not using the second link. It did use it, but seemingly at a snail's pace. Either that, or having two links caused both to drop to around 875MB/s for some reason. I don't think I ever worked out a way to monitor the links to see the data flow per-cable.

If you are getting full speed @kapone , that would be interesting. And odd. It could come down to OS differences, as I'm using Solaris and it's just possible there's some driver/OS issue in play. I never tested full array performance outside Solaris because I didn't have enough spare drives and I didn't want to have to wipe and restore the main array. Though I am now considering rebuilding the array to add more redundancy (3 x RAIDZ3 rather than 3 x RAIDZ2 - 3 x 3 disks of redundancy instead of 3 x 2), and if I do that I might try another OS test.