SAS2 expanders $60 (IBM, LSI chip, Intel alternative)

kapone

Well-Known Member
May 23, 2015
772
367
63
So you're saying that on 510A firmware, like me you could only connect max 16 drives. But on 602A firmware, you could use more ports for drives and connect 20 drives? That's good to know. I'm pretty sure I never tested that on later FW, because it was only on the 510A firmware that I could get 2 x 6G connections, and there was no way I wanted more drives if I could only use a single lane to the HBA. So I think I only tested >16 drives with 510A FW.
The other way around.

With the 602A firmware, I could not use the top 8087 ports for drives. The drive would appear/disappear.
With 510A, the drive works perfectly in one of the top two ports.

Now, I haven't tried connecting 20 drives t it...yet. :) But it's promising that the top port works with 510A. Will try and update results here.

Are you able to see what total bandwidth you are able to get with 602 or later FW? Have you done any performance test with two cables from HBA to expander, and if so are you able to confirm if the total bandwidth is as expected for 2 x 6G links?
Working on it. This was preliminary testing only, to see what can/cannot be done. I ran tests only for a single disk, still need to run for multiple disks. With both firmwares, the single disk was averaging ~150MBps (Hitachi Ultrastar 6G 3TB drive).

When I tested, the links to HBA always negotiated at 6G, but then the resulting bandwidth was drastically crippled on all FW except 510A. So for example when using 510A with 15 x SATA3 drives, I would see a total write performance of 3+GB/s; each drive around 160MB/s. But when performing the same test with any later FW, this would drop to 1750MB/s - performance per drive of about 115MB/s.
That would not be cool, if that was the case... will need to test.
 

kapone

Well-Known Member
May 23, 2015
772
367
63
Update: Third expander board - came with 510A. Same result as the previous 510A - No drive detection issues, drive works in top port.

So, the first board with 510A was not a fluke... :)

Now, I'm gonna try and flash the one with the 602A firmware to 634A and test, then downgrade it to 510A as well and test.
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
40
Brighton, UK
Right OK thanks, sorry I misread. Now I'm confused about which combination I tested re using more than 4 ports - I'll have to re-try.

Now I'll be interested to know if you get the same SATA drive detection issues that I found in the 510A FW. You've only connected 1 drive so far, so you definitely wouldn't have noticed it yet.

Specifically, when I had 15 drives connected to the expander using 510A FW, I would find that only two or three drives showed up in my LSI BIOS and in the OS, unless the expander was powered up a few seconds before the drives/backplanes connected to it.

This 2-3 total drive number made no sense, as I had 4 x 8087 cables, each with 3 or 4 drives on them. I seem to remember that I worked out that two or three drives of one port were detected, then no drives on any other port. But swapping cables around didn't necessarily change which drives did detect, and it never detected all drives on a single port. So it seemed like the expander was simply quitting in the middle of drive enumeration, except when the expander had time to fully power on/initialise before it saw any connected disks.

You're already powering your expander from a server PCI-E slot. I don't know if your drives are currently powered from the server, or an external JBOD? Either way, I would expect that when you try connect many more drives - theoretically as few as four, but definitely if you use 12+ - you will find that unless the expander is turned on a few seconds before the drives, your HBA BIOS and OS will only detect two or three of them.

(Unless we find that it is something specific to my disks/backplanes - another thing I couldn't easily test. It could be that my disks spin up a little slower than others might, which creates the problem/makes it more likely.)

If you don't have this issue, then you might not need to use any FW later than 510A anyway - unless you do want the (theoretical) option of >16 drives per expander.

To re-iterate, I found myself stuck between two bugs:
  1. On 510A, I got full 2 x 6G bandwidth to the HBA.. but I had this power-on timing SATA detection issue, requiring the expander is always powered before connected drives.
  2. On later FW, the power-on timing SATA detection issue was gone.. but I could never get full 2 x 6G bandwidth to HBA.
I tried literally every FW and could not find any that solved both problems, so I ended up working around the SATA detection issue, powering the expander from my motherboard PSU instead of my JBOD PSU, and requiring that I always turned on the server before the JBOD, then waited a few seconds before turning on JBOD. Such that the expander would always be initialised before any disks/backplanes spun up.

It was much more important to me to get full BW, and although my workaround was a bit annoying, it was usable and wouldn't have presented much long-term pain, given that 99% of the time the server and JBOD would be running 24/7.

I would only experience the annoyance on the rare occasions I needed to power down the server, as I would then need to remember to also power down the JBOD, and then wait for 15-20 seconds before also turning the JBOD back on again.

It also made cabling a bit more awkward, given my expander is physically located inside my JBOD (on an external fake PCI-E slot for power), but must be powered from the motherboard, necessitating an extended molex cable running between the two. That wouldn't be an issue for you, as you're using a real PCI-E slot for power.

All this also means that if one did not have a JBOD - ie all drives were installed and powered in the server itself - things would be much worse, because there'd be no easy way to turn the drives on later than the expander. But I think you mentioned having a monster 84 drive array, so I assume you do have some kind of external chassis?
 
Last edited:

kapone

Well-Known Member
May 23, 2015
772
367
63
Like I said, more testing is needed.

As far as server/JBOD...this is gonna be a hybrid monstrosity, 14U high, 84 bays. It'll most likely have at least two power supplies powering separate things (one set of drives, and the other PSU powers the rest).

In terms of the chassis, while it'll all be "together" it'll be a miss mash/dremel of seven 2U chassis bolted together. Why? Hopefully to reduce noise by using larger fans, as well as be more power efficient by using one or two big/high quality power supplies, instead of 7 smaller PSUs.
 

james swallow

New Member
Feb 28, 2018
2
0
1
36
@TheBloke try using an add2psu to connect both power supplies to turn on at once. that will help with the power issue.
Also how did it work with the expander on a pcie riser? wouldn't 1x to 8x reduce speeds?
 

james swallow

New Member
Feb 28, 2018
2
0
1
36
So the raid card delivers all the info and the pcie is just to power the card? I feel stupid but I'm glad I asked.
 

squidman

Member
Jul 8, 2017
92
4
8
49
Can 2nd the add2psu works really well..started when I had to hook up so many sata drives to my corsair ax1200 (oldie but goodie!), and before I moved 8 sata drives over to the external DAS box, hooked up a cheap but ok old Corsair tx550? or something to the ax1200 for the extra sata connections...worked like a dream. But then my corsair 900D REALLY was running out of space, so went the DAS route. I think it says on the box of the 900D...what will you use all this space for? Haha good one...5ssd's and around 13 hdd's later, and dual cpu mobo, was getting pretty tight! Probably a data-hoarder problem, not your average joe's ;).

PS the DAS system works ok, and for me the mobo without cpu delivers the pcie power to power the expansion card just fine (not tested for 3 months, knock on wood! Misc drives for now!), but again not for complicated setups or where the OS needs to see 'barebones', i.e. MS storage spaces, or perhaps? linux versions, etc/variants of unraid etc. For that guess you'd want to it flash the m1015 or m1115, but as it ain't broken for me at the moment (nervous about it flash working with the ibm sas expander), ain't gonna mess with it. Prbly just ebay another m1115 for flashing to IT mode (need to at somepoint for backup server, which is in the works).

Of course on old-skool cable internet, 260 down but only 20 up! Iff I had fiber would probably just scratch building a backup server and just upload to 2x cloud backup services (yes all 32+ TB at the moment uploaded to jottacloud!). Just picked up an old antec 1200v1 for this purpose, and have a 3rd e5-2696 v3 lying around (wow ram prices suck tho!), so combined backup server and vm for extra rendering horsepower..just have to figure out better alternative to unraid (all those linux variants!)..

"just because you're paranoid doesn't mean all your hdd's aren't all out to get you!" or something..;)
 

squidman

Member
Jul 8, 2017
92
4
8
49
Like I said, more testing is needed.

As far as server/JBOD...this is gonna be a hybrid monstrosity, 14U high, 84 bays. It'll most likely have at least two power supplies powering separate things (one set of drives, and the other PSU powers the rest).

In terms of the chassis, while it'll all be "together" it'll be a miss mash/dremel of seven 2U chassis bolted together. Why? Hopefully to reduce noise by using larger fans, as well as be more power efficient by using one or two big/high quality power supplies, instead of 7 smaller PSUs.
Wow looking forward to seeing the monstrosity! How many petabytes?

BTW not a Corsair fanboy at all...their Corsair link is a nightmare! But ok PSU and cases...
 
Last edited:

squidman

Member
Jul 8, 2017
92
4
8
49
Roger that, going out in the bush sort of here in Norway, but still no real SAR/etc recognized here (going out solo in the mntns with my puny SPOT)..so yeah...no stupid questions! All stupid questions welcomed! (uhhnm loaded question for the Norwegian Police and SAR who won't recognize SPOT but insist on maritime EPIRB in the mountains!..oh well iff you want to be safe in the Norwegian Mountains..guess what! G0tta buy an iridium sat-telephone...and the cough cough around 1000 usd a year subsribtion! Sorry for that, rant almost out! WTF????!

Well I guess yacht charters in Florida should also have an EPIRB...but seriously hikers doing the appalachians? Should have a MARITIME? EPIRB? or it won't be responded too, or recognized? WTF? Well that's where we are at at the moment..completely frigging hopeless...

And yeah last 10 years working within Maritime Safety, so no not a big fan,.., and you've been been warned before coming here!
 

squidman

Member
Jul 8, 2017
92
4
8
49
Sorry totally off-topic, but that got so in my nose that...had to get it off my chest. Srry folks!

Yeah will redirect to Don't know who yet...have some contacts in norway, uk etc...but mostly in sof not sar...so no idea'

Srry again..background is a lot of hikers here in Norway having no idea, but the SAR response is to 'wait' until they mobile reception? Uhhm like even iff you set of a SPOT or IRIDIUM consumer emergency beacon, the police will just sit on their ass until they hear from you (go figure?) on a mobile phone, which kind of defeats the point of an emergency beacon, but oh well.

Uhhm a few beers later here, but the gist of it is..Norwegian Police WILL NOT RESPOND to an emergency set off by SPOT or IRIDIUM consumer beacons..instead they will WAIT!!!! until you have contacted them via mobile phone!!! Only then will emergency helicopters be authorized to lift off!
So go FRIGGING FIGURE!

They MAY respond iff you have a maritime EPIRB used for offshore use!
 
Last edited:

squidman

Member
Jul 8, 2017
92
4
8
49
Sorry about that..some beers could have been involved! But in the SAR industry and that kind of got up my nose!!!
 

pablocool

New Member
Mar 13, 2012
17
0
1
34
Poland
Hello Guys,

I also owned one of this IBM SAS Expander. I replaced my HP SAS Expander and few things are not clear to me.
I am curious why after replacing epxanders one of my drives went to Rebuild state. I thought expander is transparent and should not affect array.
Second thing is picture below. Why 7 drives are seen as they were connected directly to MegaRAID controller and 4 are visible as connected via SAS Expander?

ibm.JPG

Regarding performance due to usage of RAID6 I cannot see any read/write adventage of IBM (6gbps) over HP (3gbps).
But important is that with IBM every drive has own slot number. With HP all drives shared same slot number and only Device ID was different.
Then when one of my drive failed I was unable to distinguish from logs which because slot number told me nothing. I had to check smart with smartctl.

Last think is that with either HP or IBM my chassis Supermicro SC846 does not blink with hdd LEDs. They are off all the time.
With only MegaRAID adapter directly connected to drives LEDs works fine.
 

Stefan75

Member
Jan 22, 2018
80
8
8
44
Switzerland
cellarcinema.info
My current setup is 9201-16e dual linked to a HP expander in a 24 bay SC846 (BPN-SAS-846A).
I'm using Snapraid and the max. combined disk read performance I ever saw was 1790MB/s (15 SATA disks).

I'm now thinking about replacing the HP expander with two IBM 46M0997.
I'd single link each card (but 6G) and only use 12 ports each.
This should give me a performance boost, right?
 

sfbayzfs

Active Member
May 6, 2015
247
104
43
SF Bay area
If you haven't already upgraded the firmware in your HP expander, do so.

Almost 1800MB/s real is pretty good since ~2400MB/s is the theoretical max in your situation - it is generally reported that when using SATA3 6Gbps drives, most SAS2 expanders force the links down to 3Gbps all the way back to the controller unless you get super lucky with a drive/expander combo that links at 6G for SATA3. Dual linking to an HP expander would give you no more than 40Gbps (PCIe 2.0 x8 max for your card) using SAS2 drives, or 24Gbps with SATA drives. Single linking to two expanders will have the same problem, but if you dual link to both expanders with your 16e card, you should theoretically be able to double your bandwidth with a second SAS2 expander, although HP or IBM shouldn't matter.
 

Stefan75

Member
Jan 22, 2018
80
8
8
44
Switzerland
cellarcinema.info
So that's it. It's been a frustrating couple of days. But I still have an expander capable of turning two LSI ports into four backplane ports achieving 6G with SATA, for only £30. Overall it's still a great improvement over the HP in my setup.

It does have the power-on issue, but the symptoms of that can be almost completely eliminated with some alternate wiring that ensures the expander powers-on before the backplanes and disks. That is assuming others even experience the issue, which may not be a general problem. And won't be a problem at all if you only want to use one cable from controller to expander, in which case you can upgrade to the latest FW.

And indeed, I'd certainly recommend trying the latest FW even if you do want full 2 x 6G performance, as again maybe the issue I have is not general and others might not see it.

Either way, I'd definitely highly recommend the IBM to anyone who needs good performance out of a cheap expander. If you don't need high performance from expander to controller, or if you only plan to use SAS, then the HP is a little cheaper (£25 instead of £30) and has more ports (nine versus six).

But if like me you really want to have 2 x 6G links between expander and controller with SATA disks, the IBM seems in a class of its own at this used price.
Based on these comments I hope that a single linked 46M0997 can do 4x6Gbit/s max. bandwidth... double that of a HP expander (SATA). And if I connect 12 disks that would be 2Gbit/s per disk (~200MB/s).
I'm not a bit fan of dual linking because it doesn't seem to give you the 2x speed (maybe 1.5x). These expanders probably have internal cpu/bus limits too.
 
Last edited:

DiegoTorres

New Member
Sep 4, 2018
2
0
1
I have a very similar setup to @sfbayzfs and @TheBloke.

One Dell H200e in a gen10 mircoserver attached with two SFF8088 to a DAS. The DAS is a custom build, with a couple of SFF8088 to SF8087 converters, a IBM 16-port SAS/SATA 6Gbit Raid Expander Card 46M0997 and several IcyDock
FatCage MB155SP-B cages.

When dual-linking the H200e with the expander, I get UDMA errors when writing (for example, when updating the content file in a snapraid scrub operation). I can only use one link, more specifically, the link with the amber led (the SFF8088 conectors in the H200e has a led, which can be green or amber). It doesn't make any sense.

# lsscsi |grep IBM
[2:0:17:0] enclosu IBM-ESXS SAS EXP BP 510A -

Since I currently have version 510A, I would like to test latest version in the expander. I have downloaded this: ibm_fw_exp_6gb-sas-634a_windows_32-64.exe from IBM website, and decompressed it. I am planning to use the sg_write_buffer tool to upgrade,
but I'm not sure what to upload. Is this file the firmware: image/files/dl-634a.rd2 ?

The second question I have is that I would like to go back to 510A if somethig goes wrong (probabily, per the posts I have been reading). I'm not able to find an older firmware file anywhere. Could someone give me a link to the 510A?

Thanks in advance.
 

DiegoTorres

New Member
Sep 4, 2018
2
0
1
I also used that file to update (image/files/dl-634a.rd2).
I update the firmare without problems, but I am still missing an older firmware version, just in case I would like to revert to the original fw.

Everything looks fine... lets do some tests... :)