SAS2 expanders $60 (IBM, LSI chip, Intel alternative)

Discussion in 'Great Deals' started by sfbayzfs, Sep 20, 2016.

  1. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    615
    Likes Received:
    245
    The other way around.

    With the 602A firmware, I could not use the top 8087 ports for drives. The drive would appear/disappear.
    With 510A, the drive works perfectly in one of the top two ports.

    Now, I haven't tried connecting 20 drives t it...yet. :) But it's promising that the top port works with 510A. Will try and update results here.

    Working on it. This was preliminary testing only, to see what can/cannot be done. I ran tests only for a single disk, still need to run for multiple disks. With both firmwares, the single disk was averaging ~150MBps (Hitachi Ultrastar 6G 3TB drive).

    That would not be cool, if that was the case... will need to test.
     
    #41
  2. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    615
    Likes Received:
    245
    Update: Third expander board - came with 510A. Same result as the previous 510A - No drive detection issues, drive works in top port.

    So, the first board with 510A was not a fluke... :)

    Now, I'm gonna try and flash the one with the 602A firmware to 634A and test, then downgrade it to 510A as well and test.
     
    #42
  3. TheBloke

    TheBloke Active Member

    Joined:
    Feb 23, 2017
    Messages:
    200
    Likes Received:
    39
    Right OK thanks, sorry I misread. Now I'm confused about which combination I tested re using more than 4 ports - I'll have to re-try.

    Now I'll be interested to know if you get the same SATA drive detection issues that I found in the 510A FW. You've only connected 1 drive so far, so you definitely wouldn't have noticed it yet.

    Specifically, when I had 15 drives connected to the expander using 510A FW, I would find that only two or three drives showed up in my LSI BIOS and in the OS, unless the expander was powered up a few seconds before the drives/backplanes connected to it.

    This 2-3 total drive number made no sense, as I had 4 x 8087 cables, each with 3 or 4 drives on them. I seem to remember that I worked out that two or three drives of one port were detected, then no drives on any other port. But swapping cables around didn't necessarily change which drives did detect, and it never detected all drives on a single port. So it seemed like the expander was simply quitting in the middle of drive enumeration, except when the expander had time to fully power on/initialise before it saw any connected disks.

    You're already powering your expander from a server PCI-E slot. I don't know if your drives are currently powered from the server, or an external JBOD? Either way, I would expect that when you try connect many more drives - theoretically as few as four, but definitely if you use 12+ - you will find that unless the expander is turned on a few seconds before the drives, your HBA BIOS and OS will only detect two or three of them.

    (Unless we find that it is something specific to my disks/backplanes - another thing I couldn't easily test. It could be that my disks spin up a little slower than others might, which creates the problem/makes it more likely.)

    If you don't have this issue, then you might not need to use any FW later than 510A anyway - unless you do want the (theoretical) option of >16 drives per expander.

    To re-iterate, I found myself stuck between two bugs:
    1. On 510A, I got full 2 x 6G bandwidth to the HBA.. but I had this power-on timing SATA detection issue, requiring the expander is always powered before connected drives.
    2. On later FW, the power-on timing SATA detection issue was gone.. but I could never get full 2 x 6G bandwidth to HBA.
    I tried literally every FW and could not find any that solved both problems, so I ended up working around the SATA detection issue, powering the expander from my motherboard PSU instead of my JBOD PSU, and requiring that I always turned on the server before the JBOD, then waited a few seconds before turning on JBOD. Such that the expander would always be initialised before any disks/backplanes spun up.

    It was much more important to me to get full BW, and although my workaround was a bit annoying, it was usable and wouldn't have presented much long-term pain, given that 99% of the time the server and JBOD would be running 24/7.

    I would only experience the annoyance on the rare occasions I needed to power down the server, as I would then need to remember to also power down the JBOD, and then wait for 15-20 seconds before also turning the JBOD back on again.

    It also made cabling a bit more awkward, given my expander is physically located inside my JBOD (on an external fake PCI-E slot for power), but must be powered from the motherboard, necessitating an extended molex cable running between the two. That wouldn't be an issue for you, as you're using a real PCI-E slot for power.

    All this also means that if one did not have a JBOD - ie all drives were installed and powered in the server itself - things would be much worse, because there'd be no easy way to turn the drives on later than the expander. But I think you mentioned having a monster 84 drive array, so I assume you do have some kind of external chassis?
     
    #43
    Last edited: Dec 29, 2017
  4. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    615
    Likes Received:
    245
    Like I said, more testing is needed.

    As far as server/JBOD...this is gonna be a hybrid monstrosity, 14U high, 84 bays. It'll most likely have at least two power supplies powering separate things (one set of drives, and the other PSU powers the rest).

    In terms of the chassis, while it'll all be "together" it'll be a miss mash/dremel of seven 2U chassis bolted together. Why? Hopefully to reduce noise by using larger fans, as well as be more power efficient by using one or two big/high quality power supplies, instead of 7 smaller PSUs.
     
    #44
  5. james swallow

    james swallow New Member

    Joined:
    Feb 28, 2018
    Messages:
    2
    Likes Received:
    0
    @TheBloke try using an add2psu to connect both power supplies to turn on at once. that will help with the power issue.
    Also how did it work with the expander on a pcie riser? wouldn't 1x to 8x reduce speeds?
     
    #45
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,659
    Likes Received:
    400
    Pcie is used for power only
     
    #46
  7. james swallow

    james swallow New Member

    Joined:
    Feb 28, 2018
    Messages:
    2
    Likes Received:
    0
    So the raid card delivers all the info and the pcie is just to power the card? I feel stupid but I'm glad I asked.
     
    #47
  8. squidman

    squidman Member

    Joined:
    Jul 8, 2017
    Messages:
    92
    Likes Received:
    4
    Can 2nd the add2psu works really well..started when I had to hook up so many sata drives to my corsair ax1200 (oldie but goodie!), and before I moved 8 sata drives over to the external DAS box, hooked up a cheap but ok old Corsair tx550? or something to the ax1200 for the extra sata connections...worked like a dream. But then my corsair 900D REALLY was running out of space, so went the DAS route. I think it says on the box of the 900D...what will you use all this space for? Haha good one...5ssd's and around 13 hdd's later, and dual cpu mobo, was getting pretty tight! Probably a data-hoarder problem, not your average joe's ;).

    PS the DAS system works ok, and for me the mobo without cpu delivers the pcie power to power the expansion card just fine (not tested for 3 months, knock on wood! Misc drives for now!), but again not for complicated setups or where the OS needs to see 'barebones', i.e. MS storage spaces, or perhaps? linux versions, etc/variants of unraid etc. For that guess you'd want to it flash the m1015 or m1115, but as it ain't broken for me at the moment (nervous about it flash working with the ibm sas expander), ain't gonna mess with it. Prbly just ebay another m1115 for flashing to IT mode (need to at somepoint for backup server, which is in the works).

    Of course on old-skool cable internet, 260 down but only 20 up! Iff I had fiber would probably just scratch building a backup server and just upload to 2x cloud backup services (yes all 32+ TB at the moment uploaded to jottacloud!). Just picked up an old antec 1200v1 for this purpose, and have a 3rd e5-2696 v3 lying around (wow ram prices suck tho!), so combined backup server and vm for extra rendering horsepower..just have to figure out better alternative to unraid (all those linux variants!)..

    "just because you're paranoid doesn't mean all your hdd's aren't all out to get you!" or something..;)
     
    #48
  9. squidman

    squidman Member

    Joined:
    Jul 8, 2017
    Messages:
    92
    Likes Received:
    4
    Wow looking forward to seeing the monstrosity! How many petabytes?

    BTW not a Corsair fanboy at all...their Corsair link is a nightmare! But ok PSU and cases...
     
    #49
    Last edited: Mar 24, 2018
  10. pricklypunter

    pricklypunter Well-Known Member

    Joined:
    Nov 10, 2015
    Messages:
    1,514
    Likes Received:
    431
    There's no such thing as a stupid question. Those that ask what seems like a simple question might feel dumb, once, those that don't ask remain dumb forever ;):D
     
    #50
  11. squidman

    squidman Member

    Joined:
    Jul 8, 2017
    Messages:
    92
    Likes Received:
    4
    Roger that, going out in the bush sort of here in Norway, but still no real SAR/etc recognized here (going out solo in the mntns with my puny SPOT)..so yeah...no stupid questions! All stupid questions welcomed! (uhhnm loaded question for the Norwegian Police and SAR who won't recognize SPOT but insist on maritime EPIRB in the mountains!..oh well iff you want to be safe in the Norwegian Mountains..guess what! G0tta buy an iridium sat-telephone...and the cough cough around 1000 usd a year subsribtion! Sorry for that, rant almost out! WTF????!

    Well I guess yacht charters in Florida should also have an EPIRB...but seriously hikers doing the appalachians? Should have a MARITIME? EPIRB? or it won't be responded too, or recognized? WTF? Well that's where we are at at the moment..completely frigging hopeless...

    And yeah last 10 years working within Maritime Safety, so no not a big fan,.., and you've been been warned before coming here!
     
    #51
  12. squidman

    squidman Member

    Joined:
    Jul 8, 2017
    Messages:
    92
    Likes Received:
    4
    Sorry totally off-topic, but that got so in my nose that...had to get it off my chest. Srry folks!

    Yeah will redirect to Don't know who yet...have some contacts in norway, uk etc...but mostly in sof not sar...so no idea'

    Srry again..background is a lot of hikers here in Norway having no idea, but the SAR response is to 'wait' until they mobile reception? Uhhm like even iff you set of a SPOT or IRIDIUM consumer emergency beacon, the police will just sit on their ass until they hear from you (go figure?) on a mobile phone, which kind of defeats the point of an emergency beacon, but oh well.

    Uhhm a few beers later here, but the gist of it is..Norwegian Police WILL NOT RESPOND to an emergency set off by SPOT or IRIDIUM consumer beacons..instead they will WAIT!!!! until you have contacted them via mobile phone!!! Only then will emergency helicopters be authorized to lift off!
    So go FRIGGING FIGURE!

    They MAY respond iff you have a maritime EPIRB used for offshore use!
     
    #52
    Last edited: Mar 24, 2018
  13. squidman

    squidman Member

    Joined:
    Jul 8, 2017
    Messages:
    92
    Likes Received:
    4
    Sorry about that..some beers could have been involved! But in the SAR industry and that kind of got up my nose!!!
     
    #53
  14. pablocool

    pablocool New Member

    Joined:
    Mar 13, 2012
    Messages:
    14
    Likes Received:
    0
    Hello Guys,

    I also owned one of this IBM SAS Expander. I replaced my HP SAS Expander and few things are not clear to me.
    I am curious why after replacing epxanders one of my drives went to Rebuild state. I thought expander is transparent and should not affect array.
    Second thing is picture below. Why 7 drives are seen as they were connected directly to MegaRAID controller and 4 are visible as connected via SAS Expander?

    ibm.JPG

    Regarding performance due to usage of RAID6 I cannot see any read/write adventage of IBM (6gbps) over HP (3gbps).
    But important is that with IBM every drive has own slot number. With HP all drives shared same slot number and only Device ID was different.
    Then when one of my drive failed I was unable to distinguish from logs which because slot number told me nothing. I had to check smart with smartctl.

    Last think is that with either HP or IBM my chassis Supermicro SC846 does not blink with hdd LEDs. They are off all the time.
    With only MegaRAID adapter directly connected to drives LEDs works fine.
     
    #54
  15. Stefan75

    Stefan75 Member

    Joined:
    Jan 22, 2018
    Messages:
    71
    Likes Received:
    7
    My current setup is 9201-16e dual linked to a HP expander in a 24 bay SC846 (BPN-SAS-846A).
    I'm using Snapraid and the max. combined disk read performance I ever saw was 1790MB/s (15 SATA disks).

    I'm now thinking about replacing the HP expander with two IBM 46M0997.
    I'd single link each card (but 6G) and only use 12 ports each.
    This should give me a performance boost, right?
     
    #55
  16. sfbayzfs

    sfbayzfs Active Member

    Joined:
    May 6, 2015
    Messages:
    244
    Likes Received:
    102
    If you haven't already upgraded the firmware in your HP expander, do so.

    Almost 1800MB/s real is pretty good since ~2400MB/s is the theoretical max in your situation - it is generally reported that when using SATA3 6Gbps drives, most SAS2 expanders force the links down to 3Gbps all the way back to the controller unless you get super lucky with a drive/expander combo that links at 6G for SATA3. Dual linking to an HP expander would give you no more than 40Gbps (PCIe 2.0 x8 max for your card) using SAS2 drives, or 24Gbps with SATA drives. Single linking to two expanders will have the same problem, but if you dual link to both expanders with your 16e card, you should theoretically be able to double your bandwidth with a second SAS2 expander, although HP or IBM shouldn't matter.
     
    #56
  17. Stefan75

    Stefan75 Member

    Joined:
    Jan 22, 2018
    Messages:
    71
    Likes Received:
    7
    Based on these comments I hope that a single linked 46M0997 can do 4x6Gbit/s max. bandwidth... double that of a HP expander (SATA). And if I connect 12 disks that would be 2Gbit/s per disk (~200MB/s).
    I'm not a bit fan of dual linking because it doesn't seem to give you the 2x speed (maybe 1.5x). These expanders probably have internal cpu/bus limits too.
     
    #57
    Last edited: Aug 20, 2018
  18. DiegoTorres

    DiegoTorres New Member

    Joined:
    Sep 4, 2018
    Messages:
    2
    Likes Received:
    0
    I have a very similar setup to @sfbayzfs and @TheBloke.

    One Dell H200e in a gen10 mircoserver attached with two SFF8088 to a DAS. The DAS is a custom build, with a couple of SFF8088 to SF8087 converters, a IBM 16-port SAS/SATA 6Gbit Raid Expander Card 46M0997 and several IcyDock
    FatCage MB155SP-B cages.

    When dual-linking the H200e with the expander, I get UDMA errors when writing (for example, when updating the content file in a snapraid scrub operation). I can only use one link, more specifically, the link with the amber led (the SFF8088 conectors in the H200e has a led, which can be green or amber). It doesn't make any sense.

    # lsscsi |grep IBM
    [2:0:17:0] enclosu IBM-ESXS SAS EXP BP 510A -

    Since I currently have version 510A, I would like to test latest version in the expander. I have downloaded this: ibm_fw_exp_6gb-sas-634a_windows_32-64.exe from IBM website, and decompressed it. I am planning to use the sg_write_buffer tool to upgrade,
    but I'm not sure what to upload. Is this file the firmware: image/files/dl-634a.rd2 ?

    The second question I have is that I would like to go back to 510A if somethig goes wrong (probabily, per the posts I have been reading). I'm not able to find an older firmware file anywhere. Could someone give me a link to the 510A?

    Thanks in advance.
     
    #58
  19. Stefan75

    Stefan75 Member

    Joined:
    Jan 22, 2018
    Messages:
    71
    Likes Received:
    7
    I also used that file to update (image/files/dl-634a.rd2).
     
    #59
  20. DiegoTorres

    DiegoTorres New Member

    Joined:
    Sep 4, 2018
    Messages:
    2
    Likes Received:
    0
    I update the firmare without problems, but I am still missing an older firmware version, just in case I would like to revert to the original fw.

    Everything looks fine... lets do some tests... :)
     
    #60
Similar Threads: SAS2 expanders
Forum Title Date
Great Deals 846 SAS2 BPN $120 7 available Aug 27, 2019
Great Deals cse836+sas2+h8dgi $290 May 9, 2019
Great Deals CSE-826 w SAS2-826EL1 , no PSU/Trays - €60/BO Aug 18, 2018
Great Deals BPN-SAS2-846EL1 $175 incl shipping (7 avail) Jul 11, 2018
Great Deals BPN-SAS2-826EL1 - $36 May 12, 2018

Share This Page