LSI 9202-16e available on eBay (but nowhere else)

[dba]

A few of us have been watching for the LSI 9202-16e card to make an appearance. The card has never been available for retail purchase. It has technically been "available" to OEMs for some time, but when I spoke to LSI a few months ago they told me that nobody was actively OEM-ing it, which means that you can't actually buy it from an OEM either.

Given that there are so many nice LSI cards that you can buy, why is the 9202 interesting? Because it promises to be the fastest LSI card ever in terms of MB/Second. The 9202-16e is basically two LSI 9200-8e cards glued together - 16 ports instead of 8, x16 PCIe 2 interface instead of x8, and TWO LSI 2008 controllers instead of just one. If you are short of PCIe slots and in need of more slots or more throughput, it's an appealing product.

LSI sort of hinted that the card would be made available for retail sales in June or July of this year, at an undisclosed price. It would probably be priced just a bit less than two 9200 cards, let's guess $800. Imagine my surprise, then when I discovered a handful of these never-before-available cards for sale on eBay as "refurbished" for $400. See: http://www.ebay.com/itm/LSI-SAS9202...S-SATA-SSD-HBA-HOST-BUS-ADAPTER-/110893777410

 

[Patrick]

Did you buy one? That is not a bad price at all. Also, very interesting that it is PCIe x16!

One note also is that these do not use SFF-8088 instead SFF-8644 cables.

 

[dba]

I immediately bought one - and might regret not picking up two. Now I need to find cables for a reasonable price.

 

[Patrick]

Yea... in the LGA 2011 platforms when you have tons of low latency PCIe bandwidth, this would let you put a ton of controllers a chassis.

Half thinking I may pick one up myself.

 

[mobilenvidia]

Its the way to go to stay away from expanders.

 

[Patrick]

Ok dying to know, how has the card been doing dba?

 

[dba]

The SFF-8644 cables arrived yesterday and the 9202 controller is scheduled to arrive today. By the end of the week I'll know something.

My plan is to move from my current setup of 30 SSD drives with five LSI RAID controllers in five cards to six controllers in five cards (the new LSI has dual controllers of course) and then benchmark using Oracle IO calibration. If I see the throughput increase that I hope for, I'll spring for another card to make it seven LSI controllers total - the maximum possible in my Supermicro server. If there are no other bottlenecks, I just might break the 10,000MB/Second Oracle read throughput barrier - that's 200,000,000 rows/second in my database application. Unfortunately, there does appear to be some non-linearity springing up and I might get stuck at around 8,000MB/Second.

The above will be a disappointing message to those wishing to see IOMeter or AS-SSD benchmarks for this exciting new card, but my drives are currently stuffed with data and I don't have enough spares to throw together the 12-16 SSD drive system needed for a full synthetic benchmark.

Ok dying to know, how has the card been doing dba?

 

[Patrick]

Let me know if you do get another. Maybe I'd try seeing if I could push 4GB/s off of this controller, take a few pictures and do a quick writeup (unless you wanted to) and then drive the card over to you. How much were the SFF-8644 cables BTW?

 

[dba]

If this new dual-controller x16 card scales like two LSI single-controller x8 cards then 5 to 5.1GB/s IOMeter reads should be achievable with JBOD (and possibly RAID0) using 14 to 16 SATA3 SSD drives.

If time permits, I would like to try my hand at doing a quick writeup for your site. If not, and I do end up getting a second card, I will certainly loan it to you for a bit. We could also collaborate - I have a few spare SSD drives and if you have some as well then perhaps we have enough. I've got the AMD server MOBO for the test and I'm sure that you have an Intel LGA-2011 that would make for a good comparison.

The cables are a bit rare and go for at least $85 each new... when you can find them in stock. I found two used cables on eBay for $19 each including shipping. Buying something used on eBay from a vendor in China is extremely risky - I remember one electronic item that showed up looking like it was used as windshield scraper for a few years. In this case the purchase turned out very well. Seeing that I bought two cables from him or her, they upgraded me to FedEX shipping at no charge and the cables arrived in three days looking quite good.

Let me know if you do get another. Maybe I'd try seeing if I could push 4GB/s off of this controller, take a few pictures and do a quick writeup (unless you wanted to) and then drive the card over to you. How much were the SFF-8644 cables BTW?

 

[mobilenvidia]

$19 cables you just got ?

 

[dba]

Not quite. I looked at those but ended up going with someone calling themselves "greatdeal0223" - a horrible name but excellent feedback.

$19 cables you just got ?

 

[mobilenvidia]

Yes most of my HTPC build bits are coming from Hong Kong.
Just keep an eye on the feedback, 10k positive means chances are the item will turn up.

I've not been stung yet with any ebay purchases over the last 12 years (touch wood)

 

[dba]

I installed the LSI 9202 card and ran a few quick tests. Interestingly, in the Windows device manager it shows up as two separate LSI 2008 chips instead of as one controller.

I ran a series of IOMeter tests with 8x OCZ Vertex3 120GB drives - 4 drives attached to each of two SAS ports and two SAS ports left empty. The performance is essentially identical to a 9200-8e, no matter which two SAS ports I used, with maximum reads at 2190MB/Second and maximum IOPS at 252,000. I won't be able to test with more drives or with drives spread across all four SAS ports until the rest of my cables arrive next week. I did manage to scrape up 16 Vertex3 120GB drives so when those cables do arrive I'll be able to run a full set of IOMeter, AS-SSD, and ATTO tests.

Testing each individual drive separately yields ~539MB/second so I know that the individual drives are doing OK.

The SFF-8644 cables arrived yesterday and the 9202 controller is scheduled to arrive today. By the end of the week I'll know something.

My plan is to move from my current setup of 30 SSD drives with five LSI RAID controllers in five cards to six controllers in five cards (the new LSI has dual controllers of course) and then benchmark using Oracle IO calibration. If I see the throughput increase that I hope for, I'll spring for another card to make it seven LSI controllers total - the maximum possible in my Supermicro server. If there are no other bottlenecks, I just might break the 10,000MB/Second Oracle read throughput barrier - that's 200,000,000 rows/second in my database application. Unfortunately, there does appear to be some non-linearity springing up and I might get stuck at around 8,000MB/Second.

The above will be a disappointing message to those wishing to see IOMeter or AS-SSD benchmarks for this exciting new card, but my drives are currently stuffed with data and I don't have enough spares to throw together the 12-16 SSD drive system needed for a full synthetic benchmark.

 

[Patrick]

I installed the LSI 9202 card and ran a few quick tests. Interestingly, in the Windows device manager it shows up as two separate LSI 2008 chips instead of as one controller.

Oddly enough, that's exactly what my first thought was, that the implementation was basically stick two SAS 2008 controllers attached to their own eight PCIe lanes on a single card. What got me off of that thought (other than some sleep) was the fact that the LSI site uses a three heatsink design, different than that model:

The big question is what is under the small heatsink since I would guess the first two cover the SAS 2008 controllers.

 

[mobilenvidia]

The 3rd heatsink could be for the NVRAM chip ?

 

[dba]

I would be just fine if the 9202 were two 9200s stuck together. I need the bandwidth that comes from more PCIe lanes and more RAID controller chips but since I don't use the hardware RAID feature, I wouldn't mind not being able to create volumes across all four SAS ports. Given the results from my first tests, however, I am really hoping that you are correct and there is a "third chip" doing something significant. Here is why:

I used eight SSD drives and two of the four x4 SAS ports in my test. Had this been a "9200 sandwich", there would be been some combination of ports that would have placed four disks on one LSI controller and four on the other with each controller having access to its own eight PCIe lanes. There would have been another combination where both ports were attached to just one of the LSI controllers - which would have been slower. No matter which two SAS ports I used, however, the performance was identical and, further, it was nearly identical to a single LSI 9200-8e card.

When I receive the balance of my cable order, I'll hook up 16 drives and then re-test. Hopefully this will somehow cause the card to utilize both controllers and all 16 lanes at the same time, providing more bandwidth than during my 8 disk tests. I have to admit that I'm not feeling confident. I cannot imagine an architecture that is ONLY able to utilize both controllers when ALL of the SAS ports are in use. A very careful read of the specification sheet reveals that LSI is not making performance claims for the x16 interface. Instead of declaring that x16 provides better actual throughput, they merely state that x16 PCIe provides "faster signaling for high bandwidth applications".

By the way, my card does not look like the one in the photo. The actual card utilizes a single (but HUGE) heat sink that covers most of the card.

Oddly enough, that's exactly what my first thought was, that the implementation was basically stick two SAS 2008 controllers attached to their own eight PCIe lanes on a single card. What got me off of that thought (other than some sleep) was the fact that the LSI site uses a three heatsink design, different than that model:

The big question is what is under the small heatsink since I would guess the first two cover the SAS 2008 controllers.

 

[cactus]

Maybe a PCI-e bridge chip

 

[Patrick]

I would be just fine if the 9202 were two 9200s stuck together. I need the bandwidth that comes from more PCIe lanes and more RAID controller chips but since I don't use the hardware RAID feature, I wouldn't mind not being able to create volumes across all four SAS ports. Given the results from my first tests, however, I am really hoping that you are correct and there is a "third chip" doing something significant. Here is why:

I used eight SSD drives and two of the four x4 SAS ports in my test. Had this been a "9200 sandwich", there would be been some combination of ports that would have placed four disks on one LSI controller and four on the other with each controller having access to its own eight PCIe lanes. There would have been another combination where both ports were attached to just one of the LSI controllers - which would have been slower. No matter which two SAS ports I used, however, the performance was identical and, further, it was nearly identical to a single LSI 9200-8e card.

When I receive the balance of my cable order, I'll hook up 16 drives and then re-test. Hopefully this will somehow cause the card to utilize both controllers and all 16 lanes at the same time, providing more bandwidth than during my 8 disk tests. I have to admit that I'm not feeling confident. I cannot imagine an architecture that is ONLY able to utilize both controllers when ALL of the SAS ports are in use. A very careful read of the specification sheet reveals that LSI is not making performance claims for the x16 interface. Instead of declaring that x16 provides better actual throughput, they merely state that x16 PCIe provides "faster signaling for high bandwidth applications".

Really interesting. I would have thought the third chip is some sort of PCIe switch. If you are on the same controller with any combination of ports, then something must be sitting in front of the two SAS 2008 controllers.

Have you looked at the card with MSM?

 

[mobilenvidia]

The LSI9201-16e uses a giant Heatsink too
It cover the the SAS2008 and the expander chip

It could be the expander to link the 2 cards together, would seem odd to have an expander.
No need for a PCIe switch as it has 16x lanes, so each chip has access to 8x lanes.
Unless using a 2nd device on the outer lanes may cause issue to be recognised, and that chip fixes it

Need to get the heatsink off, if only we knew some one with a card that was fearless and brave

 

[dba]

The LSI web site says that the LSI SAS2008 chip has an eight lane PCIe interface. The card has a 16 lane interface, so there must be a PCIe switch on the card. I very much hope that that extra switch does not end up killing throughput instead of improving it.

Sorry my Kiwi friend, no disassembly yet! I'm going to wait until I am done testing.

The LSI9201-16e uses a giant Heatsink too
It cover the the SAS2008 and the expander chip

It could be the expander to link the 2 cards together, would seem odd to have an expander.
No need for a PCIe switch as it has 16x lanes, so each chip has access to 8x lanes.
Unless using a 2nd device on the outer lanes may cause issue to be recognised, and that chip fixes it

Need to get the heatsink off, if only we knew some one with a card that was fearless and brave