Unique OEM Dual LSI 2308 + Mellanox ConnectX-3 VPI Combo Expansion Cards

BlueFox

Well-Known Member
Oct 26, 2015
1,100
532
113
I pulled these out of a Data Direct Networks appliance last year and never got around to playing around with them. I think as far as expansion cards go, these are pretty unique. You have 2 x LSI 2308 HBAs (under the top left and top right heatsinks), which are each hooked up to pair of SFF-8614 external SAS ports (4 total). On top of that, there is also a Mellanox ConnectX-3 VPI NIC with single QSFP port (bottom left heatsink). To make all the magic happen, there is a 32 lane PCIe switch (PLX 8732), so no dealing with PCIe bifurcation or the likes (bottom right heatsink). As such, they show up as 3 independent devices and take the standard drivers so far as I can tell.

Under Linux, everything is recognized, however I get errors on the HBAs for some reason. Maybe they need to be flashed with a current IT/IR firmware?

Code:
bluefox@test1:~$ dmesg | grep mpt2sas
[    2.317399] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8139932 kB)
[   17.736004] mpt2sas_cm0: sending diag reset !!
[   48.925580] mpt2sas_cm0: diag reset: SUCCESS
[   53.899130] mpt2sas_cm0: _base_spin_on_doorbell_int: failed due to timeout count(10000), int_status(c0000000)!
[   53.900847] mpt2sas_cm0: doorbell handshake int failed (line=4625)
[   53.902538] mpt2sas_cm0: _base_get_ioc_facts: handshake failed (r=-14)
[   53.904383] mpt2sas_cm0: failure at /build/linux-uI6OXA/linux-4.15.0/drivers/scsi/mpt3sas/mpt3sas_scsih.c:10768/_scsih_probe()!
[   53.920618] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8139932 kB)
[   69.298519] mpt2sas_cm1: sending diag reset !!
[  100.508994] mpt2sas_cm1: diag reset: SUCCESS
[  105.482682] mpt2sas_cm1: _base_spin_on_doorbell_int: failed due to timeout count(10000), int_status(c0000000)!
[  105.484531] mpt2sas_cm1: doorbell handshake int failed (line=4625)
[  105.486350] mpt2sas_cm1: _base_get_ioc_facts: handshake failed (r=-14)
[  105.488336] mpt2sas_cm1: failure at /build/linux-uI6OXA/linux-4.15.0/drivers/scsi/mpt3sas/mpt3sas_scsih.c:10768/_scsih_probe()!
I have not had time to tinker with them further and likely won't anytime soon (seems telecommuting full time just means I work more). NIC appears to work as expected at least. Due to this, please be aware that they are being sold as-is and meant to be a potentially interesting project. I've priced them accordingly. International shipping is an option, though the extra cost would be at your expense.

$25 shipped each - 0 Available - 6 Sold

 
Last edited:

BlueFox

Well-Known Member
Oct 26, 2015
1,100
532
113
It is. About the same size as my desktop GPU, sans the heatsink of course.
 

BlueFox

Well-Known Member
Oct 26, 2015
1,100
532
113
Unfortunately I don't recall (and can't check now). Maybe one of the people that bought one can answer once USPS delivers them.
 

Crazywatermelon

New Member
May 18, 2016
4
3
3
Received mine yesterday, no issues with the card and as reported the NIC works no issue but I can't get the 2308 chips to work. I have had no success flashing the card, sas2flash complains when running any command and says it needs firmware but it rejects any firmware even when using the older sas2flash (P14) that's usually fine with anything and Megarec won't detect it. If anyone winds up being able to flash it I'd appreciate a hand.
 

Freebsd1976

Active Member
Feb 23, 2018
227
33
28
Received mine yesterday, no issues with the card and as reported the NIC works no issue but I can't get the 2308 chips to work. I have had no success flashing the card, sas2flash complains when running any command and says it needs firmware but it rejects any firmware even when using the older sas2flash (P14) that's usually fine with anything and Megarec won't detect it. If anyone winds up being able to flash it I'd appreciate a hand.
the nic is qcbt or fcbt?
 

Sleyk

Well-Known Member
Mar 25, 2016
1,039
425
83
Stamford, CT
Hello! So I snagged two of these bad bois from @BlueFox. Cards came nicely packaged. (Thanks Brother!)

Thanks everyone for all your patience! I got these a little while ago, but I just got a chance to finish up my testing.

The TLDR? Not good news so far. Things can change, but before you dive into the post, just know I was not successful :.(

Now, if you still wanna follow my testing, sure thing! Let's do it. I will report my findings and what happened.

Ok, so let's get into it.

So these cards are straight proprietary, OEM variants, with their own custom firmware. They are labeled exactly as shown in the pic in the first post above: "Data Direct Networks".

I first dropped one into my testbench, and began my testing. I too, found the QSFP port to be functioning normally. I didn't do any in-depth testing on the NIC, as I was solely interested in the dual-SAS2308 chipsets onboard. I used a QSFP to SFP+ adapter. Please, no questions on the NIC, as the card is essentially unusable for 99% of people, so let's forget about the NIC going forward.

Now, I fired up SAS2FLSH.

The very first thing I did was type: sas2flsh -list so I could see what will be listed, but it appeared that the firmware was nonfunctional from the start, and it required a hostboot. It did report itself as having a 2308_2(B2) Chipset. (Remember that for later)



After which, I obviously wanted to hostboot the card, so I typed: 9211it.bin, hoping to hostboot the card, so I could flash it, but alas, I made a rookie mistake, because the chipset is not a SAS2008 chipset, it is a SAS2308 chipset, so the older SAS2008/9211it firmware won't work here, lol! Behold my rookie mistake and laugh!



Realizing my silly mistake, I typed the correct firmware, 9207it.bin, which is surely the LSI 9207it.bin firmware to hostboot right?
I was wrong, yet again...



What was this? I thought to myself. Isnt the 2308 chipset a 9207-8i/8e? But alas, it did not work. So I stopped and thought for a moment. Then it came to me. Yes! I remembered that some OEM's took the 9205 and rebranded it as a 9207. (Well, there's more to it than that, but for the sake of simplicity, let's go with that).

So I went to Broadcom's site, and hunted up a copy of the LSI 9205-8e IT firmware. I then crossed my fingers, and gave it a try. I was super excited when I saw that it RECOGNIZED the 9205it firmware bin file and started the hostboot...



But alas, as you can see once again, it succeeded in hostbooting the card, but it failed to copy over the firmware. Now, I was really at a loss, because the 9207 firmware didnt work, and even the 9205 firmware didnt work. This pissed me off and I went again, hunting down some more firmware to try on this forsaken card. I wasn't gonna give up so easily.

I found on Broadcom's site, three firmwares of interest, that might be most compatible with the card:

1. LSI 9205-8e (IT and IR)
2. LSI 9206-16e (IT)
3. LSI 9202-16e (IT)

Of which the first, the 9205-8e firmware I had already tried and was able to get a successful hostboot.

I then began to tinker and try to see if I could get any of these firmwares to load onto the card. But no matter what I did, it always ended in failure. I also noticed that perhaps due to me hostbooting with the 9205-8e firmware, it now reported itself as a 2308_2(C1) controller, instead of 2308_2(B2) chipset. I suppose it changes based on what firmware you use to hostboot. In any case, the flash/hostboot attempt would always fail, and I would always get the message: "Failed to Upload Image! Firmware validation failed!" :mad:



This is when I started to suspect that the card was locked the hell down by Data Direct Shitworks. I mean, if it is a 2308 chipset, it must be able to see the firmware. But alas, multiple attempts and all sorts of finagling and tinkering led nowhere. I got the exact same message every time. What a damn crock.

I then thought to go in another direction, and see if I could low-level this bad boi, and maybe get to the megaraid firmware all LSI chipsets have embedded in them, to see if perhaps I could rewrite the SBR and conquer this bitch. However, this too, ended in failure, as the venerable megarec itself, could see no MR controllers onboard in the system, which essentially means, were screwed. Lol!



Then, after trying everything I knew, I tried one last ditch effort to see if I could make this thing move. I hex edited a few areas of the 9206-16e firmware, using notepad++ and tried to flash the card with that. It returned a firmware fault. The card did see the firmware, but I supposed when it checked it for security validations, it failed. All the while, the card still reported itself as a 2308_2(C1) chipset.



Now, out of options, I decided to do a full erase, and reset the damn thing, essentially giving up. I went ahead, and did a sas2flsh -o -e 7 and erased the whole thing. It actually went through the process, so I know sasflsh has control over the card, but something just doesnt want to work right. I was curious to see what the card looked like prior to hostbooting with the 9205-8e firmware (which was the only firmware that let me successfully hostboot) and I typed: sas2flsh -list once more, just to see what would come up. I dont know how, but surprisingly, I got an output. The output was essentially nothing, but still I got something:



If you notice, the card once again reports itself as a 2308_2(B2) chipset. This was so odd and strange, but alas, such it is. My suspicion was that the card carries some sort of low level backup that overwrites a -o -e 7 erase. This can only be done by the OEM themselves.

My suspicions were quickly verified, because now that I was out of options, and nothing worked, I did the one thing I hated above all else. The one thing I try my utmost NEVER to do.

I called the company to ask for help. Forgive my tech tinkering sin.

Yes folks, I contacted Data Direct Networks. I got the answers I was suspecting, but I'm still slightly pissed off about it. Damn you proprietary OEM's. Damn you.

I called DDN, and I got a customer rep, who happily transferred me over to tech support (Bleh, I wanted to vomit). The guy was rather nice on the phone though. I told him I had one of their cards, and that I needed help to perhaps score their firmware flash utility to make the thing come alive.

It was here he stopped me. He asked me if I had a support contract with Data Direct Networks. I told him no. (I didn't feel like lying, plus I wouldn't have been able to pass verification anyway if he asked). He then said he can only talk to people who are under contract for support.

Ok, ok, fair enough. Alot of companies do this. They only offer support to paying clients who pay a premium for support. I told him that was fine, and I then explained to him that I am only trying to work with the card, as I got it from a friend. I told him I have sufficient knowledge to flash the card, I just need to know what tools or utilities they use, as the normal tools do not seemingly work.

He kept saying that he couldn't assist me, as I didn't have a contract. I then tried to level with the chap, tech guy to tech guy, and told him I get it, but to please give me something. I suppose the guy felt sorry for me, and he told me that Data Direct Networks has custom proprietary software that validates the card once in one of their boxes, and as such, even if they helped me, as I didn't have one of their custom servers, the card would never work.

That's when it hit me. Data Direct Networks, henceforth and forever now known to me as (Data Direct Shitworks) locked down the card. The 2308 chipset is visible to sas2flsh, but nothing can flash it except for their proprietary software, included in their custom server boxes.

It was at this point, I fully understood, and decided to give up.

I thanked the tech support rep, and hung up.

And that folks, is where the journey ends. The card indeed possesses dual 2308 LSI chipsets, but they are locked down like a mofo. Only Data Direct can make it function, and it is only able to function in their custom server boxes.

At least, for now. I think since sas2flsh can see the chipset, it may be possible.

However...

I fail to find the motivation to dedicate all my time to further defeat Data Direct Shitworks proprietary software for a 2nd Gen LSI chipset for external use. I just dont see it. Plus, 2 x 9207-8e cards can be had very cheaply, so no real reason for us to even bother trying to defeat this thing.

There are simply plenty of external port cards out there to bother.

So there you have it. My full testing experience. I really like this card. I do. But gosh darn it, DDN locked this card down so tight, its simply pointless.

I hope you at least enjoyed the read though. It was fun playing with this card. I suppose someone may pick this up and try to break this thing, as sas2flsh can almost flash it, but honestly, I wouldn't waste the time.

Thanks to Bluefox for introducing this card to me. I never even knew about Data Direct as an OEM. I may keep it, or I may sling them on ebay to some other poor sap who has the time to tinker. Again, I don't think it's impossible, and the solution may even be very simple, but I just don't see it right now.

Ah well, at least I must say, it was fun. :)
 
Last edited:

BlueFox

Well-Known Member
Oct 26, 2015
1,100
532
113
Quite the write-up! Have to say, it's a bit disappointing that they did that considering the server I pulled these out of was not one of their proprietary ones, but in fact a Supermicro 6037R-TXRF (only deviation from stock was adding a second fan to cool the cards). I bought it off eBay last year, but as expected, the drives had been pulled. Photo if you're curious: https://drunkencat.net/misc/DSC_0892.jpg

I think the only thing left to try would be to use an EEPROM programmer to flash the LSI chips directly. Presumably they're the 8 pin ones with the yellow dot directly to the right of the respective heatsinks. CH341A + SOIC8 clip (+ maybe a 1.8V adapter depending on the chip) only runs ~$10 on eBay. Not sure how much more effort you'd want to put into these though?
 

Sleyk

Well-Known Member
Mar 25, 2016
1,039
425
83
Stamford, CT
Quite the write-up! Have to say, it's a bit disappointing that they did that considering the server I pulled these out of was not one of their proprietary ones, but in fact a Supermicro 6037R-TXRF (only deviation from stock was adding a second fan to cool the cards). I bought it off eBay last year, but as expected, the drives had been pulled. Photo if you're curious: https://drunkencat.net/misc/DSC_0892.jpg

I think the only thing left to try would be to use an EEPROM programmer to flash the LSI chips directly. Presumably they're the 8 pin ones with the yellow dot directly to the right of the respective heatsinks. CH341A + SOIC8 clip (+ maybe a 1.8V adapter depending on the chip) only runs ~$10 on eBay. Not sure how much more effort you'd want to put into these though?
Hmm, that is indeed interesting. Perhaps DDN cards may work in another system through some sort of security licensing deal or validation? I suppose it would make sense that they would license the card and software to make it work. It may be one such case indeed. I get why they feel like they should lock down their cards, but it's just a shame. These cards would be selling like hotcakes if they didn't do this. I suppose they want to recuperate all the money they paid to get this card custom-designed for them :.)

In any case, I may be underestimating just how big they are. They may not even want people outside of datacenter and cloud applications using the thing. When you think of it like that, you can't help but get a turning up your nose at whitebox users (cough, cough, VMware, cough) type thing. I suppose it could be solved with their special tools, but gotta be a DDN contact to get them. Not impossible, and maybe someone can snag these tools from them if they know of someone who works there or can get access to it?

Heheh, I suppose I could try to snag an EEPROM programmer, but alas, I lack the motivation to continue testing with it. Who knows, maybe I may get back to it, as I have another project I am working on that may most likely require an EEPROM flash. Until then, will probably consider keeping it and doing random testing on it from time to time. Who knows, I may stumble on something.
 
  • Like
Reactions: cesmith9999