Unique OEM Dual LSI 2308 + Mellanox ConnectX-3 VPI Combo Expansion Cards

[BlueFox]

I pulled these out of a Data Direct Networks appliance last year and never got around to playing around with them. I think as far as expansion cards go, these are pretty unique. You have 2 x LSI 2308 HBAs (under the top left and top right heatsinks), which are each hooked up to pair of SFF-8614 external SAS ports (4 total). On top of that, there is also a Mellanox ConnectX-3 VPI NIC with single QSFP port (bottom left heatsink). To make all the magic happen, there is a 32 lane PCIe switch (PLX 8732), so no dealing with PCIe bifurcation or the likes (bottom right heatsink). As such, they show up as 3 independent devices and take the standard drivers so far as I can tell.

Under Linux, everything is recognized, however I get errors on the HBAs for some reason. Maybe they need to be flashed with a current IT/IR firmware?

bluefox@test1:~$ dmesg | grep mpt2sas
[    2.317399] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8139932 kB)
[   17.736004] mpt2sas_cm0: sending diag reset !!
[   48.925580] mpt2sas_cm0: diag reset: SUCCESS
[   53.899130] mpt2sas_cm0: _base_spin_on_doorbell_int: failed due to timeout count(10000), int_status(c0000000)!
[   53.900847] mpt2sas_cm0: doorbell handshake int failed (line=4625)
[   53.902538] mpt2sas_cm0: _base_get_ioc_facts: handshake failed (r=-14)
[   53.904383] mpt2sas_cm0: failure at /build/linux-uI6OXA/linux-4.15.0/drivers/scsi/mpt3sas/mpt3sas_scsih.c:10768/_scsih_probe()!
[   53.920618] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8139932 kB)
[   69.298519] mpt2sas_cm1: sending diag reset !!
[  100.508994] mpt2sas_cm1: diag reset: SUCCESS
[  105.482682] mpt2sas_cm1: _base_spin_on_doorbell_int: failed due to timeout count(10000), int_status(c0000000)!
[  105.484531] mpt2sas_cm1: doorbell handshake int failed (line=4625)
[  105.486350] mpt2sas_cm1: _base_get_ioc_facts: handshake failed (r=-14)
[  105.488336] mpt2sas_cm1: failure at /build/linux-uI6OXA/linux-4.15.0/drivers/scsi/mpt3sas/mpt3sas_scsih.c:10768/_scsih_probe()!

I have not had time to tinker with them further and likely won't anytime soon (seems telecommuting full time just means I work more). NIC appears to work as expected at least.

 

[Samir]

Really neat card! glws!

 

[Crazywatermelon]

PM'd

 

[abq]

Really neat card! glws!

Ditto, very cool cards - but probably beyond my abilities.

 

[R30730]

PM coming with some Qs.

 

[Patrick]

That is very cool!

 

[Kev]

Looks like a full-height card?

 

[BlueFox]

It is. About the same size as my desktop GPU, sans the heatsink of course.

 

[BlueFox]

Thanks for all the interest. I'm all out, but for anyone that missed out, there is an eBay seller that has some available, albeit for a bit more: DATA DIRECT NETWORKS INC 04-00232-614 REV 2 614 REV A0 3914 | eBay

 

[Freebsd1976]

thanks @BlueFox ， the Mellanox ConnectX-3 VPI is include is QCBT or FCBT?

 

[BlueFox]

Unfortunately I don't recall (and can't check now). Maybe one of the people that bought one can answer once USPS delivers them.

 

[Freebsd1976]

thanks

Unfortunately I don't recall (and can't check now). Maybe one of the people that bought one can answer once USPS delivers them.

, wait someone receive card and report back , btw did you success with the 2308?

 

[BlueFox]

No, I mentioned that in my original post.

 

[Crazywatermelon]

Received mine yesterday, no issues with the card and as reported the NIC works no issue but I can't get the 2308 chips to work. I have had no success flashing the card, sas2flash complains when running any command and says it needs firmware but it rejects any firmware even when using the older sas2flash (P14) that's usually fine with anything and Megarec won't detect it. If anyone winds up being able to flash it I'd appreciate a hand.

 

[Freebsd1976]

Received mine yesterday, no issues with the card and as reported the NIC works no issue but I can't get the 2308 chips to work. I have had no success flashing the card, sas2flash complains when running any command and says it needs firmware but it rejects any firmware even when using the older sas2flash (P14) that's usually fine with anything and Megarec won't detect it. If anyone winds up being able to flash it I'd appreciate a hand.

the nic is qcbt or fcbt？

 

[BlueFox]

Quite the write-up! Have to say, it's a bit disappointing that they did that considering the server I pulled these out of was not one of their proprietary ones, but in fact a Supermicro 6037R-TXRF (only deviation from stock was adding a second fan to cool the cards). I bought it off eBay last year, but as expected, the drives had been pulled. Photo if you're curious: https://drunkencat.net/misc/DSC_0892.jpg

I think the only thing left to try would be to use an EEPROM programmer to flash the LSI chips directly. Presumably they're the 8 pin ones with the yellow dot directly to the right of the respective heatsinks. CH341A + SOIC8 clip (+ maybe a 1.8V adapter depending on the chip) only runs ~$10 on eBay. Not sure how much more effort you'd want to put into these though?

 

[Dave Corder]

I know this is an older thread, but I just stumbled upon it this past weekend. I'm intrigued a bit by this card - it'd save me a slot in my somewhat constrained whitebox FreeNAS build, if only it actually worked

I'm kinda tempted to pick one of these up and try flashing a new SBR directly to the EEPROM - I think I have all the pieces I'd need for that, but I've not done it before. Is there anyone who might be able to assist with that if I wanted to give it a try?

 

[BlueFox]

Shouldn't be too much work. EEPROM for the HBAs is the 8 pin chip to the right of each heatsink with a yellow dot on it. With a CH341A programmer and SOIC8 clip, you should be all set.

 

[Dave Corder]

Well, just snagged one of these cards from a seller on eBay for 20 bucks. Looks like I have a weekend project coming up.

I've used my lunch break to brush up on LSI flashing and SBRs, which took me to the threads on cross-flashing the Dell integrated storage slot cards. And there's some info there on a procedure to cross-flash using megarec and then lsirec/lisutil to flash the card in Linux instead of sas2flash. I might be able to use some of that info with this card...will have to do some testing before I go the EEPROM route, just to see if I manage to stumble on anything that @Sleyk didn't.

Actually, scratch that, first step will be to dump the SBR with the CH341A so I have a backup in case I accidentally brick something, and then deconstruct it to see if it can reveal any secrets.

 

[sparx]

You should probably be able to flash this card with the right binary file. For 3008 based cards there is a check if you are doing a correct upgrade for the card, and since you dont have a firmware file for this specific card but trying to run a "vanilla" card in it, it will not pass the sa2flash check and plainly refuse to upgrade.
I would download all the 2308 releases that are available. If you can get the binary from the card itself you should be able to hex compare the binary from the card with a vanilla card 2308(of same release version) and see what changes are made. There will be a few. But it could be manageable to make the corrections in a newer vanilla binary to get it working.