Mellanox Innova2 CONNECT-X 5 25gbps SFP28 and XILINX Kintex ULTRASCALE+ DPU $250 + Bestoffer

[applepi]

Xilinx Kintex XCKU15P Fpga On Board | eBay

ElectroRam will always describe the item to almost accuracy.

www.ebay.com

https://docs.mellanox.com/display/Innova2Flex/NVIDIA+MELLANOX+INNOVA-2+FLEX+OPEN+ADAPTER+CARD

ThinkSystem Mellanox Innova-2 ConnectX-5 FPGA 25GbE 2-port Adapter Product Guide (withdrawn product)

The ThinkSystem Mellanox Innova-2 ConnectX-5 FPGA 25GbE 2-port Adapter is an advanced programmable network adapter that combines the advanced ConnectX-5 Ethernet network controller ASIC with an onboard state-of-the-art FPGA. Such a combination enables customers to develop custom-made offloads...

lenovopress.com

Accepted $150 each for 2 of them, only issue programming these is that it requires a Vivado License to put custom HDL on the FPGA, using it as a NIC should still work.

 

[l.code]

Looks like to be ''MNV303611A-EDLT" ?

 

[applepi]

Looks like to be ''MNV303611A-EDLT" ?

As it has the RAM and SlimSAS OPENCAPI connector it's the MNV303212A-ADLT.

 

[applepi]

Okay just got these cards today, It seems they just had no firmware installed on the device, I flashed the latest firmware availble through the mellanox site and reflashed using the recovery options in flint similar to how the connectx-4 cards were with blank firmware. After flashing I rebooted and ran mlxupdate to update to the latest firmware and then set the GUID and MAC to arbitrary values that are different from the other NIC I have.

I have not yet flash the FPGA on both NICs but one appeared to be flashed (based off the LED status)

Once I install linux on my test lab computer I should be able test flashing the default FPGA firmware and testing out these NICs.

PS. I also found some ADM-PCIE-8V3 on ebay for $50 which are a bit older FPGA but have QSFP28 transceivers connected directly to the FPGA

 

[l.code]

@applepi , Good to hear.

Still waiting for the item's delivery.

Will post updates here too.

 

[l.code]

The links for `ADM-PCIE-8V3 ` boards? Failed to find.

 

[applepi]

The links for `ADM-PCIE-8V3 ` boards? Failed to find.

I don't know why I can't see it as a completed listing either.

 

[applepi]

Okay, just a heads up to anyone trying to get these cards working, DO NOT flash the Lenovo firmware or firmware in the bundle. It will secure-fw your card and you will no longer be able to flash unsigned firmware to the card. I got around this by using a CH341A mini programmer and a russian CH341 programmer software to flash the Winbond 128MBIT WSQ128 SPI flash to backup and wipe the firmware and then I could have the card boot into frimware recovery mode and flash unsigned firmware.

 

[americanfuzzylop]

According to https://docs.mellanox.com/display/Innova2Flex/NVIDIA+MELLANOX+INNOVA-2+FLEX+OPEN+ADAPTER+CARD there should be an active heat sink to cool the FPGA. Is it present on your card? Or, what's your cooling solution?

Trying to get one of those to experiment with FPGA development.

 

[applepi]

According to https://docs.mellanox.com/display/Innova2Flex/NVIDIA+MELLANOX+INNOVA-2+FLEX+OPEN+ADAPTER+CARD there should be an active heat sink to cool the FPGA. Is it present on your card? Or, what's your cooling solution?

Trying to get one of those to experiment with FPGA development.

Yes they come with a cooler, the only problem is that one of my cards wouldn't keep timing once programmed

 

[americanfuzzylop]

Bummer, mine does not come with a cooler. Would you mind taking a picture of yours? Thanks!

wouldn't keep timing once programmed

Did you mean there's timing violation of the logic programmed to the FPGA?

By the way, I saw a lot of folks (100+) purchased the card. Am wondering if there's an online mailing list, etc. to discuss experience playing with this card.

 

[anything1233]

@americanfuzzylop Here’s mine (clickable):

Ordered 3x, haven’t tested yet. But even with the meticulous packaging and a mile of adhesive tape, one arrived with a bent PCB, chewed up OpenCAPI connector, and a stuck fan. But it’s fixable. Though that timing issue is a little intimidating... @applepi please keep us posted

 

[americanfuzzylop]

But even with the “meticulous” packaging, one arrived with a bent PCB, chewed up OpenCAPI connector, and because of that the cooler has stuck as well

That's too bad... sorry to hear that

And thanks for the photo! I guess I will just stick two heat sinks to the FPGA and the Eth PHY.

What do you plan to do with the FPGA? It's a pretty decent chip. I do have Vivado licenses (have not checked if this particular chip is covered, but all Ultrascale+ chips should be covered), but have not decided what to do.

 

[Shimrod]

Hi
I saw this boards on Ebay. A lot of stuff for the price if they are working.
Any updates on success on using them?

Cheers
Rolf

 

[aleruggeri87]

Hi everybody,
I also saw the board a few weeks ago on ebay, and looking for additional info I found this thread.
Anyone of the guys who has bought the board (@applepi, @anything1233, @americanfuzzylop) has some update to share?
I'm a little bit worried for what @applepi said regarding the timing; on the pictures on ebay I can see that a few decoupling capacitors are missing, what is the situation on your boards?

Thanks and regards,
Alessandro.

 

[Cute_SC2]

I bought some 2 years ago, the boards are sold broken, only the fpgas themself are working and needed to be soldered onto another working board.
There is a JTAG header on the top left side of the board that can be used for programming, also it can be programmed over pcie with the mellanox tools.
For simple function testing without loaded bitstream the heatsink can be skipped.

 

[mwrnd]

I just came across this forum. I have been working on detailed notes for my Innova-2 for a while.

GitHub - mwrnd/innova2_flex_xcku15p_notes: Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development

Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development - mwrnd/innova2_flex_xcku15p_notes

github.com

These boards are being sold with no guarantees but with some effort they can be refurbished and make excellent FPGA Development Boards. I was able to fix capacitor, JTAG, and a DDR4 Byte Lane errors and now have a mostly functional board.


@Cute_SC2 - Please do not skip the heatsink. Running the Flex Image the FPGA gets above 70C. At idle with no SFP cables/modules the ConnectX-5 IC on my board overheats within minutes and its SFP status LEDs go red. It gets hot enough to burn skin.

@aleruggeri87 - When I slow the memory down to DDR4-1600 my designs meet timing. Better cooling also seems to help.

@applepi - Consider a thorough wash of your board in 99% Isopropyl Alcohol in an Ultrasonic Cleaner to remove any dust under the BGA ICs. Also, try improving your cooling solution. Larger heatsink and a server blower fan aimed directly at the board.

@anything1233 - "one arrived with a bent PCB" - Do you think it can be salvaged? If you or anyone else has a truly unsalvageable board and are planning to desolder the ICs, please consider spending some time to buzz out the IC-to-TestPoint connections with a multimeter.

These boards are too capable to scrap. The XCKU15P has 1M+ Logic Cells. Closest retail product would be the Xilinx Alveo U25.

If the power supply works it is worthwhile to try and fix your board.

 

[bayleyw]

I just came across this forum. I have been working on detailed notes for my Innova-2 for a while.

GitHub - mwrnd/innova2_flex_xcku15p_notes: Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development

Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development - mwrnd/innova2_flex_xcku15p_notes

github.com

These boards are being sold with no guarantees but with some effort they can be refurbished and make excellent FPGA Development Boards. I was able to fix capacitor, JTAG, and DDR4 Byte Lane errors and now have a mostly functional board.

innova2_flex_xcku15p_notes/Troubleshooting at main · mwrnd/innova2_flex_xcku15p_notes

Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development - innova2_flex_xcku15p_notes/Troubleshooting at main · mwrnd/innova2_flex_xcku15p_notes

github.com

GitHub - mwrnd/innova2_flex_xcku15p_notes: Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development

Nvidia/Mellanox Innova-2 Flex Open Programmable SmartNIC Setup and Usage Notes for XCKU15P FPGA Development - mwrnd/innova2_flex_xcku15p_notes

github.com

@Cute_SC2 - Please do not skip the heatsink. Running the Flex Image the FPGA gets above 70C. At idle with no SFP cables/modules the ConnectX-5 IC on my board overheats within minutes and its SFP status LEDs go red. It gets hot enough to burn skin.

@aleruggeri87 - When I slow the memory down to DDR4-1600 my designs meet timing. Better cooling also seems to help.

@applepi - Consider a thorough wash of your board in 99% Isopropyl Alcohol in an Ultrasonic Cleaner to remove any dust under the BGA ICs. Also, try improving your cooling solution. Larger heatsink and a server blower fan aimed directly at the board.

@anything1233 - "one arrived with a bent PCB" - Do you think it can be salvaged? If you or anyone else has a truly unsalvageable board and are planning to desolder the ICs, please consider spending some time to buzz out the IC-to-TestPoint connections with a multimeter.

These boards are too capable to scrap. The XCKU15P has 1M+ Logic Cells. Closest retail product would be the Xilinx Alveo U25.

If the power supply works it is worthwhile to try and fix your board.

Wow, that's an incredible amount of documentation. Thanks!

 

[anything1233]

@mwrnd the docs you‘ve made are pure gold! The sheer amount of effort you dedicated for this is simply mind-boggling! Thank you!

Do you think it can be salvaged?

The bent part mainly affected power delivery part, and while I haven’t tested the FPGA itself yet, ConnectX-5 does work just fine. Just had to fix the cooler.

I was able to fix capacitor

Which one?

 

[Patrick]

@mwrnd Absolutely awesome documentation.