Mellanox Innova2 CONNECT-X 5 25gbps SFP28 and XILINX Kintex ULTRASCALE+ DPU $250 + Bestoffer

tozmo

Active Member
Feb 1, 2017
116
73
28
72
@mwrnd What's the hobbyist "use case" for that beefy FPGA? How much for the Verilog toolchain? ;-)
I'd like to know the use case also. Maybe to unpack and re-iso Linux distros at light speed? Play FLAC's across your network onto your Amazon Fire without buffering?
 

Stephan

Well-Known Member
Apr 21, 2017
563
360
63
Germany
With that kind of super-beefy FPGA and at that literally throw-away price, a student of MINT subjects interested in RISC-V might get one or two for research. There's been nice progress of synthesizable cores on FPGAs, but a lot of low hanging fruit is still up for grabs. Like read what DEC Alpha published on 21064 back in the 1990s, what Jim Keller is saying in interviews about branch predictors, and what Apple did for M1 to make it THAT much faster than everything ARM before. Impress your friends, create a RISC-V core that is faster than Apple and possibly even has no Spectre sidechannels. 150 MHz on this card, with tweaks 2 GHz on TSMC 7nm.
 

mwrnd

New Member
Jul 5, 2022
5
12
3
@bayleyw @anything1233 @Patrick - Thanks for the kudos! I spent too much time struggling with Mellanox's documentation so I wrote my own.

>> I was able to fix capacitor
> Which one?

PCIe Lane 0 was missing a capacitor. C641 at the card edge in the picture. Board is working so either my fix was successful or the differential signal is getting through on the other polarity of the pair.
Innova-2_Replaced_Capacitor.jpg

> bent part mainly affected power delivery part, and while
> I haven’t tested the FPGA itself yet, ConnectX-5 does work just fine

That sounds promising. What shows up when you run lspci?
 
Last edited:
  • Like
Reactions: Patrick

mwrnd

New Member
Jul 5, 2022
5
12
3
> What's the hobbyist "use case" for that beefy FPGA?

@Stephan - Vivado can synthesize and implement designs much faster when your project barely needs 25% of the FPGA's resources. No need for aggressive optimization. I am currently experimenting with Xilinx's DPU AI Accelerator.

@applepi - Yes, I use an Evaluation License for experiments and switch to the Vivado AWS AMI on z1d.xlarge Spot Instances with the NVMe used for swap for distributable designs. Comes out to about $0.50/hour and I rarely need more than 3 hours for a design. Ignore what Xilinx recommends for its AMI. Utilizing more than 4 cores on an instance with less than 32GB of memory overwhelms kswap. Vivado needs about 4GB per vCPU it uses on top of a base ~10GB, plus whatever Ubuntu needs.

@Stephan - RISC-V - Yes, it is on my experiment TODO list :)

@tozmo - Nvidia Networking has the FlexDriver project for the Innova-2 which can supposedly do direct NIC-to-FPGA Bump-In-The-Wire processing of packets. On-the-wire encryption/decryption is one example. Unfortunately it requires you to somehow get FlexDriver IP (src/flc.dcp) from NVIDIA Networking. The Product Brief also mentions transparent data compression and decompression.

The Innova-2 shows up in lspci as having a PCIe Bridge with the GbE ports and FPGA being independent but in one convenient package. The Product Brief mentions it has PCIe Switch but I have not yet found any documentation on how to do direct ConnectX-to-FPGA communication.

Some other interesting project ideas: NYUZI GPGPU, RV64X GPU, MIAOW GPU, LiteX Projects Page
 
Last edited:

TrumanHW

Member
Sep 16, 2018
150
21
18
Okay, just a heads up to anyone trying to get these cards working, DO NOT flash the Lenovo firmware or firmware in the bundle. It will secure-fw your card and you will no longer be able to flash unsigned firmware to the card. I got around this by using a CH341A mini programmer and a russian CH341 programmer software to flash the Winbond 128MBIT WSQ128 SPI flash to backup and wipe the firmware and then I could have the card boot into frimware recovery mode and flash unsigned firmware.


For the longest time I didn't know how to do my ROM work using a CH341 on a Mac ... and was one of the only reasons for which I kept another little PC (now I have them for other reasons, but I still FAR and away prefer the HEX editors, etc., on Macs) ...

Did you know that there's a cake-simple terminal application you can do ROM flashing via MacOS ..?

The terminal app is called FlashROM

I can send you a tutorial on that (very easy and you're obviously super knowledgeable so usually I'd expect to be asking you questions). ... and, if you wanted to use a different BIOS interface than the CH341, I'll look up the name of the other application that works with that.

Anyway, nicely done reprogramming these. Gives me an idea of how to get SFP28 cards for a little less than market now, too. :)

I went crazy trying to find ways to read the ROM off my CH341 and it just never dawned on me that the computer was already seeing the programmer (device) I just needed to install the application and dependencies to make it work (for years, to my shame). Maybe this helps someone else. :)


If you have a ... TL866

There's another application called called MiniPro ...

Anyway, thanks for this post.
 

mwrnd

New Member
Jul 5, 2022
5
12
3
@ciclonite - Thanks! Please update us on your testing of the boards and whether DDR4 works for you. Do you have different board revisions or are they all the same? @Cute_SC2 has noted the possibility of different DDR4 pinouts on the various revisions.

> idea of how to get SFP28 cards for a little less than market

@TrumanHW - The cards manufactured ~5 years ago are being refreshed/replaced. The board I have has a significant retransmit rate with a Direct-Attach or Active-Optical cable between the interfaces at 25GbE. I have asked about acceptable iperf3 Retr rates. In any case, this is the best value higher-end FPGA board I have come across scouring eBay the past few years.
Innova-2_iperf3_Test.jpg