https://www.ebay.com/itm/147411792503 - $85 offer accepted (took a few hours). $90 auto accept.
$85 - $90 - NVIDIA MBF2H332A-AECOT BlueField-2 SmartNIC P-Series DPU 25GbE PCIe4
- Thread starter foureight84
- Start date
-
- Tags
- bluefield bluefield-2 dpu mellanox
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
incredible deal:
modern ConnectX-6 Dx-class dual-25GbE NIC with an onboard 8-core ARM Linux computer, 16GB ECC RAM, 64GB flash, OOB/IPMI, crypto/RDMA/RoCE/SR-IOV/DPDK/DOCA offloads, and all under familiar mlx5 driver. all the tech, and extra PC.
the cutting edge ++
reminds me of mikrotik CCR2004-1G-2XS-PCIe, but all the modern tech.
modern ConnectX-6 Dx-class dual-25GbE NIC with an onboard 8-core ARM Linux computer, 16GB ECC RAM, 64GB flash, OOB/IPMI, crypto/RDMA/RoCE/SR-IOV/DPDK/DOCA offloads, and all under familiar mlx5 driver. all the tech, and extra PC.
the cutting edge ++
reminds me of mikrotik CCR2004-1G-2XS-PCIe, but all the modern tech.
So you could run some lightweight Linux Distribution on it ?
Usually a PCIe Card is meant to provide Connectivity to the Host whose PCIe Slot it is plugged into.
What is exactly the Point of that ?
I'm not debating if ConnectX-6 is a great Deal, but I don't really see how having an onboard Computer is really a huge Thing.
The only Thing I could see, but probably it's disabled for Security Reasons, is it it was able to control the Host Computer THROUGH the PCIe Slot. Basically kind of a ASPEED BMC but on a PCIe Card. At the same Time I think that would be a Security Nightmare.
one usecase: a router in a server, if you got only 1U in the rack. but with these features, i wonder if traffic capturing/SPAN is viable.
i've just ordered, but i also expect some sort of kernel-bypass solution(seems code changes are reqd) and line-rate recording/mirroring (im in a tiny HFT)
but more consisely, this NIC is like connect-x with all the tech avail by-2023. only fiber though.
i've just ordered, but i also expect some sort of kernel-bypass solution(seems code changes are reqd) and line-rate recording/mirroring (im in a tiny HFT)
but more consisely, this NIC is like connect-x with all the tech avail by-2023. only fiber though.
Last edited:
Why not simply use Proxmox VE (or XEN if you feel like it) on the Host, virtualize OPNSense, then possibly using this NIC with SRIOV for Virtual Machines if you need "direct" Access ?
ARM SoC practically means that you could run OpenWRT, some lightweight/normal Linux Distribution (e.g. Debian), but won't let you use OPNSense for instance.
I guess it also depends how many Physical/Virtual Functions exist, e.g. 1 for the internal Computer, 1 for the Host, X virtual Functions that you can assign to other Guests.
we have different usecases, our proxmox is served by 15year old cards just fine, im just caring about other usecases. if you are not interested in most-recent reincarnation of connnect-x with max kit+pc-on-board, move along.
its not only about sr-iov. and not only about roce. it's a max-kit to me. (and honestly to see what nvidia did to mellanox)
Last edited:
EDIT: Found an old STH article. https://www.servethehome.com/zfs-wi...he-nvidia-bluefield-2-dpu-nvme-arm-aic-iscsi/
That explains it better what you write up. But it's more complicated than the article covers in actuality. I'm going to be running it in an x86 host but the main goal is to custom compile Suricata to utilize hardware offloading via DOCA flow. Plus it will be highly selected ruleset for suricata so that I can use it for 10GBE symmetric IPS (I want to see how close to full speed is possible). There's no way the 8-core can perform software deep packet inspection.
Last edited:
That seller publishes them every now and then in small batches. I have a couple, they worked fine, but had leftovers of previous customer. Notably, you needed to get to the EFI and reset the signing keys or disable secure boot to boot the stock BFB image and reflash it (the EFI password is the default one from the manual, though, so no surprises).
By default it runs Ubuntu, in latest releases of DOCA - ubuntu 24.04. It is 8xCortex A72 cores at 2.5 GHz each, so not so weak, and 16GB of RAM (btw, with ECC) is also enough for quite some things.
Purpose - you have an SDK that allows you to do some shenanigans - e.x. run a part of your SDN on the NIC and make it transparent to the host. Another example is NVMe-oF - NIC gets NVMe-oF from somewhere and emulate a local NVMe drive on same PCIe bus for the host to boot from for example. I personally haven't played much with the SDK, but what I do is I put one of those cards into a dedicated box and I run VPP (Vector Packet Processing) on it and use it as my home router (I have 25G internet connection from my ISP at home, so the NIC do all the heavy lifting with NAT and stuff, and behind it I have a Mikrotik CRS504 switch - that still consume less power than my previous home router which was x86-64 box with couple of ConnectX-4 Lx cards and for the purpose of NAT it is actually faster than x86 box was).
Some of those cards (M512 and H532 versions, but not 332) has ASpeed 2500 onboard as well with OpenBMC flashed, on this one there is a BMC emulator that actually runs on the card, so I wouldn't consider it a proper BMC - as half-sized card's should've been connected by NCSI to centralized BMC somewhere else.
From what I've heard, big vendors like Dell, actually run control plane of the vmware on cards like that.
Bluefield-2 controller cards are separate line of NICs though (for Bluefield-2 and Bluefield-1, they were unified in Bluefield-3 generation though) and it is not clear what you would need to do to make PCIe Host to work on those cards.
I"m not gonna try and run that headless. I'm searching for a cheap compact host to run it in. If not then I'm going to use one of my old x299 systems to power it and offload storage. It was more of a thought that it might work headless. AND you also have to flash it as well. It's a lot of work but it's something I wanted to experiment. Plus it seems like a pretty good deal for this variant of bluefield-2.
Last edited:
It can, out of the box even - you just will need to flash it once and then ssh to it over the OOB RJ45 interface. But a client card won't be able to find any PCIe devices, only controller card can. I have one BF2 Controller but it has PCB damage (a bunch of decoupling capacitors on PCIe are knocked off and I think for some even the trace is damaged) that I haven't fixed and at least from the differences that I've noticed - it has clock gen chip and different configuration of some resistors, compared to equivalent client card.
I'd like to be proved wrong, but from my reading of the DOCA documentation, you need a bluefield-3 card to do nvme emulation, which makes this one less interesting to me.
Yea you're right about that. I did more research after. You need the bluefield-2 bf2500 controller card for that setup to work.
If you want Bluefield to act as a PCIe master. However the card that STH had reviewed was not a bf2500 controller card, you can see the PN at one of the images and it was a client card. So that is still a mistery to me.
https://docs.nvidia.com/networking/...snap-for-nvme-and-virtio-blk-v3-8-0-5.0-5.pdf - there was that, but it seems it required some sort of extra license to work.
That got replaced with what you've quoted and generic PCIe device initialization doesn't work on BF-2 or older.
I have mixed feelings about these DPU cards.
I do enjoy learning about the hardware offloading capabilities of modern network adapters, however these DPUs should be thought more as full blown computers that happen to be powered within another system and have direct access to its devices (storage, gpu, network).
If you just need hardware offloading for packet processing or hardware offloading for virtual machines (SR-IOV, switchdev, vDPA), a normal modern network card has you covered.
These DPU cards are meant to implement a control-plane at data center level and to be in charge of some functions that use to run in the host part of virtualization OS.
I don't if they make sense in a homelab. You get card that is 70W, needs specific cooling, needs to be handled with DOCA proprietary drivers (Which I don't enjoy), you need to use their custom OS on the card (What happens when nvidia deprecates it? ). Also, you need to understand that you have a network connection to the card who has a network connection to the rest of the network.
I would prefer getting a used non-dpu connectx6/6lx, intel e810 and put it in SFF system and use it.
I do enjoy learning about the hardware offloading capabilities of modern network adapters, however these DPUs should be thought more as full blown computers that happen to be powered within another system and have direct access to its devices (storage, gpu, network).
If you just need hardware offloading for packet processing or hardware offloading for virtual machines (SR-IOV, switchdev, vDPA), a normal modern network card has you covered.
These DPU cards are meant to implement a control-plane at data center level and to be in charge of some functions that use to run in the host part of virtualization OS.
I don't if they make sense in a homelab. You get card that is 70W, needs specific cooling, needs to be handled with DOCA proprietary drivers (Which I don't enjoy), you need to use their custom OS on the card (What happens when nvidia deprecates it? ). Also, you need to understand that you have a network connection to the card who has a network connection to the rest of the network.
I would prefer getting a used non-dpu connectx6/6lx, intel e810 and put it in SFF system and use it.
The common BF2s aren't PCIe hosts so they have the same access to your system as a NIC, and once you install an OS you can just run them in whatever sort of slot provides power, like a raspberry pi that trades GPIO pins for 25G ports.
The model that started this thread runs within the 25W regular PCIe power limit and has a pretty normal heatsink that probably just needs a smallish fan aimed at it. While I haven't gotten around to experimenting with my bf2 all my research says the DOCA stuff is just needed on the DPU, the host just needs the standard mlx5 drivers, not terribly restrictive.
You can build your own OS(GitHub - Mellanox/bfb-build: BFB (BlueField boot stream and OS installer) build environment), it's not particularly custom, they support a decent range of Linux distros: Index of /public/repo/doca/latest
Sorry I don't think this is true. I have a dual 25G connectx 6-lx (which is scaled down version of the regular connectx-6 dx) and that card alone pulls 20W. Do not be fooled by the skimpy mellanox heatsinks, they run hot as hell.
[here](https://networking-docs.nvidia.com/bluefield2hw/specifications), after the warning about heat and airflow, says that HHHL version (The one in this thread) should be plugged in a 75W motherboard slot not 25W like you said.
If you conservatively factor in 20W for for network card and 20W for the ARM SoC, this is at minimum 40W card.
I don't see where this piece of information comes from. I can't find it in nvidia docs. That would be odd since one of the major point is to directly access your nvme storage drives to answer requests coming from the network without involving the host OS/CPU.
Again, I can understand the use of these in datacenters. You can put one of these cards in each server box and have them do out-of-band management, some firewalling, some IDS, respond to NVMe-OF requests for storage or memory without involving the host system/OS at all. Leave the host system just to run the guest workloads.
However, this is a computer within a computer, and managing is non trivial. It's an interesting piece of hardware but I think in the context of a homelab is more a headache than a resource.
You're right, my memory was that only applied to the DPUs with more cores, I'm still pretty sure I found reports that these run fine in a 25W slot but now a quick search is just returning multiple copies of that spec page.
Given the right DMA configuration any NIC has access to your NVMe or GPU without the CPU involved, this is the same, it's just a PCIe device, not the version with a PCIe root complex.
It's definitely 75W. I was doing a bunch of research last night regarding this along with DOCA SDK. Here's the documentation: Specifications | NVIDIA BlueField-2 Ethernet DPU User Guide