The best idea I've had so far is something like SUPERMICRO SYS-5019C-L 1U Rackmount Server Barebone - Newegg.com but that's going to:
a) sound like a dental drill
b) not push any air over the card
I picked up a BF2 a while back and I'm looking to turn it into a router but I'm struggling to find a low power host device that will actually cool this thing. Anyone have any suggestions? I'm expecting most of the raw packet pushing to be done on the asic but wouldn't mind having a "performant...
Sometimes it takes looking at your config in your own post on a forum to see things. Conceptually I'd _expect_ mem_encrypt=on to break RDMA; but it never even crossed my mind till I was staring at it here. Disabling and testing; will report back.
No errors in dmseg that I can find. Kernel cmdline is as follows:
BOOT_IMAGE=/boot/vmlinuz-6.17.4-2-pve root=/dev/mapper/pve-root ro idle=nomwait quiet amd_iommu=on iommu=pt mem_encrypt=on irqpoll default_hugepagesz=1G hugepagesz=1G hugepages=16
# ibstat
ibwarn: [2583199] main: stat of IB device 'mlx5_0' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_1' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_2' failed
Running 2 CX4LXs in Proxmox and all my GIDs have disappeared. I'm at a loss, just tried installing the latest DOCA package and DKMS modules; firmware is up to date.
# show_gids
DEV PORT INDEX GID IPv4 VER DEV
--- ---- -----...
Yeah, that's the understanding I had. I've used the Unifi switching infra in the past and frankly I don't like the management stack Unifi provides in this space so I was hoping for some other option. This hardware had an interesting port distribution and I had an opportunity to pick one up...
I've been trying to get ovs offloading working in a separate thread as well; everything _looks_ correct but I never see ovs generate any offloaded flows for my SR-IOV attached guests; this is a simple switching setup with vlans out to the rest of the environment (cx4lx, proxmox).
I've had no problems getting ROCEv2 working in Windows or ksmbd either virtual or bare metal, what this thread is focusing on is the eswitch capabilities that provide tc offload and ASAP2 functionality in *nix. All that said get the number of ports you're going to need for your workload!
I agree that this is gonna require VyOS surgery; I'm wondering if they're dropping offload lookups based on the driver loaded, i.e. seeing mlx5 and not bothering to look for tc_offload.
I'm wondering if this is a vyos bug here; I've got a vyos instance with vifs on a cx4lx with the same behavior. I'm not using virtual functions as the newer drivers (including mainline) can enable the eswitch without sr-iov.
vyos@vyos:~$ sudo devlink dev eswitch show pci/0000:07:00.0...
Does anyone have any info on the difference in hardware acceleration present in the CX4LX vs the later cards? I have allegedly got this all configured correctly; guests are passing traffic over SRIOV interfaces ... but I'm not seeing any offloaded flows.
I'm quite curious what difference you're seeing in packet processing rate between software switching sr-iov and offloaded sr-iov; have you run trex or anything to test the limits?
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.