Search results

  1. G

    Bluefield 2 Host Suggestions

    The best idea I've had so far is something like SUPERMICRO SYS-5019C-L 1U Rackmount Server Barebone - Newegg.com but that's going to: a) sound like a dental drill b) not push any air over the card
  2. G

    Bluefield 2 Host Suggestions

    I picked up a BF2 a while back and I'm looking to turn it into a router but I'm struggling to find a low power host device that will actually cool this thing. Anyone have any suggestions? I'm expecting most of the raw packet pushing to be done on the asic but wouldn't mind having a "performant...
  3. G

    RDMA and CX4LX Troubleshooting

    Nope, same result ㄟ( ▔, ▔ )ㄏ
  4. G

    RDMA and CX4LX Troubleshooting

    Sometimes it takes looking at your config in your own post on a forum to see things. Conceptually I'd _expect_ mem_encrypt=on to break RDMA; but it never even crossed my mind till I was staring at it here. Disabling and testing; will report back.
  5. G

    RDMA and CX4LX Troubleshooting

    No messages at all in dmesg from DMAR, after a boot mlx5 looks fine...
  6. G

    RDMA and CX4LX Troubleshooting

    No errors in dmseg that I can find. Kernel cmdline is as follows: BOOT_IMAGE=/boot/vmlinuz-6.17.4-2-pve root=/dev/mapper/pve-root ro idle=nomwait quiet amd_iommu=on iommu=pt mem_encrypt=on irqpoll default_hugepagesz=1G hugepagesz=1G hugepages=16
  7. G

    RDMA and CX4LX Troubleshooting

    # ibstat ibwarn: [2583199] main: stat of IB device 'mlx5_0' failed ibwarn: [2583199] main: stat of IB device 'mlx5_1' failed ibwarn: [2583199] main: stat of IB device 'mlx5_2' failed
  8. G

    RDMA and CX4LX Troubleshooting

    I'm at a loss. # lsmod | grep 'mlx5_ib\|uverb' mlx5_ib 540672 0 ib_uverbs 200704 2 rdma_ucm,mlx5_ib macsec 69632 1 mlx5_ib ib_core 507904 9 rdma_cm,ib_ipoib,iw_cm,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm mlx5_core...
  9. G

    RDMA and CX4LX Troubleshooting

    for i in 0 1 > do > mstflint -d mlx5_${i} q > done Image type: FS3 FW Version: 14.32.1908 FW Release Date: 17.8.2025 Product Version: 14.32.1908 Rom Info: type=UEFI version=14.25.17 cpu=AMD64,AARCH64 type=PXE version=3.6.502...
  10. G

    RDMA and CX4LX Troubleshooting

    Running 2 CX4LXs in Proxmox and all my GIDs have disappeared. I'm at a loss, just tried installing the latest DOCA package and DKMS modules; firmware is up to date. # show_gids DEV PORT INDEX GID IPv4 VER DEV --- ---- -----...
  11. G

    Ubiquiti Enterprise XG24

    Have you seen Asahi Linux - Wikipedia ?
  12. G

    Ubiquiti Enterprise XG24

    Yeah, that's the understanding I had. I've used the Unifi switching infra in the past and frankly I don't like the management stack Unifi provides in this space so I was hoping for some other option. This hardware had an interesting port distribution and I had an opportunity to pick one up...
  13. G

    Ubiquiti Enterprise XG24

    Has anyone successfully flashed a different OS on one of these? Does anyone know what ASIC is in use here?
  14. G

    nftables flowtables hardware offload with ConnectX-5

    I've been trying to get ovs offloading working in a separate thread as well; everything _looks_ correct but I never see ovs generate any offloaded flows for my SR-IOV attached guests; this is a simple switching setup with vlans out to the rest of the environment (cx4lx, proxmox).
  15. G

    Fully utilizing the ConnectX-5 eSwitch/switchdev functionality in Proxmox VE

    I've had no problems getting ROCEv2 working in Windows or ksmbd either virtual or bare metal, what this thread is focusing on is the eswitch capabilities that provide tc offload and ASAP2 functionality in *nix. All that said get the number of ports you're going to need for your workload!
  16. G

    nftables flowtables hardware offload with ConnectX-5

    I agree that this is gonna require VyOS surgery; I'm wondering if they're dropping offload lookups based on the driver loaded, i.e. seeing mlx5 and not bothering to look for tc_offload.
  17. G

    nftables flowtables hardware offload with ConnectX-5

    I'm wondering if this is a vyos bug here; I've got a vyos instance with vifs on a cx4lx with the same behavior. I'm not using virtual functions as the newer drivers (including mainline) can enable the eswitch without sr-iov. vyos@vyos:~$ sudo devlink dev eswitch show pci/0000:07:00.0...
  18. G

    Fully utilizing the ConnectX-5 eSwitch/switchdev functionality in Proxmox VE

    Does anyone have any info on the difference in hardware acceleration present in the CX4LX vs the later cards? I have allegedly got this all configured correctly; guests are passing traffic over SRIOV interfaces ... but I'm not seeing any offloaded flows.
  19. G

    Fully utilizing the ConnectX-5 eSwitch/switchdev functionality in Proxmox VE

    I'm quite curious what difference you're seeing in packet processing rate between software switching sr-iov and offloaded sr-iov; have you run trex or anything to test the limits?
  20. G

    EXPIRED HP T740 Thin Client Ryzen V1756 8GB - $53 - US (Used)

    That's helpful, doesn't look like I'd be able to use this for OS installation which was my main goal.