rdma

  1. S

    1/10G Ethernet Network for 3-Node GPU Cluster

    Hello, all. I have been given 3 servers to start learning how to stand up a GPU cluster. I have a pretty solid background on the intra-node architecture, but very little understanding of the networking that's needed for this, so please forgive my ignorance in this area : ) I plan to use one...
  2. G

    RDMA (Mikrotik + Connect-X)

    Hello folks! With the recent updates on Mikrotik RouterOS 7.15 they’ve introduced support to PFC and ECNwhich are vital to enable RDMA with RoCE. I have a setup of 3x servers with a pair of Dual 25GbE Connect-X 4 each (which supports RDMA with RoCE v2). I’d like to use two ports (one on each...
  3. A

    Recommendations for ESX7-TrueNAS infiniband RDMA?

    Now that I’ve built a pair of fully-solid state ISCSI targets using TrueNAS core I’d like to upgrade my network to provide the lowest latency from these boxes to my 3-node ESX 7.03 cluster and I’m looking for recommendations. PCI slots are at a premium on my boxes so I’m thinking a dual-port...
  4. A

    Seeking guidance on modifying RoCEv2 code for continuous reception of UDP data packets from an FPGA over Ethernet

    I’m currently working on receiving UDP data packets over Ethernet using RoCEv2, but I’ve encountered a challenge in my code. I’m trying to continuously receive data packets emitted by an FPGA, utilizing RoCEv2. I attempted to modify the code for multicast given in following document ,to unicast...
  5. U

    RoCE v1 implementation (SX6036 heatsink/silence mod running log!)

    For my setup, I already have (4) CX3 non-pro cards. I needed to get a single CX4 card in order to support the mlx5 driver that is now bundled with macOS and my macs can be on the network now via thunderbolt. I am interested in setting up a simple point to point network at home where the...
  6. N

    RNIC functions on one host aren't working. I know it's a config problem, but where to look?

    Edit: Everything below is... well, it's not wrong, except RDMA is actually not working for ANY hosts. Everything is going through IPoIB and it's awful. I thought it was working for other hosts but it is not. Original post: This host ("shark") currently has four RDMA-aware functions, and none...
  7. L

    iSCSI ZFS block storage options for ESXi with RDMA / RoCE / iSER support

    I'm currently using FreeNAS for iSCSI ZFS block storage for my ESXi server. The FreeNAS box is getting old so I'm looking for a replacement. I'm not currently using any of the other FreeNAS features besides ZFS and iSCSI. I was initially planning to upgrade the FreeNAS box to TrueNAS Core or...
  8. AveryFreeman

    Anyone using IB over ConnectX-3 w/ ESXi 7.0U1?

    Hey, I've never used an IB adapter before, but I was thinking of doing some direct-attached IB adapters for my 3 ESXi hosts now that I learned my 3 Intel 82598EB adapters I dug out of storage are now no longer supported. I do have VCSA if that matters. I have 82599 10GbE for basic networking...
  9. BoredSysadmin

    RDMA(RoCE v2) with Nutanix 7-8 nodes Cluster

    Hello, I am seeking advice or consensus if we should try to engineer our Nutanix cluster with RDMA. It stands now the nodes are fairly beefy: Each node is 2x 6226R and 1.5TB ram each,12x3.84TB SAS12 SSDs (no NVME drives). The load is a mixed server load, various biz apps, SQL databases, some...
  10. A

    RoCE v2/DCB used switches

    Hi. I want to try RoCE2 in my homelab to get better performance on my glusterfs vm storage. Are there any DCB capable switches under $500 used? 10gb ports is fine. (Not sure if DCB is required for RoCE2, actually. but hey, thats why i need smth to experiment on)
  11. I

    Help with understanding 40Gbe/56GbIB cabling

    Hi all, n00by here... but this place has a wonderful amount of knowledge. I asked a similar question in the Mellanox forums here: https://community.mellanox.com/s/question/0D51T00007CGDzUSAX/which-cables-for-a-pair-of-mcx353afcbt I have two MCX353A-FCBT cards on the way and I want to connect...
  12. A

    RDMA in ESXi 6.7 on ConnectX-2 Cards

    We have a series of 13 Dell R610 hosts with ConnectX-2 EN cards running ESXi 6.7. We have been trying to get RDMA (more importantly the iSER adapter) to enable and show up, however we have been unsuccessful. RDMA is supported by the FW, as we had gotten it to show up before with different...
  13. J

    RoCE use cases

    Hi Folks, Anyone come across or try out a killer use case for RDMA / RoCE? I know there are custom builds of some hadoop based apps but what have you found to be a great use of this tech? I have the box, now need the project. :) James
  14. EluRex

    Ceph blustore over RDMA performance gain

    I want to share following testing with you 4 PVE Nodes cluster with 3 Ceph Bluestore Node, total of 36 OSD. OSD: st6000nm0034 block.db & block.wal device: Samsung sm961 512GB NIC: Mellanox Connectx3 VPI dual port 40 Gbps Switch: Mellanox sx6036T Network: IPoIB separated public network &...
  15. D

    TOR Switch Needed for 3-Node Hyper-Converged S2D 2016 Cluster

    Using Windows Server 2016DC, I'm building a 3-node Hyper-converged, Storage Spaces Direct failover cluster. It seems to work good and passes validation using an ordinary D-Link 10GB switch except the fact that I can't see any RDMA traffic over the switch. I confirmed that I can see RDMA activity...
  16. M

    Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

    Hello, I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35 In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be...
  17. AveryFreeman

    Kernel-level SMB3 for Linux?

    Hi, kind of an odd question - I know SMB3 is being touted as a VM datastore-worthy transfer protocol on Windows Server platforms with RDMA / SR-IOV capabilities, etc. but is the same thing available for Linux? Do I understand it in that 'Samba' for Linux is a userland-only protocol and...
  18. N

    iSCSI SAN With Windows Server 2016

    Hi Everyone, hope everything is going well on your side of the fence. I am new here and in need of help getting RDMA to work on my little storage/SAN project on Windows Server 2016. I bought 2 of these Mellanox MHQH29C-XTR cards on Ebay I have updated the firmware to 2.9.8350 My client is...