rdma

Hello, all. I have been given 3 servers to start learning how to stand up a GPU cluster. I have a pretty solid background on the intra-node architecture, but very little understanding of the networking that's needed for this, so please forgive my ignorance in this area : ) I plan to use one...

Hello folks! With the recent updates on Mikrotik RouterOS 7.15 they’ve introduced support to PFC and ECNwhich are vital to enable RDMA with RoCE. I have a setup of 3x servers with a pair of Dual 25GbE Connect-X 4 each (which supports RDMA with RoCE v2). I’d like to use two ports (one on each...

Now that I’ve built a pair of fully-solid state ISCSI targets using TrueNAS core I’d like to upgrade my network to provide the lowest latency from these boxes to my 3-node ESX 7.03 cluster and I’m looking for recommendations. PCI slots are at a premium on my boxes so I’m thinking a dual-port...

I’m currently working on receiving UDP data packets over Ethernet using RoCEv2, but I’ve encountered a challenge in my code. I’m trying to continuously receive data packets emitted by an FPGA, utilizing RoCEv2. I attempted to modify the code for multicast given in following document ,to unicast...

For my setup, I already have (4) CX3 non-pro cards. I needed to get a single CX4 card in order to support the mlx5 driver that is now bundled with macOS and my macs can be on the network now via thunderbolt. I am interested in setting up a simple point to point network at home where the...

Edit: Everything below is... well, it's not wrong, except RDMA is actually not working for ANY hosts. Everything is going through IPoIB and it's awful. I thought it was working for other hosts but it is not. Original post: This host ("shark") currently has four RDMA-aware functions, and none...

I'm currently using FreeNAS for iSCSI ZFS block storage for my ESXi server. The FreeNAS box is getting old so I'm looking for a replacement. I'm not currently using any of the other FreeNAS features besides ZFS and iSCSI. I was initially planning to upgrade the FreeNAS box to TrueNAS Core or...

Hey, I've never used an IB adapter before, but I was thinking of doing some direct-attached IB adapters for my 3 ESXi hosts now that I learned my 3 Intel 82598EB adapters I dug out of storage are now no longer supported. I do have VCSA if that matters. I have 82599 10GbE for basic networking...

Hello, I am seeking advice or consensus if we should try to engineer our Nutanix cluster with RDMA. It stands now the nodes are fairly beefy: Each node is 2x 6226R and 1.5TB ram each,12x3.84TB SAS12 SSDs (no NVME drives). The load is a mixed server load, various biz apps, SQL databases, some...

Hi. I want to try RoCE2 in my homelab to get better performance on my glusterfs vm storage. Are there any DCB capable switches under $500 used? 10gb ports is fine. (Not sure if DCB is required for RoCE2, actually. but hey, thats why i need smth to experiment on)

Hi all, n00by here... but this place has a wonderful amount of knowledge. I asked a similar question in the Mellanox forums here: https://community.mellanox.com/s/question/0D51T00007CGDzUSAX/which-cables-for-a-pair-of-mcx353afcbt I have two MCX353A-FCBT cards on the way and I want to connect...

We have a series of 13 Dell R610 hosts with ConnectX-2 EN cards running ESXi 6.7. We have been trying to get RDMA (more importantly the iSER adapter) to enable and show up, however we have been unsuccessful. RDMA is supported by the FW, as we had gotten it to show up before with different...

Hi Folks, Anyone come across or try out a killer use case for RDMA / RoCE? I know there are custom builds of some hadoop based apps but what have you found to be a great use of this tech? I have the box, now need the project. :) James

I want to share following testing with you 4 PVE Nodes cluster with 3 Ceph Bluestore Node, total of 36 OSD. OSD: st6000nm0034 block.db & block.wal device: Samsung sm961 512GB NIC: Mellanox Connectx3 VPI dual port 40 Gbps Switch: Mellanox sx6036T Network: IPoIB separated public network &...

Using Windows Server 2016DC, I'm building a 3-node Hyper-converged, Storage Spaces Direct failover cluster. It seems to work good and passes validation using an ordinary D-Link 10GB switch except the fact that I can't see any RDMA traffic over the switch. I confirmed that I can see RDMA activity...

Hello, I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35 In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be...

Hi, kind of an odd question - I know SMB3 is being touted as a VM datastore-worthy transfer protocol on Windows Server platforms with RDMA / SR-IOV capabilities, etc. but is the same thing available for Linux? Do I understand it in that 'Samba' for Linux is a userland-only protocol and...

Hi Everyone, hope everything is going well on your side of the fence. I am new here and in need of help getting RDMA to work on my little storage/SAN project on Windows Server 2016. I bought 2 of these Mellanox MHQH29C-XTR cards on Ebay I have updated the firmware to 2.9.8350 My client is...