1. AveryFreeman

    Anyone using IB over ConnectX-3 w/ ESXi 7.0U1?

    Hey, I've never used an IB adapter before, but I was thinking of doing some direct-attached IB adapters for my 3 ESXi hosts now that I learned my 3 Intel 82598EB adapters I dug out of storage are now no longer supported. I do have VCSA if that matters. I have 82599 10GbE for basic networking...
  2. BoredSysadmin

    RDMA(RoCE v2) with Nutanix 7-8 nodes Cluster

    Hello, I am seeking advice or consensus if we should try to engineer our Nutanix cluster with RDMA. It stands now the nodes are fairly beefy: Each node is 2x 6226R and 1.5TB ram each,12x3.84TB SAS12 SSDs (no NVME drives). The load is a mixed server load, various biz apps, SQL databases, some...
  3. A

    RoCE v2/DCB used switches

    Hi. I want to try RoCE2 in my homelab to get better performance on my glusterfs vm storage. Are there any DCB capable switches under $500 used? 10gb ports is fine. (Not sure if DCB is required for RoCE2, actually. but hey, thats why i need smth to experiment on)
  4. I

    Help with understanding 40Gbe/56GbIB cabling

    Hi all, n00by here... but this place has a wonderful amount of knowledge. I asked a similar question in the Mellanox forums here: https://community.mellanox.com/s/question/0D51T00007CGDzUSAX/which-cables-for-a-pair-of-mcx353afcbt I have two MCX353A-FCBT cards on the way and I want to connect...
  5. A

    RDMA in ESXi 6.7 on ConnectX-2 Cards

    We have a series of 13 Dell R610 hosts with ConnectX-2 EN cards running ESXi 6.7. We have been trying to get RDMA (more importantly the iSER adapter) to enable and show up, however we have been unsuccessful. RDMA is supported by the FW, as we had gotten it to show up before with different...
  6. J

    RoCE use cases

    Hi Folks, Anyone come across or try out a killer use case for RDMA / RoCE? I know there are custom builds of some hadoop based apps but what have you found to be a great use of this tech? I have the box, now need the project. :) James
  7. EluRex

    Ceph blustore over RDMA performance gain

    I want to share following testing with you 4 PVE Nodes cluster with 3 Ceph Bluestore Node, total of 36 OSD. OSD: st6000nm0034 block.db & block.wal device: Samsung sm961 512GB NIC: Mellanox Connectx3 VPI dual port 40 Gbps Switch: Mellanox sx6036T Network: IPoIB separated public network &...
  8. D

    TOR Switch Needed for 3-Node Hyper-Converged S2D 2016 Cluster

    Using Windows Server 2016DC, I'm building a 3-node Hyper-converged, Storage Spaces Direct failover cluster. It seems to work good and passes validation using an ordinary D-Link 10GB switch except the fact that I can't see any RDMA traffic over the switch. I confirmed that I can see RDMA activity...
  9. M

    Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

    Hello, I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35 In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be...
  10. AveryFreeman

    Kernel-level SMB3 for Linux?

    Hi, kind of an odd question - I know SMB3 is being touted as a VM datastore-worthy transfer protocol on Windows Server platforms with RDMA / SR-IOV capabilities, etc. but is the same thing available for Linux? Do I understand it in that 'Samba' for Linux is a userland-only protocol and...
  11. N

    iSCSI SAN With Windows Server 2016

    Hi Everyone, hope everything is going well on your side of the fence. I am new here and in need of help getting RDMA to work on my little storage/SAN project on Windows Server 2016. I bought 2 of these Mellanox MHQH29C-XTR cards on Ebay I have updated the firmware to 2.9.8350 My client is...