infiniband

  1. S

    FDR compatibility check please

    Hi, As a complete noob for Infiniband i want to build a cluster from 4x Dell C6420 nodes each having one 56Gbps (Dual Port) QSFP HCA - Mellanox MCX354A-FCBT and i would like to connect them with one Nvidia Mellanox SX6025 switch. Now i am looking for cables. I have found the HP 670759-B23...
  2. A

    Recommendations for ESX7-TrueNAS infiniband RDMA?

    Now that I’ve built a pair of fully-solid state ISCSI targets using TrueNAS core I’d like to upgrade my network to provide the lowest latency from these boxes to my 3-node ESX 7.03 cluster and I’m looking for recommendations. PCI slots are at a premium on my boxes so I’m thinking a dual-port...
  3. C

    Switching Mellanox SX6036 from InfiniBand to Ethernet Mode

    have a Mellanox SX6036 switch that's currently set up in InfiniBand mode. I'm interested in switching it to Ethernet mode for our network needs. Has anyone gone through this process before? I am going all over the Internet, and I cannot find it. Thanks!!!
  4. N

    RNIC functions on one host aren't working. I know it's a config problem, but where to look?

    Edit: Everything below is... well, it's not wrong, except RDMA is actually not working for ANY hosts. Everything is going through IPoIB and it's awful. I thought it was working for other hosts but it is not. Original post: This host ("shark") currently has four RDMA-aware functions, and none...
  5. H

    ConnectX-3: Unable to get Ethernet mode working

    Hello everyone, Recently I purchased a pair of ConnectX-3 354A FCBT Oracle branded dual port NICs. The original firmware indicated it was an fcbt model. I then proceeded to cross flash it to original mellanox firmware. I received a QSFP+ cable yesterday, ordered from ebay: - HP Mellanox...
  6. B

    Mellanox IB SB7800 Drive Failure

    Good morning, We've had an SSD on our sb7800 infiniband switch fail, have installed a new drive with a cloned image from one of our other identical switches. So far everything looks ok in the management interface, aside the Host ID. Though we do get 1003, 1006 errors when trying to modify...
  7. M

    Best 300TB storage for $100K challenge

    Hi, I work for a research hospital. I would appreciate any comments or advice for file system and hardware design for a high-end storage solution. We have a few microscopes (running windows 10), each can produce 3-4GB/s of image data that needs to written to the shared storage and to be...
  8. S

    Understanding Infiniband IPOIB vs ETH mode

    I looking for a better understanding on the difference between IPOIB and ETH mode on ConnectX-3 adapters (40/40 IB/Eth) Couldn't find answers to my questions online; if someone could point me towards the right direction, it would be much appreciated. 1. My understanding is the IPOIB runs on top...
  9. L

    Infiniband Network for small GPU-Cluster

    Hi everyone, I am currently in the process of planning an Infiniband Network for our new GPU cluster. We want to be able to run simulation / AI workloads across the big GPU nodes and are hence looking to connect everything with a 100 Gb Infiniband-Network. At first, I was thinking of the...
  10. M

    Looking for HP z840 workstation users with Infiniband network cards

    Hi all, I have a couple z840 workstations that I'd like to connect with Mellanox ConnectX-3 VPI 40Gb network cards. These are PCIe gen3 cards that the mobo on the z840 supports (I have tested the PCI slots with two graphics cards and the machines boot fine). When I install the network card...
  11. Sniper_X

    Stumped by 10Gb speed limit on 40/56gb IB cards...

    I'm using a pair of dual port Connectx-3 cards (Part# MCX354A-FCBT on FW 2.42.5000 ) and I seem to be stuck at 10Gb. My ultimate goal is to get to pure Infiniband connectivity between the servers and the storage. I have Dell PowerConnect 8132(N4032) switches AND a pair of Mellanox IS5022...
  12. AveryFreeman

    Anyone using IB over ConnectX-3 w/ ESXi 7.0U1?

    Hey, I've never used an IB adapter before, but I was thinking of doing some direct-attached IB adapters for my 3 ESXi hosts now that I learned my 3 Intel 82598EB adapters I dug out of storage are now no longer supported. I do have VCSA if that matters. I have 82599 10GbE for basic networking...
  13. lihp

    Mellanox IS5022 InfiniBand Switch 8 ports managed - QDR

    No, not my offer. Since I am on EDR at home by now, I don't need it... Mellanox IS5022 InfiniBand Switch 8 ports managed | eBay
  14. G

    Mellanox/EMC SX6012 - Revival Attempt

    New project. I received an SX6012 for shipping costs only. Tried to get it running and to get a console today. First of all, the switch starts. Both fans in the middle are running 100%. Both fans in front of the power supplies are running on minimum rpm. Mgmt port shows straight green and...
  15. V

    EU WTB: Mellanox SX6036 FDR Infiniband Switch

    Hello again! Probably should have consolidated the FS and WTB ads in one post but for clarity I figured I'd post twice. If you're here from my FS post, I am looking for the above equipment. Does anyone have one for sale? Please let me know. Happy new year!
  16. V

    EU FS: Mellanox Infiniscale IV IS5030 18-port DDR switch

    Hello boys and girls, I have here for sale the above item. I want to move up to an FDR switch as part of upgrading my home lab, so if anyone wants to get started, play around and learn about Infiniband networking with a cheap switch (DAC cables go for pretty cheap on eBay as well), this might...
  17. G

    EU WTB: Mellanox 6012/6015/6036 (temporary done)

    It's Christmas ... Looking for cheap un-/managed 6012/6015's here in EU only. If you can include cables you were working with, SUPER !!! Ok, if you pay the shipping 'Overseas' are welcome as well ;) If you have a 6036 for cheap, I will consider your offer as well.
  18. G

    IB - CPU Frequency Mismatches - Sun Falcon (Mellanox MHRH2A-XSR)

    I have a little issue while testing InfiniBand connections. While system (Bios-wise) is set to performance and the CPU governance is set to performance, I still receive the message/alert/warning of CPU frequency mismatches. Both CPU's are E5-2673v3's. The cards are old 10EUR eBay findings of SUN...
  19. A

    Question about best hardware components for cluster computing

    Dear Friends I want to hook up around 10 individual powerful PCs and create a cluster out of them. My main use for this cluster is running CFD models which can run through MPI over Ethernet of Infiniband networks. As far as I know, Infiniband is superior for this purpose, but I dont have any...
  20. XeonSam

    Advice on vSAN networking : Infiniband

    I'm setting up a vSAN network with 3 Nodes. I have set up a vSAN cluster with 10G Ethernet (base-T) 2 years ago and wanted to delve into infiniband for a new test environment. I've done some reading up on this but wanted to get some tips from people who have successfully implemented a similar...