1. J

    RoCE use cases

    Hi Folks, Anyone come across or try out a killer use case for RDMA / RoCE? I know there are custom builds of some hadoop based apps but what have you found to be a great use of this tech? I have the box, now need the project. :) James
  2. mattaw

    Help with Connectx-3 back-to-back using FS QSFP-BD-40G

    Hey all, I used the instructions to cross flash two HP Connextx-3 cards QSFP to Mellanox MCX354A-FCBT's. I then installed two Fiberstore (FS) QSFP-BD-40G modules into one card configured as dual Ethernet. No dice, the link won't come up and the lights on the card stay off. Things I have...
  3. P

    Mellanox-Connectx2 Setup

    I recently got two MNPA19-XTR Mellanox-ConnectX2 cards and installed them into my servers. I followed another thread on here and they are both running fw=2.10.0720, the vstat is here: hca_idx=0 uplink={BUS=PCI_E Gen2, SPEED=5.0 Gbps, WIDTH=x8, CAPS=5.0*x8}...
  4. T

    FS - Mellanox IS5600 648-port 40Gb/s InfiniBand Chassis Switch

    I picked up a Mellanox IS5600 and do not have any personal use for it. Total Newb when talking about commercial-ish switches. This unit taken out of a working environment. Retailed for $500,000 retail LOL. Seriously. Half a milli! Listed for $3,000 USD plus shipping. Paypal or local cash...
  5. E

    DAC compatible with Cisco P81E VIC and Ubiquiti US-16-XG

    I ordered some SFP-H10GB-CU3M compatible (supposedly) DACs from, and although they're nice and thin, and work with my Mellanox Connectx-3 cards in my NAS', they are not compatible with my P81E 10GbE cards in my UCS servers. Can anyone recommend a good DAC in 3-5m distances to connect my...
  6. M

    10gbit interface compatibility (Intel X553 <-> Mellanox ConnectX-2)

    I've been working on a Supermicro SYS-E300-9A (Intel Atom C3858,) putting together a fully virtualized router + whatever, and I'm getting around to being complete. The last step for me is routing my existing 10gbit setup through it, but I've hit a snag. My NAS has a Mellanox ConnectX-2 card...
  7. MikeWebb

    Faster then 10Gb home, home office and lab recommendations (IB, VPI etc?)

    Hi there all, I've just finished setting up a nice AIO proxmox server and between it and my workstation (direct connection), it saturates 10Gb or gives a faster then local SSD speed depending on work load (NFS maxes out at 4GB/s but thats a different thread)......and now I want more. Well not...
  8. G

    Mellanox ConnectX / HP MT26448 - Multiple PFs not yet supported

    I got a bunch of these Mellanox ConnectX HP branded MT26448 cards on eBay for use in a Proxmox cluster, and mostly they're fine. However, once every 3 - 5 reboots, they don't come up and we see the this in the logs: I've found a few posts about this issue in various places, but none with a...
  9. U

    Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

    Hello! I have the opportunity to purchase some used dual-port Mellanox MCX354A-QCBT ConnectX-3 cards with VPI network adapter. (10 GbE Ethernet and 40 Gbps InfiniBand.) These would be perfect for my cluster after correct configuration.. They were originally pulled from EMC systems and...
  10. A

    QDR Mellanox Performance Test Results with Varying BAR-space Size

    System: Desktop (headnode): I7-5960X, GA-X99-SLI motherboard Server (slave nodes): 6027TR-HTR (four X9DRT-HF). Only using one node for this testing. 2x E5-2667 V2 OS: CentOS 7.5 with Infiniband Support and various other packages. No custom OFED installs. HCAs: One Sun/Oracle X4242A QDR...
  11. A

    Mellanox Connectx-2 temperature (and alternatives?)

    My 2-port Mellanox ConnectX-2 NIC is running at a scary 72C. I have got a Norco 4u case with 6 Noctua 80mm fans. They are running at adaptive speeds. Nothing blowing air over the PCI cards, so that won't help. I might build a bracket with magnets to hold some fans (maybe two 80mm) and one of...
  12. K

    Question about Mellanox Connectx-2 VPI and Cables

    Good Afternoon everyone. You will have to excuse my ignorance when it comes to infiniband/ SFP/ SFP+ and QSFP. I have purchased two Mellanox MHQH29C-XSR ConnectX-2 VPI 10/40 Gbit Low Profile Dual-Port Adapter Cards for use in my home network (purely for learning purposes and for transferring...
  13. L

    HP 649281-B21 ConnectX-3 VPI MCX354A-FCBT $35 + $8 shipping OBO ebay

    Unbelievable deal! These are low profile dual port cards capable of 56Gbps IB and 40Gbps Ethernet. I messaged the seller and bought 32 cards for $25 each plus shipping. He said he has 1300 in stock so there's plenty for everyone :) They can be flashed from their HP firmware to stock Mellanox...
  14. A

    Mellanox IS5030 Speed Issues: Only DDR?

    Hi, I'm having speed issues that I've isolated to my IS5030 switch. System: I have two Infiniband switches: 1. A Mellanox IS50XX with 36 ports enabled and the FrabricIT internal subnet manager running, making it an IS5030, with latest firmware (IBM P/N: 98Y3756) 2. A Sun 36 port QDR Infiniband...
  15. A

    Compatible active FS SFP+ DAC for Mellanox Connectx-2/3 and possibly TP-Link switch

    Hi! I need a 10 meter active SFP+ DAC to connect my workstation peer to peer to a server with Mellanox Connectx-2 cards. I might get a TP Link SFP+ enabled switch in the future... so compatibility would be nice to have. Since the cables run for 81 USD new, which ones should be the most...
  16. L

    Mellanox ConnectX-3 Pro MCX354A-FCCT FDR10 40/56GbE 2 PORT QSFP ADAPTER $118 obo

    I missed the previous $80 CX-4 deal, but I found this MCX354A-FCCT CX354A MELLANOX CONNECT-3 4099 FDR10 40/56GbE 2 PORT QSFP ADAPTER | eBay Different from MCX354A-FBCT, the above one is MCX354A-FCCT (ConnectX-3 Pro)
  17. H

    Potential deal: MELLANOX CONNECTX-2 MNPA19-XTR 10GB with SFP+ CABLE for $23

    MNPA19 XTR 10GB MELLANOX CONNECTX 2 PCIe X8 10Gbe SFP NETWORK CARD W CABLE | eBay The bundled cable makes it an interesting deal. However, no lengths provided. Seems to be over 1 meters, but wouldn't be surprised if shipped with the shortest on market. Edit: Seems like he just bumbed the price...
  18. M

    Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

    Hello, I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35 In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be...
  19. MiniKnight

    Mellanox ConnectX-4 40GbE $80

    Sick! MELLANOX TECHNOLOGIES, INC. MCX4131A-BCAT CONNECTX-4 LX EN NETWORK INTERFACE | eBay ConnectX-4 QSFP28 cards. The picture is of a dual port model but the model number is the single port model. Spec sheet
  20. T

    DAC Question: Mellanox MCX314A-BCBT <-> Arista 7050QX-32

    Searched through the forums and can't find a definitive answer regarding a DAC for this setup. Hoping someone has first hand experience. If a gun was to my head I would buy this: Arista Networks CAB-Q-Q-1M 40G QSFP+ DAC Cable | FS.COM Just looking to make sure I get something that works...