Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

[kapone]

Are you able to do RoCEv2 and still use sr-iov? I haven't fully tested it, but i've been toying with it and for some reason my cx3pro seems to default to RoCEv1 when I enable sr-iov so i shelved the idea. I haven't seen a way to set the default to RoCEv2(that actualy works...). I'm having to use a few workarounds as a result of not having sr-iov but its not the end of the world. I was still able to get my mirrored SAN up and running using drbd and linstor all using nvme/RoCEv2. I went a head and automated it's setup from a fresh install because, I would not wanna set this all up again....

ROCEv2 on...what? the PF? And the VFs? or just the VFs?

This particular test was not using SR-IOV, just two bare metal machines with the adapters in them. But...hang tight, I'm reworking on Starwind SAN and will know shortly if ROCEv2 works on VFs or not. It should...

 

[Schemer]

ROCEv2 on...what? the PF? And the VFs? or just the VFs?

This particular test was not using SR-IOV, just two bare metal machines with the adapters in them. But...hang tight, I'm reworking on Starwind SAN and will know shortly if ROCEv2 works on VFs or not. It should...

I was looking at RoCEv2 on the PF, ideally I would be able to use both. All of my hosts with cx3pros are baremetal Proxmox and for whatever reason when I created VFs the default for for RoCE would go to v1 instead of v2.

I would be excited to know your experience with Starwind, I played around with it a little bit, but ultimately stuck with drbd, because it was the only way I could think of to mirror SANs synchronously for failover with 0 downtime.

I'm also wondering if there is a decent way to share cx3 pro configs... because basic configs for the card are fine, but somethin on the OS side is not...

 

[Schemer]

Looks like my issues were stemming from my CX3 PFs and VFs all being in the same iommu group, so when once gets messed with it would mess with everything else in the iommu group. This was specific to the lenovo m720qs that I was running, and fortunately was fixable... Finally got SR-IOV working, so i can pass through a VF to a VM for shared storage for kubernetes. The bad news is.... now I have nothing to stop me from learning Kubernetes.