@firemeteor I am using stock drivers with EMC CX3 rev A3+A4 cards on kernel 5.4 with okrasit's patches. Looked into compiling OFED vendor drivers once, but holy moly what a mess. Since I don't want another hell that is RDMA, which for me has no advantages, I cancelled all plans to make an Archlinux package and just went with the stock driver.
I would rather avoid the vendor driver mess if the stock one worked for me...
I wasn't looking for any advanced feature from vendor driver either, just for bug fixes.
I know very little about RDMA, but if it requires any client-side cooperation it won't be helpful to me...
Thankfully my driver building efforts had been paid-off and I was able to get the 4.9 LTS driver running on Debian 11.2 (Kernel 5.10).
I had to fix some configuration issue and some code compatibility issue though.
From reading your post my own feeling is SR-IOV is already thin ice, with every device. Putting a bridge on top looks against the KISS principle to me. I like the better trampled paths and when there are only two hits in Google for SRIOV FDB linux bridge mellanox you know the roadrunner has moved over the edge, waiting for gravity to assert itself. If you can simplify the concept, I would.
I think we should blame the poor quality of the stock driver, and those people behind the decision of such version separation.
As an old, mature (I suppose?) and well-defined feature, SRIOV should 'just works' as long as the driver stack if solid.
The concept of SRIOV should be clear and from implementation perspective it just comes with an embedded bridge for traffic routing purpose.
Attaching one bridge to another really shouldn't have caused too much trouble....
This should be true at least on the PF side of the NIC, as it should be plug-n-play and configuration-free.
The FDB management is for a more advanced feature related to security concern on VF side promiscuous mode.
The same security model should apply to other SRIOV NIC vendor like Intel too.
I don't really need this and just to have a try -- maybe I run into another driver bug, or maybe I used wrong configuration...
X520-DA2 also ok, only 10 Gbps of course. Very mature Intel product. CX3 of course of interest to people with FDR cable to run 56 Gbps for low prices.
Glad that I can stick to my current CX3 card without seeking for Intel replacement.
I didn't realize that the CX3 is capable of doing 56Gbps. Is this specific to some of the models only?
What I have should be the low-end CX311/312 cards. I didn't see 56Gbps mentioned anyway in the spec.
I also have no idea what a FDR cable really means since I'm a totally freshman to SFP+ NICs.
What I have is a Finisar AOC cable, probably not qualify that 56Gbps either.
But anyway, it's not what I was shooting at and other part of my system will probably won't able to keep up the pace.
www.mellanox.com
