Search results

Looks like my issues were stemming from my CX3 PFs and VFs all being in the same iommu group, so when once gets messed with it would mess with everything else in the iommu group. This was specific to the lenovo m720qs that I was running, and fortunately was fixable... Finally got SR-IOV working...

I was looking at RoCEv2 on the PF, ideally I would be able to use both. All of my hosts with cx3pros are baremetal Proxmox and for whatever reason when I created VFs the default for for RoCE would go to v1 instead of v2. I would be excited to know your experience with Starwind, I played around...

Are you able to do RoCEv2 and still use sr-iov? I haven't fully tested it, but i've been toying with it and for some reason my cx3pro seems to default to RoCEv1 when I enable sr-iov so i shelved the idea. I haven't seen a way to set the default to RoCEv2(that actualy works...). I'm having to use...

Alright so it looks like I finally had some luck after like two days.... i migrated this vm from my pve1 to another host and it looks likes I was able to get it to work on my pve2 and pve3 node. I believe my cx3 pro might be setup slightly differently than my other hosts but im going to have to...

It seems to be all traffic, i did a nc -vz 192.168.20.50 8006 which worked from all my physical hosts on that vlan, but this didn't work from my vm. I even tried with all firewalls disabled without much luck. It's super weird, I can see the hosts sending the return traffic but it just disappears...

Nice! I'm hoping to see if anyone has any example configs or just general tips of how they setup their vlans. I'm still not entirely familiar with vlans yet, and it's been hard for me to find the origin of this issue. As far as I can tell all my proxmox hosts with physical nics have been...

Have any of yall with a sx6036 been able to successfully have a vm guest inside of proxmox properly communicate to other physical hosts on a vlan? I've been having an issue where my vm guest can ping other hosts on the vlan, and the other hosts reply properly, but the reply never even reaches...

how big is your lab? Thats a lot of cables/nics

Shrug, you could very well be right, im not an expert on it at all honestly so you judgment is probably better, I referenced the below thread and fixed it in case it was a potential issue. At the very least the temps are not a concern which i suppose is the main worry. Edit: ahh we might be on...

I heavily altered an existing 3d model for a noctua fan in order to increase the length of the noctua fan because what I had heard is that the length of the fan is a big difference maker for static pressure. My temps are good and I helped one other person do the same mod as well, but entirely up...

I set the fan speed to 15% which is the lowest that I could set it before it would spinup due to high heat and it was not low enough for me, My rack isn't that far from my desk and there weren't many convenient places for me to put it in my apartment. Your mileage may vary, but my fans were'nt...

Just be wary that the mellanox sx6036 switch is really loud, i bought mine for either 90 or 100 dollars but i ended up spending another 90 dollars to replace its fans with Noctua NF-A4x20 FLX and 3d printed a fan extension for it to keep it the same length as the stock. Fan mod was definitely...

I'm currently using 5 MCX314A-BCCT. Three in a lenovo m720q(i5-8500t) and two in my SAN lenovo p520(Xeon w-2135) in a generic sense they work pretty well, but using special features is a little bit harder. I think I got these for roughly 13 bucks each in March

I might be having some better luck with infiniband on a fresh install.... im going to have to Yea I bought all longer cables haha, im really only comparing to the 3M cables/exact model that previously bought that I know should be able to hit 56Gbps when I was doing my initial research. I also...

I would really like to get it working and I might take a few more days to see if its possible. Just haven't been able to get it working, i've got my vlans and PFC setup but I believe it couldn't find the GUIDs? Not entirely familiar with everything yet. I would consider doing half IB and half...

Interesting, I think I should have been fine in that case? I think I was able to hit 30-40+ during benchmarks for the most part, but who knows how bogged down it could get

Gotcha this is good info... I think im going to do a fresh install of proxmox with just mlxconfig/mstflint and see if it works better without ofed drivers in the way. its a hassle to remove the ofed drivers and get to a clean untainted state(let me know if you have good ideas on how to do this)...

Would you happen to know if other people have had issues enabling SR-IOV and RoCE v2? They seem to work separately but when I enable SR-IOV i'm not able to enable RoCE v2 for whatever reason. Im running Mellanox CX-3 Pro MCX314-BCCT flashed to the latest version. All on proxmox pve9, I was...

have you seen any good 3d prints that address this? i've seen a couple that extend it to 28mm, but was wonderinf if there was a better one out there