Wow I might get one. Can you send me the link to the $75 one lolJust as I bought the 4 x SX6036 for 93 USD / each shipped, I found another one that accepted offers and offered 35 USD / Piece (plus 40 USD Shipping within the US) and got accepted.
So total for this last one was 75 USD shipped. Oh well. Life sucks.
Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)
- Thread starter gb00s
- Start date
-
- Tags
- connect-x3 ebay mellanox
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Nope, it was a single one from that Seller
.Same for the 93 USD / Piece, I emptied their Stock after buying 4 Pieces
.Have any of yall with a sx6036 been able to successfully have a vm guest inside of proxmox properly communicate to other physical hosts on a vlan? I've been having an issue where my vm guest can ping other hosts on the vlan, and the other hosts reply properly, but the reply never even reaches the hypervisor. As far as I can tell something is happening to the vlan traffic somewhere on ingress/egress from the switch. was hoping if anyone had any config examples
Of course. My entire infra is (now) based on Proxmox with a pair of SX6036 as the core switches. There's zero issues with VLANS/VMs/Physical hosts/dual mode ports etc etc.
What exactly are you looking for?
Nice!
I'm hoping to see if anyone has any example configs or just general tips of how they setup their vlans. I'm still not entirely familiar with vlans yet, and it's been hard for me to find the origin of this issue. As far as I can tell all my proxmox hosts with physical nics have been communicating just fine. It's also possible for the guest vm to send and receive traffic to/from the other hosts and they're able to reply. but for some reason... according to the pcaps i've been running not even the hypervisor for the guest sees the icmp reply traffic which probably means that it could be getting eaten somewhere in the switch. I'm trying to understand what best practice looks like on the sx6036 looks like.
Is the Issue only with ICMP ? Might be worth looking into the sysctl and Firewall Rules, if you have Firewall enabled in
Datacenter.It seems to be all traffic, i did a nc -vz 192.168.20.50 8006 which worked from all my physical hosts on that vlan, but this didn't work from my vm. I even tried with all firewalls disabled without much luck. It's super weird, I can see the hosts sending the return traffic but it just disappears soon as it leaves its interface
Last edited:
I have my own Set of Issues with Proxmox Virtual NICs and Mikrotik Switches since the last update completely messed up DHCP and Multicast on Mikrotik RouterOS (different Topic, I didn't receive the SX6036 yet).
I got however some Tips for Proxmox VE from a User on that Forum though (I still need to try it out), so maybe that Part can be useful for you
.I am using
trunks=2-4094 in the Options for the Virtual NIC (you need to edit the Configuration File by Hand, it's not possible via the GUI), although based on what kharkin0 said in this Post, it might not even be required and tag=1 (to use the default VLAN 1 as untagged) is probably messing up Stuff that is already tagged according to him:
What's going on with Mikrotik Routeros Switches DHCP not being passed through?
Not sure if anything changed since PVE v8.2 (I'm using that as an example), because I don't see "trunks" setting. So in this case: net0 would encapsulate everything in VLAN 59, a.k.a. If on VM I'd send VLAN100 on this interface, it would result in me doing a double-tag/vlan-in-vlan/QinQ and...
forum.mikrotik.com
What's going on with Mikrotik Routeros Switches DHCP not being passed through?
Do you mean like net1 from your Screenshot ? Yes Where did you enable VLAN Aware for your Bridge On the bridge. So while tag should only apply to Packages that are NOT tagged yet (and cause them to be tagged), trunks should just let those VLANs be accessible through the Port...
forum.mikrotik.com
Alright so it looks like I finally had some luck after like two days.... i migrated this vm from my pve1 to another host and it looks likes I was able to get it to work on my pve2 and pve3 node. I believe my cx3 pro might be setup slightly differently than my other hosts but im going to have to see how differently it was configured.
I made sure that my vmbr1 bridge wasn't vlan aware and made sure that the nic for the vm did not have a vlan tag and then I configured my actual vlan inside of the vm.
This took waaay waay longer than it should have, im definitely going to have to figure out what is misconfigured with this nic, as the vm is not working with a equivalent setup.
EDIT: Looks like it was configured just about the same as the others, as far as I can tell. As far as I know its pretty likely that i just have a broken proxmox install on this node from when I was trying to get nvme/RoCE working and I was messing with different drivers. It looks like I may be in working order on all nodes now...
I made sure that my vmbr1 bridge wasn't vlan aware and made sure that the nic for the vm did not have a vlan tag and then I configured my actual vlan inside of the vm.
This took waaay waay longer than it should have, im definitely going to have to figure out what is misconfigured with this nic, as the vm is not working with a equivalent setup.
EDIT: Looks like it was configured just about the same as the others, as far as I can tell. As far as I know its pretty likely that i just have a broken proxmox install on this node from when I was trying to get nvme/RoCE working and I was messing with different drivers. It looks like I may be in working order on all nodes now...
Last edited:
I just bought a few more of ConnectX-3 based Adapters, but I hope they are the "right" ones:
No clue if this is MCX354A-FCCT or MCX354A-FCBT, since both are FDR and 56G with a short Bracket (not that the Bracket matters much).
One is Pro and the other is non-Pro.
Mellanox CX354A is such a Mess, as this Post (and copied Screenshot) from @i386 suggests :
EDIT 1: based on this Listing, it would seem that it's the non-PRO Version, i.e. MCX354A-FCBT
In other Words I ordered Garbage.
No clue if this is MCX354A-FCCT or MCX354A-FCBT, since both are FDR and 56G with a short Bracket (not that the Bracket matters much).
One is Pro and the other is non-Pro
.Mellanox CX354A is such a Mess, as this Post (and copied Screenshot) from @i386 suggests
:EDIT 1: based on this Listing, it would seem that it's the non-PRO Version, i.e. MCX354A-FCBT
In other Words I ordered Garbage
.
Last edited:
It probably can be cross-flashed to the FCBT Mellanox Firmware. But yeah, it will stay a non-Pro (regular) ConnectX-3.
Do you think it's best if I ask the Seller to cancel the Order ? Or they might be "OK" even though ROCEv2 is out of the Window (RDMA and ROCEv1 should still work according to what I read) ?
I'd cancel it if possible. ROCEv1 is crap, don't even think about it, even though it'll technically work.
Edit: The cards below are total stock, OEM Connectx3-Pro cards. Right out of the gate. I've bought them in bulk at less than $10/ea.
I asked the Seller to cancel the Order. Let's see if they agree.
The Link you shared is 15 USD and not open to Offer.
I'll try PiosPartsLap in Europe again (last Order was 10 EUR / Piece without Bracket). Frustrating lately because they don't freaking read Emails though !
OK the Seller agreed to cancel the Order. Very Fair on his behalfJust send them a message with your offer.
.@luckylinux - Once you get these up, all wired up through the SX6036, and configured correctly in terms of software, and.... But it's worth it.
The following is a bone stock Debian Trixie install (Iscsi SAN boot!) on two machines, connected through a SX6036, with CX3 pro adapters in them. This is out of the box RDMA testing (after installing a few things), with no tuning, even the MTU is 1500. The top machine (debsmall2) is the client, and the bottom machine (debsmall3) is the server.
Yes, these are my small machines...just a quad core with 16GB RAM, no disks (SAN boot).
Bandwidth test: ~37gbps.
Latency test: ~1usec (That's a micro second)
But it's worth it.The following is a bone stock Debian Trixie install (Iscsi SAN boot!) on two machines, connected through a SX6036, with CX3 pro adapters in them. This is out of the box RDMA testing (after installing a few things), with no tuning, even the MTU is 1500. The top machine (debsmall2) is the client, and the bottom machine (debsmall3) is the server.
Yes, these are my small machines...just a quad core with 16GB RAM, no disks (SAN boot).
Bandwidth test: ~37gbps.
Latency test: ~1usec (That's a micro second)
Nice@luckylinux - Once you get these up, all wired up through the SX6036, and configured correctly in terms of software, and....
The following is a bone stock Debian Trixie install (Iscsi SAN boot!) on two machines, connected through a SX6036, with CX3 pro adapters in them. This is out of the box RDMA testing (after installing a few things), with no tuning, even the MTU is 1500. The top machine (debsmall2) is the client, and the bottom machine (debsmall3) is the server.
Yes, these are my small machines...just a quad core with 16GB RAM, no disks (SAN boot).
Bandwidth test: ~37gbps.
View attachment 45658
Latency test: ~1usec (That's a micro second)
View attachment 45659
. I actually just received the QSFP+ Cables.I ****** up and chose Air Shipping for those, probably wasted quite a bit of Money just like that, instead of waiting for Sea Shipping. You weren't joking ... these are super THICK and HEAVY !About your Test I'm a bit confused.
The Command you ran says
ib_ (Infiniband), but why is then reporting Link type: Ethernet and Data ex. method: Ethernet ? Are you running IPoIB (IP over Infiniband) ?That’s just a test utility. It runs over both IPoIB and ROCE (Ethernet, which is what I’m using).The Command you ran saysib_(Infiniband), but why is then reportingLink type: EthernetandData ex. method: Ethernet
Are you able to do RoCEv2 and still use sr-iov? I haven't fully tested it, but i've been toying with it and for some reason my cx3pro seems to default to RoCEv1 when I enable sr-iov so i shelved the idea. I haven't seen a way to set the default to RoCEv2(that actualy works...). I'm having to use a few workarounds as a result of not having sr-iov but its not the end of the world. I was still able to get my mirrored SAN up and running using drbd and linstor all using nvme/RoCEv2. I went a head and automated it's setup from a fresh install because, I would not wanna set this all up again....
Last edited: