ConnectX-4 LX $25.99-Seller raised to $39.99

[duckduck]

I'm not even looking at ebay outside Europe. Not worth the hassle anymore.

I see ConnectX-4 LX for 80 EUR and XXV710-DA2 for 100 EUR. Both from Germany. Might be a lot more than few years ago, but I only need 3 for now.
But nice to know which cards will have the best chance with ASPM.

 

[luckylinux]

An extra 20 watts would be about 90 cents a month here, not that I'd notice it given my rack is currently at something like 930w. I can't imagine a cpu in a 10gbe machine would have the opportunity to go to sleep very often if ever, if I'm building a server or firewall it tends to.....do things

Lucky you what should I say ...

 

[thigobr]

Last year I upgraded my CX3s dual port SFP+ to XXV710s and they have ASPM enabled by default. No further tweaking needed. They are also surprisingly cooler than the older CX3.

 

[luckylinux]

I believe that not supporting ASPM and not letting the CPU enter into deeper C-states is a feature and not a bug.

In server workload scenarios, predictable, consistent and low latency response is a feature. It's possible that driver themselves set up the card to not go sleep (ASPM) or let the CPU go to sleep (package c states).

Intel X710 (and allegedly Intel XXV710 and Intel E810) do support ASPM.

And, as I said, the Mellanox ConnectX-4 LX seems to support ASPM, when plugged into a PCH/DMI/Chipset Slot. But NOT if plugged into a CPU-connected PCIe Slot.

 

[luckylinux]

Does the Intel NICs work in the CPU connected PCIe lanes as well Or just via the chipset?
I first saw the ASPM issue with the CPU PCIe lanes on W680 boards. But seems to go much further back at least on Intel motherboards. Not sure about AMD.

My Experience on Intel Systems (Supermicro X10SLL-F/X10SLM(+)-F/X11SSL-F/X11SSM-F) with the Intel X710 worked in CPU Connected PCIe Slots as well as PCH/Chipset connected PCIe Slots. Maybe the difference is between PC6 (CPU Connected) and PC7/PC8 (PCH/Chipset Connected). Anyhow, the Difference between those States is very small compared to the Difference between PC2/PC3 and PC6.

Mellanox ConnectX-4 LX on those Systems in CPU Connected PCIe Slot would be stuck at PC2/PC3. In PCH/Chipset connected PCIe Slot it would be PC6.

I don't think I ever tested the Mellanox ConnectX-4 LX on an AMD System. I could now, but on AMD it's also more difficult to Troubleshoot since powertop and turbostat report MUCH LESS Information. Typically on AMD you only have 3 Package States IIRC. And the Software Tools are not always working great ...

So I cannot emphasize this enough: Testing with powertop / turbostat has to be complemented with measurements using a Watt-Meter ! Just like ASPM reporting of PCIe Devices can lie to the System (and BIOS can lie to the System as well), so can powertop and turbostat. I also had a ConnectX-2 reporting ASPM Supported on a Supermicro X9DRi-LN4F+ IIRC (not sure about the Package C-State the System was in, but that was clearly NOT possible).

EDIT 1: the Issue on the W680 System (if that's Intel 11th-13th Generation) if CPU PCIe Connected is probably due to PCIe Multi-VC (Multiple Virtual Channels). I mentioned that on my Repository briefly: GitHub - luckylinux/aspm-troubleshooting: aspm-troubleshooting

But nope, I don't own such System and while disassembling BIOS, the Multi-VC or Virtual Channel Options didn't turn up either.

 

[luckylinux]

Thanks. This will save me a lot of time, having to try things out myself.

I'm still in the planning of what gear to get for a little home proxmox cluster setup. Aiming to keep the power usage of three nodes, close to that of my current server. With the same electricity prices as you (DK). Even if ASPM only brings it down a little overall. If the different NICs cost about the same. Might as well get the kind where it works.

Depends also how much Time you want to invest. It's a BIG Rabbit Hole to go down with.

Now I'm also probably having to do another Rabbit Hole with BIOS Patching and Flashing on an ASRock AM4 Motherboard.

If you want to save some Time and Money and it's only 3 Systems, get the Intel X710 or possibly the Intel XXV710 (I have a few of the latter, but I never managed to test them, but one User on Reddit IIRC confirmed they also support ASPM).

Now that eBay is completely messed up with Shipping to DK (and using services like Jetcarrier does NOT seem to be worth it IMHO), the Intel X710 shot up in Price like a Rocket though . I used to get these for 40-50 EUR / Piece including Shipping and VAT Imports. Now it's more than Twice that if you order from within Europe (since cannot order anything from US since Novemver 2024).

Otherwise get a Solar System. Won't help much in Winter, but at least Spring-Autumn that helps a lot .

 

[luckylinux]

I'm not even looking at ebay outside Europe. Not worth the hassle anymore.

I see ConnectX-4 LX for 80 EUR and XXV710-DA2 for 100 EUR. Both from Germany. Might be a lot more than few years ago, but I only need 3 for now.
But nice to know which cards will have the best chance with ASPM.

And if you want full 2x10gbps or 2x25gbps Speed you would need to use the CPU connected PCIe Slot, thus the Mellanox ConnectX-4 LX in the PCH Slot (if you have a PCH Slot - I don't on some AMD Motherboards !) would have lower Bandwidth.

Personally I can live with 1x10gbps or say 5-8gbps Routing between the 2 Ports/Interfaces (in the PCH Slot), but if you want full Bandwidth AND ASPM working at the same Time, I feel the Intel X710 (and Intel XXV710) would be a better Option. It Depends on what you want basically.

The Mellanox ConnectX-4 LX MCX4121A-ACAT I got for like 20 GBP / 25 EUR from the UK (including Shipping & Import Charges & VAT) so the Difference was noticeable.

But at the Price the 640sfp28 / MCX4121A-ACAT it's listed at in Europe/UK at around 70 EUR/Piece, it's definitively NOT worth it IMHO.

EDIT 1: @duckduck: you might want to try a few Low Ball Offers. Some Sellers straight refuse, some Accept, some give a Counter Offer of 10-20%. YMMV.

The 640sfp28 is a bit cheaper e.g. here Mellanox HP ConnectX4-LX 2-Port 640SFP28 PCIe x8 3.0 25GbE Adapter 840140-001 FP - Piospartslap but I NEVER had the HP Model and attempted to cross-flash personally (as @MountainBofh suggested we should be able to do).