PCIe 3.0 Switch/Multiplexer/Expander for Vega Mining

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

exahash

New Member
Sep 22, 2017
14
3
3
54
A lot of people are mining Cryptonight and Cryptonight-lite coins using Radeon RX Vega cards under Windows because of two key stumbling blocks that make it impractical to mine with Vegas under Linux.

1) The first stumbling block is scalability: in order mine with a Vega under Linux, the drivers demand that each Vega card has to be connected to a PCI-e 3.0 (aka gen3) slot which is directly connected to the CPU. Most consumer motherboards only have one or two such slots, the rest are really PCI-e 2.0 and/or they are connected via a chipset that prevents proper gen3 operation. Many Threadripper boards have 4 proper slots, and Epyc boards can have 8+, but those solutions are very expensive compared to the alternative - just use Windows, where the driver works with Vega cards on inexpensive PCI-e 2.0 slots. For miners like me who have for years run numerous Linux rigs with 6+ GPUs the inability to use cheap low-end mobos and cpus with linux is painful.

I've been looking into PCIe "splitters" (aka multiplexers or expanders) and there are many listings on Amazon, Ebay, and AliExpress for PCIe 2.0 compliant cards (e.g. https://www.amazon.com/dp/B073W9KCFC and https://www.amazon.com/dp/B074N2DMLK ) but I've been unable to find a card that can multiplex a single PCIe gen3 slot into two or more gen3 slots. I have the card from the first link and have tested it in gen3 slots to confirm.

All of my research on PCIe gen3 switches has turned up either raw chips or very high-end server interconnects - no retail cards like those I linked to above, which leads me to my main question: is there such a thing and where can I buy it?

2) The second stumbling block is tunability: under Linux it is not possible to tune the card via the driver to the degree that is possible under Windows. Under Windows the gpu and mem clocks and associated voltages can be set directly using the right software and be made to "stick" using the soft pp_table value in the registry, whereas under the current linux drivers the clocks can only be changed using overdrive percentages, voltages can't be changed, and the pp_table is loaded exclusively from the card's vBIOS at boot. With older AMD GPUs the solution was to flash a mod'd vBIOS to the card under DOS/Windows then run the card in Linux, but the security model changed with Vega such that the vBIOS has to be signed by AMD for it to work.

The only workaround I've found is to hack the open source driver (ROCm) to load the soft pp_table value from disk at boot time. I've yet to undertake this step, but I've spent enough time looking at the code to see it's possible. I'm not going to spend the time (probably a day+ with my rusty skills) hacking the driver if I know I can't stick at least 6 Vegas in a rig... which brings me back to the original question:

TL;DR - Is there a PCIe multiplexer card on the market like this: https://www.amazon.com/dp/B073W9KCFC that can turn one gen3 slot into two or more gen3 slots?
 

Aluminum

Active Member
Sep 7, 2012
431
46
28
Mining is about profit, for ideal profit you might end up doing things you normally wouldn't. Vega on linux costs extra hardware that may-or-may-not-exist (or work) and probably much more time to sort this out. Time = money.

Real miners adapt, run windows if vega hardware aligns with your mining goals. I find it better on nvidia for some things too, tuning a bunch of cards with afterburner is super easy, gpu down, mem up, profit!
 

exahash

New Member
Sep 22, 2017
14
3
3
54
Mining is about profit, for ideal profit you might end up doing things you normally wouldn't. Vega on linux costs extra hardware that may-or-may-not-exist (or work) and probably much more time to sort this out. Time = money.

Real miners adapt, run windows if vega hardware aligns with your mining goals. I find it better on nvidia for some things too, tuning a bunch of cards with afterburner is super easy, gpu down, mem up, profit!
Sure, that's why all my Vegas are running under Windows right now.

But let's be clear: I'm not looking for "some way" to get my Vegas mining (the world already has that). I'm looking to make them work under linux. Why? Aside from the simple fact that Windows sucks, maintaining these rigs is taking far more time than my linux rigs. I have a few dozen linux GPU rigs, some with 15+ months of uptime (since the last power outage). I only have a half dozen Windows Vega rigs, and only one has more than 2 weeks uptime. Under linux remote monitoring and software upgrades are easy and can be automated. Not so under windows - gotta rdp into each rig. It doesn't scale. (I know MS has admin utils for large orgs but I'm not in that league)

On the topic of profit - I already have a pretty good sunk cost in existing hardware. If I can add a part to each rig that costs $xxx or less and eliminates at least an hour of maintenance per month then it's a net positive.

And this really goes WAY beyond me. There are hundreds if not thousands of miners out there who would much rather run linux for the same and many more reasons.

Any idea about PCI-e gen3 switches? :cool:
 
  • Like
Reactions: Joel

jims2321

Active Member
Jul 7, 2013
184
44
28
Have you thought about run window virts on top of linux and passing thru the gpus? That would allow you to configure the Vega and clone the virts. I have personally had window virts with uptimes at work in excess of 400 days. That should get you around the limit of linux until some industrial person or persons decide to tackle the limitations of pci-e 2.0 in linux.
 

exahash

New Member
Sep 22, 2017
14
3
3
54
Have you thought about run window virts on top of linux and passing thru the gpus? That would allow you to configure the Vega and clone the virts. I have personally had window virts with uptimes at work in excess of 400 days. That should get you around the limit of linux until some industrial person or persons decide to tackle the limitations of pci-e 2.0 in linux.
I had not thought of that, I'll give it a try. Thanks for the idea!
 

Aluminum

Active Member
Sep 7, 2012
431
46
28
The 3.0 switches you want barely exist on a few motherboards, m2 multicards and the odd controller card or two and are very expensive (thanks fraudcom) add latency, suck more power and put out extra heat. You won't be seeing them standalone on separate cards or risers for plain pcie-pcie because there is no other market use case for them. (nvidia don't need it, radeon shouldn't but...)

The vega thing is 100% software as you stated, so which costs more time and starts working sooner: joining linux kernel and driver teams or dealing with windows admin ecosystem? :)

I hate windows on many levels but it is very stable for mining on a good build with the right drivers and config, as long as you don't let it do anything else on its own.

That card is sketchy af, definitely not using a recent pex switch (5 port 3.0 is $60) either.