AMD RADEON PRO V620 32GB GDDR6 GPUs - $565

[jbrukardt]

Bit of an odd duck sale, but theres no place better than this community for making enterprise stuff work to our needs

https://www.reddit.com/r/homelabsales/comments/1ks0fuu

V620 GPU specs:

• 32GB of GDDR6 memory with Infinity Cache, 512GB/s memory bandwidth
• Dimensions: Full Height, Double-slot, Length - 10.5" (267 mm)
• These are passive cooling, so to be used in server with sufficient airflow.
• Power requires 2x 8-Pin connectors for 300W TBP per GPU
• AMD Radeon PRO V620 Specs
• There is no Video output
• Please ensure compatibility/fit with your server before purchasing

The main challenges with these appear to be as follows:

1) To utilize the mxGPU SR-IOV functions, non-public drivers are needed. It seems to date, noone on the internet has been able to get these drivers, meaning you can do proper multi-tenant splitting of these GPUs. I think this is likely solvable, but there just hasnt been enough of folks like us to poke at this specific GPU yet

2) No display out, these are cloud-gaming machines. Theres a mini-displayport that may be enable-able if you flash these to WX6800 BIOSes

3) They need a heatsink swap, or fan shroud attachment (3d printed ones are available): Pardon Our Interruption...


Pro's are, 32GB of reasonably fast GDDR6, good FP16 performance, price.

 

[juddle14]

I recognize that Reddit post. Anyone feel free to reach out direct (here or Reddit) if interested in some V620s. Have more than a few available.

 

[jbrukardt]

I recognize that Reddit post. Anyone feel free to reach out direct (here or Reddit) if interested in some V620s. Have more than a few available.

of course you're already here

 

[CyklonDX]

Can you set up a box *linux alma/rocky 9.4 or 9.5 for remote access?
Its a big price if one can't really use it.

according to rocm docs driver that does sr-iov functionality is installed through rocm not through site.
If one can't get that, the card is ewaste as its performance is comparable to that of 6800 but without any updated drivers.

 

[RobstarUSA]

DOes this work well with ROCm and LLM ? I know a few people that would be interested if that is the case. I'm wondeirng how speed is on both inference & training...

Also wondering how these might compare in both power & speed compared to the 70w Arc B50 pro that was announced at computex.

 

[RobstarUSA]

Can you set up a box *linux alma/rocky 9.4 or 9.5 for remote access?
Its a big price if one can't really use it.

according to rocm docs driver that does sr-iov functionality is installed through rocm not through site.
If one can't get that, the card is ewaste as its performance is comparable to that of 6800 but without any updated drivers.

I think this is a great idea. Much easier to sell them if we can test them first.

 

[foureight84]

How does this compare to something like an rtx 3090 for AI usage?

 

[jbrukardt]

How does this compare to something like an rtx 3090 for AI usage?

Inference or training?

In general its slower, but the 32GB of VRAM lets you use bigger models than a 3090 could do

 

[foureight84]

Inference or training?

In general its slower, but the 32GB of VRAM lets you use bigger models than a 3090 could do

Inferencing. Gotcha. I'm running two 3090s atm. 3 of these are equivalent to the cost of 2 3090s and the memory is really tempting but if it's significantly slower tokens/s then it might not be worth it.

 

[autoturk]

Inferencing. Gotcha. I'm running two 3090s atm. 3 of these are equivalent to the cost of 2 3090s and the memory is really tempting but if it's significantly slower tokens/s then it might not be worth it.

here are some LLM benchmarks: https://www.reddit.com/r/ROCm/comments/1iuyioj

 

[CyklonDX]

quote from reddit
"Mistral small 3.1 2503 Q4_K_M 15.15tokens/sec"
this is around perf you would expect from amd 6800 performance.

on 3090 on mistral small 3.1 24B Q6_K model you would be getting around 40tk/sec
7900xtx you would be looking at around 45-60tk/s depending on driver and clocks.
9070 likely has even better numbers if you can get 32G pro version.

 

[iraqigeek]

quote from reddit
"Mistral small 3.1 2503 Q4_K_M 15.15tokens/sec"
this is around perf you would expect from amd 6800 performance.

on 3090 on mistral small 3.1 24B Q6_K model you would be getting around 40tk/sec
7900xtx you would be looking at around 45-60tk/s depending on driver and clocks.
9070 likely has even better numbers if you can get 32G pro version.

To add another data point, just tested with Devstral-Small-2505-UD-Q4_K_XL (Unsloth quant of the Mistral Small finetune, 14.5GB GGUF) and get 50.2tk/s on short prompts on a single 3090 (no power limit). On a longer prompt (1.4k tk), I get 47.6tk/s.

Considering 3090s have come down in price recently (I grabbed another for 550 with a waterblock included), the only advantage the V620 has is the extra 8GB of VRAM.

 

[custom90gt]

The thing that would be cool in my use case would be SR-IOV to split up the GPU to multiple gaming VMs. The driver issue is the thing that keeps me from purchasing though.

 

[juddle14]

The thing that would be cool in my use case would be SR-IOV to split up the GPU to multiple gaming VMs. The driver issue is the thing that keeps me from purchasing though.

Hey, we have 4x of these set up right now in a SuperMicro 1029GQ-TNRT. We haven't had any issues getting drivers. What drivers are you concerned with? What test would you want us to run? I'm hoping to clear up any confusion on them.

 

[jbrukardt]

Hey, we have 4x of these set up right now in a SuperMicro 1029GQ-TNRT. We haven't had any issues getting drivers. What drivers are you concerned with? What test would you want us to run? I'm hoping to clear up any confusion on them.

2 things

A) SR-IOV GPU splitting support. Can you allocate 1 GPU to multiple VMs and have it work well

1. if so, how many VMs (some cards have a limit of 4)
2. What hypervisor

B) ROCm or vulcan support for LLM inferencingB

1. if so, what engine (vllm, ollama, llama.cpp etc)

The above simultaneously. Some of the indicators are that support for each of these things is independent drivers

 

[nk215]

Hey, we have 4x of these set up right now in a SuperMicro 1029GQ-TNRT. We haven't had any issues getting drivers. What drivers are you concerned with? What test would you want us to run? I'm hoping to clear up any confusion on them.

To share a single GPU among multiple VMs, we'll need a host driver and a guest driver. The host driver is impossible to find. For example, a Linux host driver is gim-dkms_1.0.0.1234577_all

It's impossible to get that driver unless you are a cloud provider or something along that line.

 

[juddle14]

To share a single GPU among multiple VMs, we'll need a host driver and a guest driver. The host driver is impossible to find. For example, a Linux host driver is gim-dkms_1.0.0.1234577_all

It's impossible to get that driver unless you are a cloud provider or something along that line.

What host specifically? We tested with Ubuntu and Mint and the host drivers were available without issues.

 

[nk215]

What host specifically? We tested with Ubuntu and Mint and the host drivers were available without issues.

It's the KVM driver I am talking about.

You have a machine running Ubuntu, you can get a driver so that your Ubuntu machine recognizes the GPU. That's great.

What I want is to install KVM (Kernel-based Virtual Machine) to turn the Ubuntu machine into a hypervisor host. That way, I can run a bunch of virtual machines on that host. The driver that I will need to install would allow the host (Ubuntu with KVM installed in this case) to share the GPU with multiple virtual machines. That driver is the file: gim-dkms_1.0.0.1234577_all

The driver that allows your Ubuntu machine to share its GPU with multiple virtual machines is very different from the driver that allows your Ubuntu machine to use the GPU itself. The former driver is what people can't find.

Nvidia is doing the same thing. Its host driver (KVM driver) is locked behind a paywall while its guest drivers are publicly available.

 

[jbrukardt]

It's the KVM driver I am talking about.

You have a machine running Ubuntu, you can get a driver so that your Ubuntu machine recognizes the GPU. That's great.

What I want is to install KVM (Kernel-based Virtual Machine) to turn the Ubuntu machine into a hypervisor host. That way, I can run a bunch of virtual machines on that host. The driver that I will need to install would allow the host (Ubuntu with KVM installed in this case) to share the GPU with multiple virtual machines. That driver is the file: gim-dkms_1.0.0.1234577_all

The driver that allows your Ubuntu machine to share its GPU with multiple virtual machines is very different from the driver that allows your Ubuntu machine to use the GPU itself.

so few people have these, I do wonder if theyve upstreamed it since that last post that says its impossible to find. Proof is in the pudding, if juddle can show the GPU paravirtualized, that would be an indicator

 

[CyklonDX]

I have researched a bit into v620 in past, as well as spoke with amd rep on their forums - supposedly the gim driver was moved from request form to amd to rocm; and rocm should install the correct gim driver through rocm. Thats what I want to try, thats why I requested remote access to one box with this card. If I can try, and its successful those cards aren't e-waste - if it doesn't, those cards are ewaste and prob shouldn't go for more than 250 usd - maybe even less since its headless.