I have a single V100 32GB being used for local LLM inference. This card appears to have an nvlink connector on the top edge. I am trying to find out if this card would support being connected to another card using nvlink (to be able to have true tensor parallelism). Does anyone know if this is something that can be done or has been done?
V100 w/NVLink
- Thread starter Shmel
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
^^^ That's my recollection as well, all NV100 PCIe 32gb should work with the NVlink bridge.
It's been a number of years now since I last played with those. You will need the bridges and I don't remember which ones those NV100 cards need, and for some reason, I don't think the links for consumer cards work, but I just don't remember.
I used them with an older ASUS Xeon E5 series workstation board, in fact, the same one used in the first-generation DGX Station. Hot, loud, and expensive, but that was the best one could get. I sold my old cards off long ago.
It's been a number of years now since I last played with those. You will need the bridges and I don't remember which ones those NV100 cards need, and for some reason, I don't think the links for consumer cards work, but I just don't remember.
I used them with an older ASUS Xeon E5 series workstation board, in fact, the same one used in the first-generation DGX Station. Hot, loud, and expensive, but that was the best one could get. I sold my old cards off long ago.
Pictures:
It does look like there is a connector on the board. I am hoping to confirm that it will work before I splurge on a second one.
It does look like there is a connector on the board. I am hoping to confirm that it will work before I splurge on a second one.
a same model of gpu (so another v100); you can connect it with nvlink bridge
you would need a bridge like that. (up to 2 links with another v100 -> this doesn't mean you can have 3 cards active nvlink, it means you can have 2 cards with 2 nvlink bridges connecting them for better performance.)
you would need a bridge like that. (up to 2 links with another v100 -> this doesn't mean you can have 3 cards active nvlink, it means you can have 2 cards with 2 nvlink bridges connecting them for better performance.)
I’m pretty sure you’re good to go with that card and NVLink. It’s been a long time, there were some sku of cards in that era that had the edge connectors but they were non-functional as the circuit boards were used for multiple designs. You may get some info doing some internet archeology.
Puget Systems in particular has a very good HPC blog, and their archive goes back almost 15 years, long before those cards were made.
They had a lot of very good posts back when those cards were brand new.
Puget Systems in particular has a very good HPC blog, and their archive goes back almost 15 years, long before those cards were made.
They had a lot of very good posts back when those cards were brand new.
I came across this article: [Benchmark] Dual-GPU AllReduce Only Uses 8% of PCIe Gen3 x16 — It Moves Little, Very Often where someone ran TP across PCIe into system RAM and later got NVLink working and saw no real improvement in token generation. My plan is to get a PLX card (because all my systems are Mini ITX based and it's cheaper than building a whole new system from scratch) and a second (watercooled) V100. Bykski's waterblocks appear to leave the nvlink connector exposed, so it's a $75 gamble. And even then, considering the article where someone used internal llamacpp TP routing (through system RAM), seems like nvlink doesn't really buy much as far as token generation. If/when I get this new set up running, I'll update here with my results.
NVLINK is basically mandatory to use TP and retain good prefill performance. The payload during prefill is huge and you need to be able to move those payloads at the system's prefill rate, which is thousands of tokens per second.
V100 may be old enough and small enough that you can get away with PCIe 3.0 (the largest model you can fit is Qwen-27B which has a small hidden size and high compute requirements) but as the NVLINK bridge is pretty cheap you might as well use it.
I looked over old data, and I can't be certain this card has active NVLink. The same circuit board was used to implement more than a single finished GPU card, so some in this era have the connectors, but are not active.
The same base printed circuit board was also for the Quadro GV100, which specifically did support connecting two cards with NVLink bridges.
If you got lucky, you might find of Quadro GV100 and the NVLink bridges. The Quadro also has built-in blower for cooling, so perhaps making for an easier build.
The DGX Station, the original back then, had four V100 cards in it, and a special NVLink bridge that connected all four cards in a mesh, but I don't think the bridge nor the cards were ever available separately.
Your card has a Dell Part number on it (T39XP). so it was either originally sold in a Dell workstation, or sold by Dell as an accessory.
The same base printed circuit board was also for the Quadro GV100, which specifically did support connecting two cards with NVLink bridges.
If you got lucky, you might find of Quadro GV100 and the NVLink bridges. The Quadro also has built-in blower for cooling, so perhaps making for an easier build.
The DGX Station, the original back then, had four V100 cards in it, and a special NVLink bridge that connected all four cards in a mesh, but I don't think the bridge nor the cards were ever available separately.
Your card has a Dell Part number on it (T39XP). so it was either originally sold in a Dell workstation, or sold by Dell as an accessory.
There is a follow up article where the author go nvlink to actually be utilized and found that prefill performance went up around 14% and had minimal effect on generation. Considering these numbers, I think I will be fine with PCIe (without the need to sell my existing card/waterblock and buying all new cards).
This is the same conclusion I have reached. That the physical edge connector is there, but it isn't connected to anything. My expectation is that `nvidia-smi nvlink -c` would return something, even if there is no nvlink peer.
Weirdly enough, the edge-pins are probably connected, just not enabled. There's probably some weird way to hack the BIOS for the Quadro GV100 onto it and activate it. But, the age and rarity being what it is, there's probably little chance of finding all the images and methods for doing that.
I just don't remember all the permutations when I was working with cards in this era around 2017-2018. When I sold the company, I also sold off all the pro cards and just kept a couple of RTX2080Ti blowers and a pair of RTX Titans for myself. The Titans are 2080's with 24GB of VRAM and definitely do NVLink. I've seen pairs going on eBay for $1200. Been thinking I need to put them all out for sale. The RTX Titans were really pretty good back then for training models and doing development AI work.
The DGX Spark is much easier to deal with, and runs significant models, even as a single.
I just don't remember all the permutations when I was working with cards in this era around 2017-2018. When I sold the company, I also sold off all the pro cards and just kept a couple of RTX2080Ti blowers and a pair of RTX Titans for myself. The Titans are 2080's with 24GB of VRAM and definitely do NVLink. I've seen pairs going on eBay for $1200. Been thinking I need to put them all out for sale. The RTX Titans were really pretty good back then for training models and doing development AI work.
The DGX Spark is much easier to deal with, and runs significant models, even as a single.