How are you getting on? Have you tried it without riser? What was the problem? I have one 16GB card that works fine at x16. I'm waiting for another 32GB card and another PCIe adapter with three fan outputs. What length of riser did you use?
SXM2 over PCIe
- Thread starter CyklonDX
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Yes. But some people want more than 16GB to run larger models or other things needing more vram
I understand.
I also want more vRAM. I bought multiple 16gb v100's because they are cheaper.
I just want to understand why some people buy 32gb v100's instead of multiple 16gb v100's since the 16gb are much cheaper - or not?
I could understand that decision if someone needs only 32gb because then you would have only one gpu and less problems.
But if you buy more than one, why spending more money?
I can understand spending more if you have not many PCIe-Slots - but if you have, why spending more money than you have to?
Not all tasks or models can easily scale across multiple GPUs. And GPU-GPU bandwidth (with or without nvlink) can be limiting.
There’s arguments for and against it. Just depends what your actual use case is.
There’s arguments for and against it. Just depends what your actual use case is.
Inference scenario:
Choose a single 32GB V100 If you work with large models or datasets that won't fit in 16GB. 3-10x tokens per second compared with 2x 16GB V100.
Choose two 16GB V100s If your models fit in 16GB and your primary constraint is parallelism. 2x request per second compared with 1x 32GB V100.
It's been working fine, I'm able to run fairly sizable models at reasonable speeds. Still on 6 inch 8x riser. I tried several x16 risers none work. Never figured out why. It's in a very old server maybe that has something to do with it. Its a dell r720. I chose that because I already had it, and it has 512gb of ram for cpu offloading for running even bigger models very slowly. I plan on adding a second 32gb v100 at some point
Thanks for the reply, I'll be careful with the raiser. Even without it, my board sometimes goes into x8 mode. About larger models (Inference), as I understand it, it's better to run it on the CPU if there's not enough space for the GPU. Otherwise, the PCIe (even Gen5) will be a bottleneck and be significantly slower. I haven't yet figured out why I need two cards for the Inference scenario.
In my experience speed is still better only doing a partial cpu offload and getting as many layers on the gpu as possible. But again it's very old cpus. I'm primarily running gemma-3-27b-it-qat-q4_0 as it fits entirely in the gpu unless I need to turn the context way up for a large prompt. I've downloaded some other models but, haven't had time to mess with it in a few weeks, I've been trying to get my other ai servers up and running that do text to voice, voice to text and image generation. Just down to the text to voice.
That was the first one I got working, it's been a couple years. Its a project called subsai. It's meant to generate subtitles for videos using whisper. But once I got it working I'm not really using most of its features just using batch files to control whisper. I have not tried to connect it to my frontend yet, I'm waiting until everything is working Overall the speed and accuracy is pretty good even on the ewaste i'm running it on. It's a 13th gen celeron with a k80.
Does anyone know about the -N variants of the V100? Where they come from? Their history? Any other inform about their specs?
I have several normal SXM2 V100s that I’ve been running for a while. All have a default power limit of 300W and configurable from 150-300W. Memory clock of 877MHz.
But my friend buy one on eBay recently and it identifies itself with the device name of “Tesla V100-SXM2-16GB-N” rather than the normal name that I see on mine which is just “Tesla V100-SXM2-16GB”. His -N version was considerably slower on the same workloads and upon further inspection found that it has a default power limit of just 160W and configurable from 150-240W. And has a slower memory clock at 810MHz (roughly 8% reduction).
So does anyone have any more information about this model and how to identify it before you buy it? Or what it was originally used in? Some custom SKU from a hyperscaler or something?
I have several normal SXM2 V100s that I’ve been running for a while. All have a default power limit of 300W and configurable from 150-300W. Memory clock of 877MHz.
But my friend buy one on eBay recently and it identifies itself with the device name of “Tesla V100-SXM2-16GB-N” rather than the normal name that I see on mine which is just “Tesla V100-SXM2-16GB”. His -N version was considerably slower on the same workloads and upon further inspection found that it has a default power limit of just 160W and configurable from 150-240W. And has a slower memory clock at 810MHz (roughly 8% reduction).
So does anyone have any more information about this model and how to identify it before you buy it? Or what it was originally used in? Some custom SKU from a hyperscaler or something?
To repurpose the board you just need the riser adapters to use the proprietary PCIe slots rather than the oculink ports. There are both custom risers (one end male proprietary, other end male PCIe) and PCIe adapters (to use standard risers) pretty freely available on the Chinese marketplaces.
I have a couple of questions regarding these OAM daughterboard/switchboard setups, as I'm finding information on them to be fairly difficult to come by, and I don't know anyone working in DC.
1) Nvlink is proprietary but the SMX4 boards appear to have similar amphenol connectors to AMD Infinity Fabric. Is this effectively a standardized interface such as PCIe which would support any kind of OAM?
2) Are enterprise surplus OAM's vendor-locked in a similar manner to CPU's (because if they are, that would be a predictably horrible waste..)
3) anyone making waterblocks for these things?
The biggest issue I'm running into with LLM's in general is lack of VRAM. That and AMD being too quick to drop older architectures, for instance trying to coax vllm to run on an older GPU etc..
I'm not interested in running nvidia gear.
1) Nvlink is proprietary but the SMX4 boards appear to have similar amphenol connectors to AMD Infinity Fabric. Is this effectively a standardized interface such as PCIe which would support any kind of OAM?
2) Are enterprise surplus OAM's vendor-locked in a similar manner to CPU's (because if they are, that would be a predictably horrible waste..)
3) anyone making waterblocks for these things?
The biggest issue I'm running into with LLM's in general is lack of VRAM. That and AMD being too quick to drop older architectures, for instance trying to coax vllm to run on an older GPU etc..
I'm not interested in running nvidia gear.
Amphenol has an entire line of these connectors, and they all look the same. There's no standard. Every company does their own thing. They all use those Amphenol connectors because they support high speeds.
No idea, but haven't heard any such thing.
Bykski in China. Haven't tried their SXM blocks, but their PCIe GPU blocks are well regarded in the community. Have three on my 3090 Fes and they've been running for about a year without issues.
I suggest you contact them directly and see what they can do
Welcome to the club.
There are some misconceptions here, greatly exaggerated by AMD's general sloppiness in their SDK builds.
I have a lot of Mi50s and been using them with ROCm 7.1 without issues. ROCm builds lack the required tensorfiles, but the matching ROCBLAS builds contain those tensorfiles. So, you all you need to do is copy those files for your gfx and you're off to the races.
Thanks for the info. I've been running MI100-tweaked builds of VLLM mostly from this fellow with some success, fwiw.
Seems he hasn't updated that fork in a 500 LLM years.
There used to be similar forks for Pascal and Mi50 (gfx906). The Pascal one seemed to be a small start-up, while the Mi50 was a single guy. Both stopped maintaining them.
Moral of the story, the project with reliable support for AMD in my experience has been vanilla llama.cpp. While some optimizations continue to be Nvidia only (mainly because they're being contributed by Nvidia engineers), most benefit all backends. Performance is also catching up to vllm, even if it's not there yet.
Hey everyone, I’m looking for help troubleshooting my setup. I recently relocated a previously working AOM-SMX2 build, and now only half of the board is posting. Under normal conditions, the [5,4,*,2] and [9,8,*,6] LEDs were solid yellow lights. After the move, I’m now seeing [5,4,*,2] and [9,8,7,6] solid yellow lights.
Has anyone encountered this specific LED pattern before, or have insight into what these indicators represent on the board? Any guidance on what might have changed during relocation would be appreciated.
Has anyone encountered this specific LED pattern before, or have insight into what these indicators represent on the board? Any guidance on what might have changed during relocation would be appreciated.
Double check your PCIe connections. When I had an issue like this (don’t recall the exact LED configuration) it was sketchy PCIe cables. You have to use high quality cables with this board. Too many adapters degrades the signal. I had to use the kind of custom cable that has the AOM-SXMV connector on one end and normal PCIe male finger on the other end so I had no other adapters to get a reliable connection.