System RAM relevant to VRAM and LLM performance?

[T_Minus]

I've been testing NVIDIA GPU for local LLM just because I had a 3090 TI in my desktop, which also has 32GB System RAM... has anyone run into a situation where when using the system for LLM use only they needed more system RAM or a specific ratio to VRAM?

I'm not talking about Openclaw or another management platform that can eat system ram by doing other things...

context- debating dropping my 5090 system from 64gb ddr5 to 32gb


additional question... Has anyone compared other system components and how they affect LLM performance when utilizing NVIDIA GPU?
IE: Is my AMD 7900 CPU & desktop motherboard ram limitations a potential bottleneck for a 5090, and if so in which scenarios? or is it a 1 t\s kind of thing...


What about specific models ie: MOE vs dense -- is it noticeable to have a 8 channel RAM system vs consumer desktop, and assuming that increases performance > on MOE model vs dense? Noticeable for the faster RAM or not worth the huge expense? RTX 6000 relative to system replacement is very similar in price. Then again a new server-class system can handle >1 RTX6000 which FOR SURE has MUCH greater (and quicker) ROI than multiple 5090 systems for example.


For personal use, not such an issue but as we think of scaling this out to multiple users, and ROI all of these things start to matter, especially considering the power requirements and potential consolidation of larger systems vs many more medium\smaller ones.


If you don't have answers but experiences in comparisons please share

For home use, and niche image\video work it's hard to beat the Mac's power usage, and idle power draw!!

 

[marcoi]

I can maybe help on the desktop side with some testing if needed.
This AI stuff new to me and still learning the bits.
I have LM studio installed on my desktop with 5090 card.
Specs from LM studio hardware tab.


I was trying nanoclaw on a minisforums n5pro system by running it in a Ubuntu VM. First time i was using the N5Pro hardware for LM studio and it was super slow. I then loaded up qwen on the 5090 and pointed the Ubuntu VM to it and the time for commands to run was instantly faster.


When the command was sent to 5090 i would see power spikes from 300w range up to 900w for a few seconds when the card was doing it thing.

I don't know if there some simple benchmark or things to try that could help the conversation?

 

[T_Minus]

I agree, a standard way to test would be great to compare results.

9800X3D | 32GB | 3090 TI | While running windows 11 desktop

llama-cli -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL --ctx-size 16384 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 --chat-template-kwargs '{\"enable_thinking\": false}'

I believe this was from their docs, so there may be optimizations to be had.

Performance: Generation: 107.3 t/s

 

[marcoi]

Any guide on how to get llama-cli running in windows 11?
never-mind i found it here Releases · ggml-org/llama.cpp

Also make sure to download the cuda dll and put them into the same folder as the exe.

also in cmd prompt the original command failed on the last arg, seems like it needs only 1 dash

llama-cli -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL --ctx-size 16384 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 --chat-template-kwargs "{\"enable_thinking\":false}"


terminal
.\llama-cli.exe -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL --ctx-size 16384 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 --chat-template-kwargs '{\"enable_thinking\":"false"}'

 

[marcoi]

running version - llama-b8292-bin-win-cuda-13.1-x64
Windows 11 pro 26200.7840
Nvidia drivers 591.86

When I run your command I get the prompt so i was just asking to benchmark its performance lol. Not sure if there is a way to really benchmark this?

Anyway it returned the following: [ Prompt: 134.6 t/s | Generation: 15.4 t/s ]

doesnt seem to be using the video card when running. Using like 6.8GB ram and little extra power.
So not sure i am doing the test right?

 

[marcoi]

alternative benchmark Releases · mlcommons/mlperf_client

 

[T_Minus]

running version - llama-b8292-bin-win-cuda-13.1-x64
Windows 11 pro 26200.7840
Nvidia drivers 591.86

When I run your command I get the prompt so i was just asking to benchmark its performance lol. Not sure if there is a way to really benchmark this?

Anyway it returned the following: [ Prompt: 134.6 t/s | Generation: 15.4 t/s ]

doesnt seem to be using the video card when running. Using like 6.8GB ram and little extra power.
So not sure i am doing the test right?

that's weird, the model in my command (if you ran the exact quantization) should barely fit in a 3090 TI.

Qwen3.5 - How to Run Locally Guide | Unsloth Documentation

Has different example commands for different purposes, I found it really sensitive to parameters for useful output but also performance.

 

[marcoi]

i missed adding the DLLs so the first time it ran off CPU.
Now with dll i see the card memory loaded upto 26.3GB

I asked AI prompt to tell me a story about an old man with super powers
Results: [ Prompt: 65.4 t/s | Generation: 172.0 t/s ]

I asked it what is server the home, it went into a loop of some kind. I then killed the query and asked if it is broken, it apologize that it had a technical glitch. (lol) and reported the number 1 response was a popular tech website.

[ Prompt: 323.5 t/s | Generation: 177.4 t/s ]

 

[marcoi]

Okay so i have a system with
AMD Ryzen 9 5950X and 128 GB Ram, 4 dimm 3600MT, kingpin 3090 card

video card memory went to 23.3GB used. and llama-cli is using 566MB system RAM
I asked AI prompt to tell me a story about an old man with super powers
Results:[ Prompt: 68.3 t/s | Generation: 118.6 t/s ]

i asked what is serverthehome.com
Results: [ Prompt: 1679.5 t/s | Generation: 118.7 t/s ]

 

[maxvyaznikov]

From my experience, system RAM speed and capacity don't make a huge difference for LLM inference as long as the model fits entirely in VRAM. Where it really starts to matter is when you're running models that spill over into system memory — that's when faster RAM and more channels can help avoid a massive slowdown. I went from 32GB to 64GB DDR5 on my rig mainly because I kept running into issues with larger quantizations eating into system RAM while the OS and other stuff were running.
Honestly for a 5090 with 32GB VRAM, I'd keep the 64GB system RAM. Once you start experimenting with bigger models or running a server with multiple requests, that extra headroom pays off fast. The CPU side matters less in my experience — even a midrange chip keeps up fine since the GPU is doing the heavy lifting during generation.

 

[maxvyaznikov]

From my experience, system RAM speed and capacity don't make a huge difference for LLM inference as long as the model fits entirely in VRAM. Where it really starts to matter is when you're running models that spill over into system memory — that's when faster RAM and more channels can help avoid a massive slowdown. I went from 32GB to 64GB DDR5 on my rig mainly because I kept running into issues with larger quantizations eating into system RAM while the OS and other stuff were running.
Honestly for a 5090 with 32GB VRAM, I'd keep the 64GB system RAM. Once you start experimenting with bigger models or running a server with multiple requests, that extra headroom pays off fast. The CPU side matters less in my experience — even a midrange chip keeps up fine since the GPU is doing the heavy lifting during generation.