Build for 24/7 local LLM for inference with 2 separate GPUs

[DHCPdude]

I want to build a headless compute machine to run a RTX Ada 4000 (20GB) with a RTX Pro 5000 (48GB) or RTX PRO 4500 (32GB) in parallel for inference. The goal is not running one large model using 2x GPUs, but rather running separate models on each GPU.
Why these GPU config? because I already had a RTX Ada 4000 and don't want to sell it for now, but it's not enough to run larger models.


This is going to be 95% time for inference and 5% occasional fine tuning LoRA / QLoRA type. This machine will be only running LLM, the agents and apps will run on other machines and use this machine. The reason for this path instead of using cloud is mainly protecting privacy.

• The goal is to run independent models on each one.
• It will be a headless machine in a 4U case in a rack.
• NAS/storage, Dockers, apps, Proxmox, etc all run somewhere else. So it can contain only enough storage for its own operation, one nvme should be enough.
• I want it to be power efficient as far as possible. Overkill CPU compute sounds unnecessary to me unless good reason for it.

This will be a Linux machine, with vLLM or llama.cpp to run the models.

The build I have in mind is this:

• ASUS Pro WS W880-ACE SE (2x PCIe 5.0 x16 slots at x8/x8 from the CPU, onboard IPMI, works with both ECC and non-ECC RAMs)
• Intel Core Ultra 7 265K
• DDR5 UDIMM 2x32GB - 5600MHz CL36
• Samsung 990 PRO 2TB

Note: I didn't go with EPYC or Xeon because DDR5 ECC RAM (RDIMM or UDIMM) prices are completely out of reach where I live and I will need at least 6+ modules, massively increases total cost of the build.

I would love to hear your opinion and criticism, and ideas for a better build.

 

[T_Minus]

- If you haven't purchased buy latest gen GPU only, don't mix and match ie: RTX 6000 or 5000 or 50xx consumer don't add older gens too.
-- Start with 2x GPU and go from there

- 2TB fast storage in your inference machine is probably plenty, but you'll want network storage to store other models, testing data, etc, etc.
-- I quickly ran out of local fast storage, and rebuilt a home NAS to move models and data quicker.


coming from someone with a 2nd 5090 on the way, and debating what to do with my 3090 TI maybe a 2nd for now then sell both... or just keep 3090 TI in my desktop\gaming rig like it's been

I also would not build a new consumer desktop system for this, either re-use what you have, go used and cheap or buy the EPYC and build it in pieces as you can add it

 

[zachj]

You don’t need to install six dimms of rdimm memory. A server will boot with a single module if you want to

 

[TrevorH]

It may boot but part of the attraction of EPYC/Xeon is the increased number of memory channels and the more of those you can use, the greater the memory parallelism and hence the greater the speed.

 

[zachj]

Of course. But the OP is cross shopping a consumer dual channel machine…if dual channel is sufficient for them in the consumer motherboard context it should be equally sufficient in the enterprise motherboard context.

 

[larrysb]

You know, it might be more budget than you want, but the mission fits the DGX Spark pretty darn well. Mine happily runs 24x7, with little noise or heat, and occupies only a tiny space. You just take it out of the box and start using it after minimal setup.