How is everyone setting up Local AI in thier home labs? Idea thread.

[Jelle458]

Thank you very much for telling me. The tvs-h1688x is a beast that I had my eyes on before, but I never really saw any kind of video card compatibility for it. Crazy that thing has a 12VHPWR connector inside it, it says that just no where.

I run QTS on my older boxes, even if I could get both power and mounted the card, I doubt QTS has a driver for it. Hero is what I wish I was running lol.

 

[marcoi]

Thank you very much for telling me. The tvs-h1688x is a beast that I had my eyes on before, but I never really saw any kind of video card compatibility for it. Crazy that thing has a 12VHPWR connector inside it, it says that just no where.

I run QTS on my older boxes, even if I could get both power and mounted the card, I doubt QTS has a driver for it. Hero is what I wish I was running lol.

i wish it had the 12vhpwr and room for a 5090 card lol. but it only regular pcie 8pin connectors. you can look up what cards qnap validated using compatibility listing and selecting gpu.

Compatibility List - QNAP

QNAP designs and delivers high-quality network attached storage (NAS) and professional network video recorder (NVR) solutions to users from home, SOHO to small, medium businesses.

www.qnap.com

That link is for my unit and the highest card tested and approved was 4060.

 

[Jelle458]

Very interesting. I just booted up my TS-1280U-RP and in the App Center I see Nvidia Driver. I want to try and attach my A2 low profile PCIe x8 GPU and see if it wants to play ball.
Man that would be so good if it could be used as a transcoder in Plex, or if I can just pass it on to a VM I suddenly see many possibilities! I would not have looked if not for this thread, thank you so much and sorry for off topic

 

[marcoi]

How is everyone setting up image creation process. Apps/LLM/etc? I been playing with amuse that amd is promoting. but wanted to know how to setup a server so i can call the backend via another app or agent.

 

[seany]

How is everyone setting up image creation process. Apps/LLM/etc? I been playing with amuse that amd is promoting. but wanted to know how to setup a server so i can call the backend via another app or agent.

ComfyUI in k3s, Krita on desktops for editing

 

[marcoi]

ComfyUI in k3s, Krita on desktops for editing

Why are you running comfyui in k3s?
I just downloaded comfyui but its not something to easily jump into. Need some time to figure it out.

 

[seany]

Why are you running comfyui in k3s?
I just downloaded comfyui but its not something to easily jump into. Need some time to figure it out.

Beefy machines stay in the rack. Krita works most places if you have ram. Tailscale means you can hit it from anywhere.

 

[marcoi]

well i decided to try something with my Miniforums N5PRO.
I currently have intel b70 card on the oculink deg1 doc. Working well.
I saw that MF has a DEG2 dock now with TB4/5 and oculink option. so i ordered on while on sale, it is coming today.
I also ordered a split motherboard power cable. The plan is to run both docks on the one N5pro pc and share one 800W psu between the two docs.
My test card will be a nvidia 2080 ti card. So i think both cards should be around 600Watts at max. I also have a 1000w psu coming since they been dropping in prices, so i can upgrade to that if needed.
If it works, I will have N5pro NPU, igpu intel dgpu and nvidia dgpu off one pc.
The plan is to run various llm on each component and have it be used for all my LLM backends for testing and dev work.

Ill post pictures and results once i get it up and running (if it runs lol)

 

[marcoi]

Well i got the dock and did a quick setup.
1. the 1 psu to two docks didnt work out, not enough pcie ports for both cards, so the new dock got the 1kw psu.
2. Deg2 only registered once i plugged in the usbc into 2nd port or port closes to the nic.
3. I also had to manually turn on the dock to get the card to power up.
4. turning on the n5pro didnt turn on the deg2. turning on the deg2 didnt turn on the N5Pro
5. no resizable bar on the card.
6. pcie x4@3.0 speed
7. 2080ti sucks on llm.. not sure if it is due to setup or the card.
8. i dont think a second b70 would work without the resizable bar.

Next test is maybe get a pcie oculink card and try to hook up that way. If that works, i may send the deg2 back and get a deg1 to save on money.

Some pics




pics of hardware: (the mado says this is the way to ai)

 

[unwind-protect]

I bought a NVidia 5090 for local LLMs and I am thinking about getting a 128 GB Strix Halo AMD. The AMD would be for 33-128 GB models and the 5090 for training at that point.

This is important to me and worth the money. I feel like I have to do second college and that wasn't cheap either.

 

[Barbapappa]

I'm in awe for some of the systems here. Mine is mini by comparison, but suits my needs for the moment. I'm using a Beelink SER10 MAX (AMD HX 470 CPU w/ 890M GPU) and 96 GB of memory. Running Ubuntu with LM Studio to play around with various models and keep up with the pace of their development.

 

[marcoi]

here is the deg2 on oculink with a pcie oculink card in the n5pro one pcie slot.


No idea how to get rebar to be enabled on the deg2, it is enabled on deg1. If anyone else has similar crazy setup and got rebar working on 2nd doc let me know how. If i can figure it out, i think i can add 2nd intel b70 card and share the memory for total 64GB.

I going to also play with xcp and passthrough of the gpus to see how well that works when i have time.

 

[tinfoil3d]

can i humbly slide into your discussion with my i5-11500 TMM with 2x32-3200, im running qwen3.6-35B versions purely on CPU and yea, while it is throttling it does work.

 

[J_UnixSurplus]

Great thread! I have been curious about as of right now. I have been using Hostinger and have been wanting to move it local. The only issues is I am not sure if my current hardware can handle running models with 20 - 30 billion parameters with only having 64Gb DDR4 and an 5070 GPU.

 

[J_UnixSurplus]

Well i got the dock and did a quick setup.
1. the 1 psu to two docks didnt work out, not enough pcie ports for both cards, so the new dock got the 1kw psu.
2. Deg2 only registered once i plugged in the usbc into 2nd port or port closes to the nic.
3. I also had to manually turn on the dock to get the card to power up.
4. turning on the n5pro didnt turn on the deg2. turning on the deg2 didnt turn on the N5Pro
5. no resizable bar on the card.
6. pcie x4@3.0 speed
7. 2080ti sucks on llm.. not sure if it is due to setup or the card.
8. i dont think a second b70 would work without the resizable bar.

Next test is maybe get a pcie oculink card and try to hook up that way. If that works, i may send the deg2 back and get a deg1 to save on money.

Some pics
View attachment 48666

View attachment 48669

pics of hardware: (the mado says this is the way to ai)
View attachment 48667
View attachment 48668

How do I build something like this??

 

[marcoi]

How do I build something like this??

do you have a mini pc or are you asking in general how to setup and play with local ai?

 

[J_UnixSurplus]

do you have a mini pc or are you asking in general how to setup and play with local ai?

More how did you setup a system like that with the GPU's and PSU exposed, I love the look of this? I know how to play around with local AI as have ran Ollama locally but considering moving my OpenClaw over locally running some obliterated models

 

[marcoi]

More how did you setup a system like that with the GPU's and PSU exposed, I love the look of this? I know how to play around with local AI as have ran Ollama locally but considering moving my OpenClaw over locally running some obliterated models

I am using Deg1 and Deg2 eternal GPU Dock (eGPU).

MINISFORUM DEG2 OCulink eGPU Dock

Featuring dual OCuLink and compatible with TBT5 devices, the MINISFORUM DEG2 eGPU Dock delivers ultra-fast connectivity. The OCuLink interface (PCIe 4.0×4), coupled with an internal M.2 2280 slot, significantly accelerates AI computing tasks. It offers flexible power supply support, compatible...

store.minisforum.com

Deg1 only has oculink port, which you pc will need, many mini pcs have them or you can get a pcie card
Dev2 does Thunderbolt 4-5 or oculink connections.

Ideally if you have a desktop/workstation you just add a GPU via pcie slot. But if you are limited like on a mini pc or laptop you can add a external gpu dock like deg1/2.

 

[tinfoil3d]

here's 6 months old video i saw recently

More how did you setup a system like that with the GPU's and PSU exposed, I love the look of this?

on eGPU docks and options and impact on various models:

 

[redtorch]

Home lab setup, main running system is 4xDell pro max GB10 where I configured them to run 1x 2x and 4x setups. Here is my earlier setup notes and troubleshooting guides: GitHub - tomsti/guides: Practical AI homelab setup guides for GB10, Mac Studio Ultra, RoCE/RDMA, MikroTik switching, NCCL, and heterogeneous workload experiments. my dashboard that is powered by this tool: GitHub - redtorchinc/node-agent: Compute cluster monitoring anyone got 4 Mac Studio Ultra M3 512GB?