How is everyone setting up Local AI in thier home labs? Idea thread.

[marcoi]

Close, but off by a bit.
From A10 (Nihombashi) to A20 (Oshiage) it was spot-on.
From A01 (Nishi-Magome) to A05 (Gotanda) it was correct.
A06 is Takanawadai (Not Shinagawa, which is the Keikyu station south of Sengakuji), A07 is Sengakuji, A08 is Mita, A09 is Daimon (missing).
Eh, yeah, even with a web scraper and a relatively well regarded open weight LLM, it somehow still manages to be incorrect.
So, more mitigations needed?

so i gave it the same question, a link to the image you had and a screen shot of the image since its orignal form was svg. i think it got it right?

All sources now agree, including the official Tokyo Metropolitan Bureau of Transportation site (kotsu.metro.tokyo.jp), which lists A01–A20 with A05 = Gotanda (五反田) — that resolves my one lingering doubt (I'd initially expected 19 stops without Gotanda, but the official site, Wikipedia's route table, and your SVG all confirm 20).

Toei Asakusa Line — all 20 stations, Oshiage → Nishi-magome:

1. Oshiage 押上 (A20)
2. Honjo-azumabashi 本所吾妻橋 (A19)
3. Asakusa 浅草 (A18)
4. Kuramae 蔵前 (A17)
5. Asakusabashi 浅草橋 (A16)
6. Higashi-nihombashi 東日本橋 (A15)
7. Ningyocho 人形町 (A14)
8. Nihombashi 日本橋 (A13)
9. Takaracho 宝町 (A12)
10. Higashi-ginza 東銀座 (A11)
11. Shimbashi 新橋 (A10)
12. Daimon 大門 (A09)
13. Mita 三田 (A08)
14. Sengakuji 泉岳寺 (A07)
15. Takanawadai 高輪台 (A06)
16. Gotanda 五反田 (A05)
17. Togoshi 戸越 (A04)
18. Nakanobu 中延 (A03)
19. Magome 馬込 (A02)
20. Nishi-magome 西馬込 (A01)

How I verified:
- Web search: The official Toei (kotsu.metro.tokyo.jp) station index lists A01 Nishi-magome through A20 Oshiage in exactly this order, and EkiView independently confirms all 20 with matching codes.
- Your SVG: I downloaded it, rendered it at 3× resolution (the text is vector paths, so it needed rasterizing), and read it in zoomed segments. The diagram shows A01–A20 left-to-right with the same names — including A05 Gotanda 五反田 with its JR Yamanote + Tokyu Ikegami transfer badges, Keikyu through-service at Sengakuji, and Keisei through-service at Oshiage. Your screenshot matches my render exactly.

All three sources (official Toei site, Wikipedia, and the diagram) are in full agreement: 20 stations, this order.

 

[Samii]

Great thread — lots of big iron here. Coming at this from the AI dev/automation side rather than the hardware side:

The box that's on 24/7 runs Proxmox, and the AI stack is all Docker on an Ubuntu VM with the GPU passed through:
- Ollama for local models (chat + smaller embedding models for RAG experiments)
- Open WebUI as the front end — the family-acceptance factor of a ChatGPT-like UI vs a terminal can't be overstated
- LiteLLM as an OpenAI-compatible gateway in front of everything, so scripts and apps talk to one endpoint and I can swap between a local model and a cloud API without changing code

Two practical notes from experience:
1. Don't expose Ollama/WebUI ports directly — Tailscale on the VM gives secure remote access with zero port forwarding.
2. Idle power is the real cost driver (echoing what several of you found with the 5090/3090 systems). I cap the GPU with nvidia-smi -pl for chat duty and only uncap for batch jobs — a 3090 at a ~200W cap still does ~100 tok/s on 7B models, plenty for chat.

For anyone starting out: a single 24GB card (3090/A5000) running Ollama + Open WebUI in Docker is the sweet spot. Everything else in this thread is scaling up from there.

 

[unwind-protect]

Still milling away with a single 5090 in an AMD 5950x/128GB system.

I will say that variants of Qwen that fit this are surprisingly capable. I just can't run the context window I want.

 

[luckylinux]

1. Don't expose Ollama/WebUI ports directly — Tailscale on the VM gives secure remote access with zero port forwarding.

You should anyways be putting a Reverse Proxy in front as a bare minimum (TLS).

Though yes, I only allow access to my Homelab via Wireguard VPN. Probably I'll be migrating to NetBird VPN though (self-hosted).

Still milling away with a single 5090 in an AMD 5950x/128GB system.

I will say that variants of Qwen that fit this are surprisingly capable. I just can't run the context window I want.

I'm pretty sure it's a highly capable Card. Way outside my Budget though . That and I'm not a big fan of that 12V Connector on the new NVIDIA GPUs.

 

[Greg_E]

Only downside I've had with net bird (free hosted plan) was uploads. Max speed I can get is about 4mpbs with a good 40mpbs up from my isp. The other side has 2gbps down and in the middle of the night is certainly not the issue. Don't have the ability to self host right now.

 

[unwind-protect]

How do you folks interface with your agents?

I started out with Mattermost, but I ran into too many problems putting a TLS proxy in front. I now use Telegram.

 

[luckylinux]

I started out with Mattermost, but I ran into too many problems putting a TLS proxy in front. I now use Telegram.

What Problems with TLS Proxy ? I just put Caddy in front of it and called it a Day (as most of the Times).

Maybe you need to disable Timeouts and Configure Caddy (or Traefik, NGINX, ...) as a Transparent Proxy (no Buffering), this is what I did e.g. on litellm, llama.cpp Hosts, etc:

# Transparent Proxy (no Buffering)
flush_interval -1

# Allow long Requests (required if Model is not already loaded into GPU VRAM)
# Instruct Caddy to leave the backend socket open for streaming tokens
transport http {
    read_timeout 7200s
    dial_timeout 7200s
}

You will also need to do that on llama.cpp etc Hosts.
And, if you (like me) have oauth2-proxy in front of your llama.cpp Instance, then you need to also configure this:

# Transparent Proxy (no Buffering)
OAUTH2_PROXY_FLUSH_INTERVAL=0s

# Allow long Requests
OAUTH2_PROXY_UPSTREAM_TIMEOUT=7200s
OAUTH2_PROXY_READ_TIMEOUT=7200s

The main Issue I have with it is that Mattermost does NOT allow OIDC SSO in the Open-Source Edition.

Yeah, I know that there are like 2 Open-Source Plugins written in go on GitHub, but given how new they are, I wasn't quite sure to go that Path (potential Security Risk).

Otherwise maybe Zulip could be an Alternative (I still need to check if SAML/OIDC are Options for self-hosted Open-Source Instances) ...

 

[marcoi]

Im using hermes cli pointed to one b70 card. Then two hermes agents that are setup with telegram accounts each pointed other b70 card. One profile for dev the other for assistance

 

[luckylinux]

For those using Hermes (or similarly OpenCode / Oh-My-Pi / Pi Agent / Crush), how do you handle everything besides the initial Setup ?

I setup mine as LXC Container / KVM Virtual Machine, but still wonder how exactly I should start configuring them (besides configuring LiteLLM to be my single AI Model Provider).

Things like Skills, Rules, Hooks, MCP, etc basically.

The Model is just the starting Point. What about the rest of the Harness / Stack ? I'm kinda lost. I saw some People having huge Skills Repositories (which could probably be adapted / reused), but not quite sure.

I feel, as usual, there needs to be a Balance between no Information and too much Information available, and so far I haven't found it .

 

[luckylinux]

Adding one more to my zoo:

• 16x V100 32GB SXM2 (two nodes), 2026: Volta has aged well. This eight-year-old monstrosity runs GLM-5.1 out to the full 202K token context (and should run GLM-5.2 out to 250K+ as well, but I haven't downloaded the weights yet). With many custom kernels (I had to write an SM70 DSA implementation, a fast indexer, and implement fp8 KV cache and fp8 attention/dense/shared experts as well as do a ton of micro-optimization) it gets 1900 tokens / sec prefill at 200K context, 25 tokens / sec decode at 16K context, and most important, nearly flat performance from 2K to 200K in both prefill and decode courtesy of linear attention. It also finally beats the rich kids with their $16K stacks of DGX Sparks (they have to prune the model and turn on MTP to get 25 tokens / sec decode) and absolutely trashes a 512 GB Mac Studio, which gets 10 tokens / sec decode and what I imagine is double-digit prefill at long context courtesy of the shitfest that is llama.cpp. In case you are wondering, fp8 KV/attention should be safe on this model - it is what the official Zai API uses.

What / Where did you buy it ? Some kind of Supermicro / Gigabyte / OEM Server ? eBay / Alibaba ?

 

[marcoi]

For those using Hermes (or similarly OpenCode / Oh-My-Pi / Pi Agent / Crush), how do you handle everything besides the initial Setup ?

I setup mine as LXC Container / KVM Virtual Machine, but still wonder how exactly I should start configuring them (besides configuring LiteLLM to be my single AI Model Provider).

Things like Skills, Rules, Hooks, MCP, etc basically.

The Model is just the starting Point. What about the rest of the Harness / Stack ? I'm kinda lost. I saw some People having huge Skills Repositories (which could probably be adapted / reused), but not quite sure.

I feel, as usual, there needs to be a Balance between no Information and too much Information available, and so far I haven't found it .

well i run my hermes agents in a VM running under XCP-NG hypervisor (my esxi replacement product) that way i can snapshot and backup as sometimes update/agent breaks the OS.
Im using ubuntu 26 with desktop and remote into it via rdc.
i setup hermes with honcho docker running locally. There was a simple script that pulls the git repo and builds honcho.
I also made another VM for the agents to run dockers on, as they would do a huge docker build on the local OS and kill the storage.
But for the most part i just tell the agent to do things.
So i told it to create two other agents and set them up with their own environments and personalities and with telegram.
Honcho helps with memories and hermes creates it own skills over time.

I think the best thing once you get past the setup is just experiment with the agent and let it go do its thing. just backup things and give it clear personality goals/traits such as double check work, do backups before editing etc.. whatever you want/need.

Over time you will learn the in/outs and get it to automate tasks/build its own skills.


Just as an example here the list of skill my agent made, that i didnt need to tell it to do.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                          ┃ Category             ┃ Source   ┃ Trust    ┃ Status  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ compression-testing           │                      │ local    │ local    │ enabled │
│ context-hygiene               │                      │ local    │ local    │ enabled │
│ debugging                     │                      │ local    │ local    │ enabled │
│ dependency-audit              │                      │ local    │ local    │ enabled │
│ dev-workflow                  │                      │ local    │ local    │ enabled │
│ docker-storage                │                      │ local    │ local    │ enabled │
│ email-setup                   │                      │ local    │ local    │ enabled │
│ email-troubleshooting         │                      │ local    │ local    │ enabled │
│ gitea-ci-release              │                      │ local    │ local    │ enabled │
│ hermes-system-check           │                      │ local    │ local    │ enabled │
│ html-to-pdf                   │                      │ local    │ local    │ enabled │
│ llm-models                    │                      │ local    │ local    │ enabled │
│ proton-wireguard-docker       │                      │ local    │ local    │ enabled │
│ qnap-firecrawl                │                      │ local    │ local    │ enabled │
│ research                      │                      │ local    │ local    │ enabled │
│ websphere-admin               │                      │ local    │ local    │ enabled │
│ websphere-remote-backup       │                      │ local    │ local    │ enabled │
│ debugging-patterns            │ debugging            │ local    │ local    │ enabled │
│ hermes-config                 │ devops               │ local    │ local    │ enabled │
│ hermes-profiles               │ devops               │ local    │ local    │ enabled │
│ hermes-update-troubleshooting │ devops               │ local    │ local    │ enabled │
│ honcho-memory-cleanup         │ devops               │ local    │ local    │ enabled │
│ security-audit                │ devops               │ local    │ local    │ enabled │
│ webapp-dev-workflow           │ devops               │ local    │ local    │ enabled │
│ webapp-testing                │ devops               │ local    │ local    │ enabled │
│ email                         │ email                │ local    │ local    │ enabled │
│ himalaya                      │ email                │ builtin  │ builtin  │ enabled │
│ github-issue-to-pr            │ github               │ local    │ local    │ enabled │
│ native-mcp                    │ mcp                  │ local    │ local    │ enabled │
│ collective-wisdom-install     │ productivity         │ local    │ local    │ enabled │
│ polymarket                    │ research             │ local    │ local    │ enabled │
│ rss-feeds                     │ research             │ official │ official │ enabled │
│ openhue                       │ smart-home           │ official │ official │ enabled │
│ reddit-reading                │ social-media         │ official │ official │ enabled │
│ content-write-integrity       │ software-development │ local    │ local    │ enabled │
│ github                        │ software-development │ builtin  │ builtin  │ enabled │
│ writing-plans                 │ software-development │ local    │ local    │ enabled │
│ blocked-page-recovery         │ web                  │ builtin  │ builtin  │ enabled │

 

[louie1961]

My set up is much more sedate. I am running a RTX 2000 Ada 16gb GPU, passed through to my docker VM. I use Ollama for all my AI needs. I am running the following models locally: bge-m3:567m for my Joplin Jarvis plug-in embeddings, nomic-embed-text for embeddings in Project NOMAD,
Whisper for Home Assistant voice recognition, and Frigate detector. Those models stay loaded all the time. I am using gpt-oss:120b-cloud for chat and RAG in both Project NOMAD and Joplin for the "chat with my notes" feature. I have qwen3.5:9b for PaperlessNGX OCR and for Joplin/Project NOMAD as an offline fall back.

I was able to pick up my GPU for a steal ($160) so rather than spend thousands on a newer/bigger/more VRAM set up, I use it to do my embeddings locally and use Ollama cloud or frontier models for everything else. At the $21 per month I am paying, it will take a long time before a multi thousand dollar GPU paid for itself. I am comfortable enough with the privacy posture of this arrangement too.

 

[luckylinux]

well i run my hermes agents in a VM running under XCP-NG hypervisor (my esxi replacement product) that way i can snapshot and backup as sometimes update/agent breaks the OS.
Im using ubuntu 26 with desktop and remote into it via rdc.
i setup hermes with honcho docker running locally.

I currently have Hermes and some other Tools as LXC Containers, but it's probably better to move them to dedicated KVM Virtual Machines soon .

There was a simple script that pulls the git repo and builds honcho.

No official Docker Image ?

I also made another VM for the agents to run dockers on, as they would do a huge docker build on the local OS and kill the storage.

I wonder if there is a way to integrate that into Forgejo Runners, since I already have a bunch of them ready to go to handle the Docker/Container Builds. That would be nice.

But for the most part i just tell the agent to do things.
So i told it to create two other agents and set them up with their own environments and personalities and with telegram.

That sounds like what you could do with Paperclip (at least, in the short Video from NetworkChuck that I saw on Youtube): Master Agent provisioning other Agents.

Honcho helps with memories and hermes creates it own skills over time.

I heard about Hindsight / ZEP / Supermemory but not about Honcho, not sure if they serve the same Purpose.

When working with LiteLLM I'm not quite sure if all Requests are Stateful / Bound to the same Host, so a Centralized Memory Database that all Hosts can access would probably be needed.

I think the best thing once you get past the setup is just experiment with the agent and let it go do its thing. just backup things and give it clear personality goals/traits such as double check work, do backups before editing etc.. whatever you want/need.

Over time you will learn the in/outs and get it to automate tasks/build its own skills.

Ah yes, I think Hermes was kind of auto-learning Skills by default and also auto-deprecating them if e.g. you didn't use them for a few Months or so .

 

[Greg_E]

Sounds like I need to dig into Hermes and learn a few things.

 

[SnJ9MX]

Adding one more to my zoo:

• 16x V100 32GB SXM2 (two nodes), 2026: Volta has aged well. This eight-year-old monstrosity runs GLM-5.1 out to the full 202K token context (and should run GLM-5.2 out to 250K+ as well, but I haven't downloaded the weights yet). With many custom kernels (I had to write an SM70 DSA implementation, a fast indexer, and implement fp8 KV cache and fp8 attention/dense/shared experts as well as do a ton of micro-optimization) it gets 1900 tokens / sec prefill at 200K context, 25 tokens / sec decode at 16K context, and most important, nearly flat performance from 2K to 200K in both prefill and decode courtesy of linear attention. It also finally beats the rich kids with their $16K stacks of DGX Sparks (they have to prune the model and turn on MTP to get 25 tokens / sec decode) and absolutely trashes a 512 GB Mac Studio, which gets 10 tokens / sec decode and what I imagine is double-digit prefill at long context courtesy of the shitfest that is llama.cpp. In case you are wondering, fp8 KV/attention should be safe on this model - it is what the official Zai API uses.

The economics on this one are interesting. It can prefill almost seven million tokens per hour, which is $10 at current API prices, but can only generate 39 cents of tokens per hour servicing a single stream. My guess is serving a single user it does about $2 of work per hour, so at current hardware pricing it would take about 8 months of 24/7 usage to break even.

The model is quite good. GLM5 is the only open source model I would consider for serious work and it is definitely better than Sonnet. It also has a nice clean reasoning trace which makes it good at easier tasks - Opus tends to overthink on straightforward tasks. Zai is a really nice company too - I know their US sales lead and the team seems to be committed to building good models and competitive products.

Should you actually build this? I am not sure. GLM5 is not Opus and never will be, because Opus keeps getting better and now we have Fable. If you just chat, you can get virtually unlimited Sonnet for $20 a month so you'll be dead by the time the cluster pays itself off. If you have a real reason to self-host (legal? medical? but would I trust my life to GLM-5.1-AWQ running on some used computers from 2018?) it kind of makes sense, but you could also just pay someone $100K for an 8x RTX Pro 6000 system and expense it.

I have a single 8x V100 32GB node - works great on qwen3.8-flash-next. How did you connect the 2nd? eyeing this for glm-5.3-flash and the next step up for models. I am also running the 1catAI vllm

 

[unwind-protect]

What Problems with TLS Proxy ? I just put Caddy in front of it and called it a Day (as most of the Times).

I am trying to set it up in Apache, as Apache is already sitting on the public IP doing TLS reverse proxy for other services inside my network. It always failed on MM. MM loads incompletely on initial visit, just a background and some icons. No usable UI.

ProxyPreserveHost On
ProxyPass /mm/ http://localhost:8065/
ProxyPassReverse /mm/ http://localhost:8065/

        <Location /mm/>
                  AuthType Basic
                  AuthName "Restricted Area"
                  AuthUserFile [censored]
                  Require valid-user

#                 RewriteEngine On
#                 RewriteCond %{HTTP:Upgrade} websocket [NC]
#                 RewriteCond %{HTTP:Connection} upgrade [NC]
#                 RewriteRule ^/?(.*) "ws://localhost:8065/$1" [P,L]
#                 RequestHeader set X-Forwarded-Proto "https"

                  RewriteEngine On
                  RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC]
                  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
                  RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
                  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

        </Location>

Tried a whole bunch of stuff from links on the web and out of AI, but the non-loading persists.

 

[luckylinux]

I am trying to set it up in Apache, as Apache is already sitting on the public IP doing TLS reverse proxy for other services inside my network. It always failed on MM. MM loads incompletely on initial visit, just a background and some icons. No usable UI.

I basically don't use Apache anymore, but the Symthoms you describe I observed many Times also with Caddy / Traefik / NGINX with some Application.

This is typically when you put the Website in a Sub-Path instead of the Subdomain's Root.

Try to open Firefox / Chromium Developer Console, go to the Network Tab and refresh.

If you get a bunch of 404 most likely it's a sub-path Problem.

 

[bayleyw]

What / Where did you buy it ? Some kind of Supermicro / Gigabyte / OEM Server ? eBay / Alibaba ?

DGX-1V, first three were off of eBay, other three were from a liquidation auction. I do have two empty ones left if you are interested, DM me...

I have a single 8x V100 32GB node - works great on qwen3.8-flash-next. How did you connect the 2nd? eyeing this for glm-5.3-flash and the next step up for models. I am also running the 1catAI vllm

ConnectX4 100G Infiniband.

 

[unwind-protect]

How much faster is a DGX than a Mac Studio M5 Ultra?

Any numbers around?

 

[luckylinux]

I just bought some 16 NVIDIA V100 32GB PCIe and previously 16 AMD Radeon Pro V620 32GB.

I unfortunately only managed to buy a single Supermicro 4028GR-TRT where I could install 8-10 GPUs. I originally wanted to buy another 1-2 but I don't see Auctions for that anymore .

I only have 40gbps ConnectX-3 Infiniband (with Mellanox SX6036 Switch) and an Option for a ConnectX-4 100g Ethernet Point-to-Point (I have 4 of these NICs) so not quite sure how I could orchestrate that. I was thinking more of doing load-balancing / high-availability via LiteLLM AI Gateway (have some Hosts serve some Models and some other Hosts serve some other Models via llama.cpp or possibly strata once it gets more stable and turn on/off as needed), but of course GLM 5.3 Flash is out of the Question like that.

In the US out of Curiosity I saw this



Now of course if you are a small Company or something like that, that's just 1 Month worth of Salary for an Employee or something like that, but for a Homelabber it's quite an Expense.

Let alone the noise for such a 2U System .