Low power home AI for learning?

[Greg_E]

I tried openSUSE LEAP 16 again with the e9173 card and Gnome (would prefer KDE, but that might have been part of the problem), now it's working. Got updates done, got vulkan installed, got a couple other amdgpu tools installed. Downloaded the localAI and ran it... Says no GPU detected, CPU only.

Installed vulkan tools and checked the following:

greg@artdumb:~/Downloads> MESA_VK_DEVICE_SELECT=list vulkaninfo
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
selectable devices:
  GPU 0: 1002:699f "AMD Embedded Radeon E9170 Series (RADV POLARIS12)" discrete GPU 0000:01:00.0
  GPU 1: 1002:15dd "AMD Ryzen Embedded V1756B with Radeon Vega Gfx (RADV RAVEN)" integrated GPU 0000:04:00.0

So it shows both GPU, but not sure how to get localAI to use vulkan and then to use both cards or at least pick the fastest (probably igpu) to run models on. Have some studying to do. I know nothing about vulkan (yet), so guidance would be welcome.

I did keep the Debian13 version on a different drive, when I get the best result, and if I decide to keep this machine doing what it's doing, I'll switch back to the larger nvme and start from scratch with all the fixes I find along the way.

And before anyone asks, the machine name, due to it's inherent slowness, is Artificial Dumbness (artdumb).

 

[Greg_E]

Looks like I need to run this in docker/podman to get all the features, the binary leaves a few things out, including the ability to run it in vulkan mode.

 

[anakronox]

I started messing around with local AI on a Minisforum HX99G (I think... been a minute) with an 8GB Radeon 6600M. It was okay, but even the models I could load into its VRAM were kinda brain-dead. That's the unfortunate reality of this sphere - it takes quite a lot of VRAM (or RAM/CPU if you have patience or massive DDR5-based servers to play with) to hold the weights for models that are knowledgeable enough unless you have a very narrowly constrained model tailored to a specific task. At least in my experience I wasn't able to put any models to actual work for me until I jumped from that machine (though was perfectly fine for 1080p gaming) up to a DGX Spark and had a ton of headroom for larger models with bigger knowledge sets and tool-calling capability.

If you haven't tested out any agent harnesses like Hermes, Pi, or even Claude Code (if you have a subscription), maybe give one of those a shot hooked up to one of the cloud-based services. Nous Portal will often open up certain big models for a week so you can play around and see what the big dogs are capable of and maybe get some inspiration for things you can have your local stuff do for you.

But hey, the journey and learning in itself is more than half the fun for me and I get the impression it's the same for you!

 

[Greg_E]

I have copliot at work, so any real involved things can be done through it, mostly wanted to see what I might be able to squeeze out of one of these devices. If I can get it mostly functional for real work, then that's a success I can apply up to something with real power. I have some parts chassis that I could use for full size graphics cards, would need main board/cpu and cards to get that going. Need to look at things I'm pulling out of service at work and see what I can make to use for offline work related tasks. We might maybe have a need for still/video images, but guessing I can't make something out of parts that can really handle that task well. I'd need a pair of the RV620 or better for that type of thing, and probably dual CPU to handle the PCIe demands (one x16 slot per CPU on the old stuff I might have).

If I can find an older mainboard that takes DDR3 ecc, I have a pile of that, and CPU for this generation will certainly be cheaper. Only requirement is UEFI boot and I can probably find something that meets this if I look around.

And finally, would need to look at prices of buying used stuff and compare to something like a DGX Spark, building it myself may not be the cheapest way forward.

 

[anakronox]

Gotcha! Nothing's really a good value and likely won't be for a while to come, but as a turnkey device with room to grow by adding more units, the Spark is a nice system. It won't win any benchmarks, but the clustering capabilities let it run some very large models. It's why I'm adding 2 more for 1 standalone for small, utility models and 2 clustered for something like Deepseek v4 Flash or the next mid-sized Qwen MoE.

 

[Greg_E]

In theory I can cluster localAI, and I have a bunch of T740 in my lab, a cluster of 6 with 2.5g networking is about the best I could do right now. Large, hot, and slow! If I get enough performance out of just the Vega 8 iGPU, then I could jump up to 10g networking, which would almost certainly be faster than each host can process data.

That said, I might try clustering two or three hosts just for the experience of doing it. Have to see what the docker image gives me.

 

[anakronox]

Clustering to serve larger models needs high bandwidth, low latency, and RDMA to get the most out of it. If for nothing else to say you did it you could try it on 2.5GbE. I honestly don’t think you’ll see usable performance until you get up to 40Gbps (or Infiniband if you can scrounge that up) with PFC and ECN and even then it will probably feel slow! This stuff really wants higher end enterprise gear but then again there are some true mad lads squeezing Gemma 4 onto iPhones!

If you can get VLLM or llama.cpp compiled to support your hardware, then things get interesting!

 

[WANg]

I tried openSUSE LEAP 16 again with the e9173 card and Gnome (would prefer KDE, but that might have been part of the problem), now it's working. Got updates done, got vulkan installed, got a couple other amdgpu tools installed. Downloaded the localAI and ran it... Says no GPU detected, CPU only.

Installed vulkan tools and checked the following:

greg@artdumb:~/Downloads> MESA_VK_DEVICE_SELECT=list vulkaninfo
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
selectable devices:
  GPU 0: 1002:699f "AMD Embedded Radeon E9170 Series (RADV POLARIS12)" discrete GPU 0000:01:00.0
  GPU 1: 1002:15dd "AMD Ryzen Embedded V1756B with Radeon Vega Gfx (RADV RAVEN)" integrated GPU 0000:04:00.0

So it shows both GPU, but not sure how to get localAI to use vulkan and then to use both cards or at least pick the fastest (probably igpu) to run models on. Have some studying to do. I know nothing about vulkan (yet), so guidance would be welcome.

I did keep the Debian13 version on a different drive, when I get the best result, and if I decide to keep this machine doing what it's doing, I'll switch back to the larger nvme and start from scratch with all the fixes I find along the way.

And before anyone asks, the machine name, due to it's inherent slowness, is Artificial Dumbness (artdumb).

Okay - so, I feel like sometimes, the fastest way to get something working is to...well, get it done the fastest way possible (i.e. Occam's razor).
For me, before you even get down to Docker and deal with the device passthroughs and API inanities, think of the fastest way you can get a local LLM setup going using 4 commands or less. For me, at least, that's lemonade-AI (which runs from Windows Linux or MacOS, or Docker/Snap/whatever), and depending on which distro you are working on, it's either add-repo and go (Ubuntu) or wget the package and run (Fedora and Debian. Of course, the headache isn't getting it to install and fire up, the headache is to try and get everything working correctly.

For example, for me, I have an idle t540, t640 and a t740 here - my t740 was running Proxmox (essentially Debian 13) so getting it to run Lemonade Server should not be an issue...unfortunately the power brick needs to be replaced so that's out for the moment. The t640 with USB-PD power input is currently non-functional (due to my proxmox Vega iGPU offloading experiments) so I am down to the t540...which is currently my frigate box (well, it has a docker image hosting 0.17) running Ubuntu 24.04 LTS, and as the Frigate box it has the Coral TPU running just fine as-is. It also has a single 8GB DDR4-2166 SODIMM and a 128GB NVMe SSD, but since it's an experimental box running something low duty, that's perfectly fine as-is.

Challenge 1: Okay, so Lemonade runs locally. How do you get lemonade to act as a server?

Easy...via the command line:

lemonade config set host=0.0.0.0
service lemond restart

Verify using:

lemonade config | grep host

And then connect to port 13305 of the IP where Lemonade is running - either use the web browser or the nice GUI app available on MacOS or Windows, and point the server URL to the one running.

Challenge 2: How do you verify that Vulkan is actually working on the iGPU?

First, make sure that the packages in Debian/Ubuntu is present:

apt-get -y install libvulkan1 mesa-vulkan-drivers vulkan-tools

Run vulkaninfo and make sure that it's not just dumping everything into llvmpipe (basic vulkan compute/software rendering pipeline), and that it's actually seeing actual GPU hardware.



And then download/install amdgpu_top, which gives you a nice little util to check the GPU ASIC component utilization and approximate power draw -


Note that the Vega 3 Embedded iGPU (gfx902 Raven2) inside the t540 doesn't exactly have the best specs (only 3 CUs or compute units), and its modest performance is further gimped on the t540 by running with passive cooling on a plastic chassis and featuring only a single memory channel (which eats into throughput even more). The prompt token rate (18 t/s) and slow sustained token-walking speed (3 t/sec) isn't great but it's not really indicative of its potential. The Vega 8 embedded on the t740 should at least give you 2-3x the prompt processing rate even if the token walk rate will probably only go from 3 to maybe 6-7 due to the bandwidth limitations. For a better idea, well, the Vega 5 CU (Radeon Graphics 320SP) on my mt46 mobile thin client is between the Vega 3 and 8, and it can perform at about 150 t/s prompt processing and 7-9 t/s tokenwalking on a 12 Billion parameter Gemma-4 model with MTP . The Vega 6 CU (Radeon Graphics 384SP) on the t755 has almost the same theoretical numbers as the Vega 8, and it can do about 90 t/s on prompt processing and token walks at ~11 t/s (keep in mind that this is on a machine that's mostly used as a hypervisor so the prompt processing rates might be slightly lower than expected), and that should probably give you a number to watch out for.
Also note that it's possible for a large chunk of the compute to happen within the CPU despite reporting vulkan support since it's possible for the Vulkan driver to send the compute to a software-only codepath (llvmpipe or something else) instead of hardware (but since it's being sent to the Vulkan driver which can only schedule tasks onto actual GPU compute units instead of the potentially single threaded software codepath) the CPU might not have all cores pegged out - that's why it's important to check the actual GPU for execution metrics (which is what amdgpu_top is good for).

Challenge 3: iGPU VRAM and GTT.

So within AMD iGPUs there are 2 values that you need to care about - namely VRAM (which is the RAM that the memory controller reserved for the iGPU's exclusive use, once allocated it's no longer accessible to the OS...this can be as high as 2GB in the HP x40 series thin clients, if I remember correctly) and GTT, which is the size of RAM pool that the iGPU driver is allowed to borrow from for its tasks...which by default is set by AMD to be 50% of the total RAM pool and can be overridden to use as high as 75% as long as you have at least 16GB of RAM to start (but that might be specific to certain iGPUs). So what does that mean? Well, in my t540 with one stick of 8GB DDR4 SODIMM in a single channel, wth 512MB allocated as "VRAM", we have....

 grep amdgpu /var/log/kern.log | grep GTT

(8096 - 512 )/2 = 3792 MB, give it about 256MB for auxilary buffers and whatnot, and 3659MB for that 50% looks about right. Note that whatever LLM you load into the iGPU via GTT, the original filesize has no bearing on the eventual unfolded size of the LLM in your VRAM. When in doubt shove more RAM in; but do keep in mind the inherent limitations of your hardware.

 

[Greg_E]

That answered a few questions that popped up while co was working on this last night. The bios in the 740 allows auto or up to 2gb for the igpu, but it didn't find much about gtt until your post, if I can push that to 75% it might be interesting, especially if I go up to 64GB system.

I think I read that the e9173 can also use the gtt to get extra ram, with the penalty of going out over the pci bus.

Not feeling ambitious tonight, probably won't work on this until tomorrow night.

 

[WANg]

That answered a few questions that popped up while co was working on this last night. The bios in the 740 allows auto or up to 2gb for the igpu, but it didn't find much about gtt until your post, if I can push that to 75% it might be interesting, especially if I go up to 64GB system.

I think I read that the e9173 can also use the gtt to get extra ram, with the penalty of going out over the pci bus.

Not feeling ambitious tonight, probably won't work on this until tomorrow night.

Well, that 75% RAM allocation limit is dependent on which iGPU your machine is paired with, and it also involve using kernel-parameters-on-boot for the amdgpu module, so it might/might not work correctly in your case. When in doubt, just assume that the 50% rule holds true and then see if you can get the 75% working (I have 128GB on my N5 Air and am more than happy with 64GB of GTT on it, but your mileage may vary).

 

[Greg_E]

I'd probably swap some ram out of another device and bring it to 64gb total before I started messing with kernel parameters. Haven't had the energy to get back on this, and grand daughter coming over while the rest of her family going to the doctors for something, so probably not today either. If I'm lucky I can get back to this over the weekend.

 

[WANg]

In theory I can cluster localAI, and I have a bunch of T740 in my lab, a cluster of 6 with 2.5g networking is about the best I could do right now. Large, hot, and slow! If I get enough performance out of just the Vega 8 iGPU, then I could jump up to 10g networking, which would almost certainly be faster than each host can process data.

That said, I might try clustering two or three hosts just for the experience of doing it. Have to see what the docker image gives me.

In theory yes - but your source of headaches look something like this:

a) You'll need something more than 2.5Gbit for vLLM clustering, so that's probably something for 40Gbit (ConnectX-3 VPI MCX354A cards are relatively cheap). There are some processing overhead working with that MCX354A, and you'll need to source some DAC cables to wire them up to each other. I also doubt that the modern Mellanox flint drivers can do RDMA for fast low CPU utilization transfers.

b) If you load up the t740 with the ConnectX3 and go "full send" on the Vega 8 iGPU, each one of those thin clients chews up at least 60-75 watts.

c) vLLM for multi-GPU clustering is only supported on the AMD side in Strix Point/Strix Halo APUs. No Phoenix, no Hawk Point, no Rembrandt, and nothing with Vega (so no tx40/x55 thin client support).

That being said, if you are using a t740 or 2 as a proxmox cluster, it's a good way to get some use out of their iGPUs since it's sure not going to cooperate and let you offload it to a guest VM...so you might as well load lemonade up and run light LLMs on them.

 

[Greg_E]

You mention Lemonade in many posts, does it have performance benefits or just one of the easiest front ends to get running?

I'm not see an openSUSE compatibility, think the Fedora process would be similar enough?

I had my test machine drawing 90 watts measured at the AC plug, there is nothing economical about this process. It really needs to grow to real enterprise hardware but I'm a long way off for that level of use. Mostly playing right now, trying to see if I can get something working well enough to do real work and explore different models.

 

[WANg]

You mention Lemonade in many posts, does it have performance benefits or just one of the easiest front ends to get running?

I'm not see an openSUSE compatibility, think the Fedora process would be similar enough?

I had my test machine drawing 90 watts measured at the AC plug, there is nothing economical about this process. It really needs to grow to real enterprise hardware but I'm a long way off for that level of use. Mostly playing right now, trying to see if I can get something working well enough to do real work and explore different models.

Why lemonade-ai? Mostly because it’s relatively easy to get it running (5 commands or less), it’s reproducible across multiple platforms (I use it on MacOS, Win11 and Linux across various pieces of hardware), and you can easily tell if the hardware is working or not (amdgpu_top, appletop, task manager, etc). My personal philosophy is to get it working first, then tune it to the needs, and Lemonade usually works. Yeah, it should just install on Fedora assuming that the dependencies are met. Hell, I got it working on Proxmox 9 (really Debian 13) in a few minutes once I added some extra source or 2 onto the packages repo.

As for efficiency? There’s nothing really efficient or "low power" at all about AI - in fact for certain workloads it’s a huge waste of power. This is both in terms of the amount of silicon you need to use in order to do it, and whether the silicon can be optimized for a particular task or not. You are essentially walking weighted branches in a neural network with billions of multi-dimensional parameters, and that is not a problem set meant for normal general purpose CPUs with their limited register set - in fact, it's the kind of things that you want an array programmable compute units to do...like, say, shaders or raytracing units on a GPU. Purpose built NPUs might be available in the long run but it'll depend on whether their software catches up with the hardware. For AMD's XDNA1 and for certain versions of Snapdragon's Hexagon NPU, the devs already ignored it for something newer.

If you ask it to list the number of primes from 1-200, a simple sieve of Eratosthenes algo will run that sucker in fractions of milliseconds - drop qwen3.x or gemma4 onto the job and it’ll probably tie up hundreds of tokens on multiple GPU compute units (requiring hundreds of watts of power), often for minutes while the inferencing process runs through (checks, double checks and etc). For simple fact lookups they often return bad responses (depending on its data crawl and its neural weights), and for inferencing…well, you’ll need to check for conclusions formed out of questionable premises. You’ll need mental guardrails, constant model scrubbing and etc. Even then it can be pathetically easy to poison LLMs.

Depending on the hardware you choose to run your LLMs with, it can be kinda silly, wasteful or downright extravagant.