Well my two AMD V620 gpu's came in and had to move the build over to a 4U case.

[Klee]

First version of my AI build, for learning purposes only and i'm not too concerned about maximum performance and I enjoy tinkering with things:

Asrock Rack EP2C621D12WS motherboard.
EVGA RTX 3090 Ti FTW3 video card.
192 gigs of ram
IBM 2 TB nvme drive.
Thermaltake view 600 case in purple, then a Rosewill 4U case.
EVGA 1600 Watt Titanium power supply.
Two Xeon Gold 6148 cpu's
Two 120 mm AIO's cpu coolers, then two air cpu coolers.
Lots of case fans.

I ordered two AMD Radeon Pro V620 32 gb gpu's to play around with and to move my RTX 3090 Ti back into my AM5 setup. Plus two fan shrouds off Ebay and also two Arctic P8 Max 80 mm fans.

The shrouds had interference issues with that setup, since I had so many fans I was using very good quality 1 - 3 adapter cables but the upper shroud interfered with them plus the shroud on the lower gpu just did not have enough room so I was pushing the upper gpu shroud into the lower ram slot and I did not feel comfortable with that plus I could not lower the bottom gpu because of clearance issues with the bottom intake fans, so I moved every thing except all the fans into a unused Rosewill 4U case.

Pic before I was done with it.

Kind of janky but everything clears, and yes I did run a molex power cable to the intake fans plus did some cable management..

 

[marcoi]

looking forward to seeing some bench of the vid cards for ai.

 

[Klee]

looking forward to seeing some bench of the vid cards for ai.

I'm no much on benchmarks now, years ago yep but for the last few years as long as I am pleased with the performance of something I just don't feel the need to do the benchmarking I used to do.

EDIT: After getting the V620's running now i'm curious about the performance of them vs the 3090 so I just might do some bench-marking.

 

[Klee]

Got it running last night.

Had this error with both gpu's installed and when only one was installed but I disabled a couple of things in the bios plus set the pci-e speed from auto to 3.0.

Now both cards are visible in Ubuntu using "lspci -k | grep -i amd -A3" and show up in rocm-smi.

 

[Klee]

Works!!

I installed Ollama and had it pull Qwen3.6:35B and ran that by telling the model to give me a 1500 word essay about early computing, the speed of the generated text was much faster than I could read and I'm a fairly fast reader.

 

[Klee]

Installed Llama.cpp, was a bit of a pain to get working on Ubuntu 26.04 but it works.

Ran llama-cli to test it out with Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf which is 38.5 gigs in size, I wanted to test with a model that was larger than the vram of one of the gpu's.

Gives the speed at the bottom.

watch -n 1 rocm-smi output

Zero optimizations, just was testing to see if it works.

So far i'm pleased, not as fast as the 3090 is on smaller models but still ok speeds.

I'm just trying different things and fully expect things to break but its a big learning experience for me finding out what works and what does not work..

 

[ctrl-alt-del]

So, the diagonal fans and shrouds just about fit into the 3x slot width, correct?

And in principle, the 3rd slot could be used by a small PCIe card (as it is only the shroud/fan that requires the extra width)?

Do your GPU shroud fans run at fixed speed or is there some fan control applied?

What is the length of the V620+shroud-fan combo?

 

[Greg_E]

Aside from additional PCIe lanes, what does the dual processor config get you?

Also do you have those cards connected to separate cpu, or are they both on the same cpu? Most of the server boards I've worked with will split some slots to each cpu.

 

[WANg]

Works!!

I installed Ollama and had it pull Qwen3.6:35B and ran that by telling the model to give me a 1500 word essay about early computing, the speed of the generated text was much faster than I could read and I'm a fairly fast reader.

Hmmm...that's llama.cpp using ROCm and --split-mode? You'll get better prompt rates on ROCm and better token rates on Vulkan. Also, are you using amdgpu_top to monitor activity across your GPUs? It can also tell you if you are being thermal throttled.

 

[WANg]

Installed Llama.cpp, was a bit of a pain to get working on Ubuntu 26.04 but it works.

Ran llama-cli to test it out with Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf which is 38.5 gigs in size, I wanted to test with a model that was larger than the vram of one of the gpu's.

Gives the speed at the bottom.

watch -n 1 rocm-smi output

Zero optimizations, just was testing to see if it works.

So far i'm pleased, not as fast as the 3090 is on smaller models but still ok speeds.

I'm just trying different things and fully expect things to break but its a big learning experience for me finding out what works and what does not work..

Yeah, I went with the lazy route and just got lemonade-ai server running - it basically just loads up base/vanilla llama.cpp and vLLM then obscures the rest.

Your prompt rate might be misreported - for a single V620 off my Minisforum N5 Air and running Qwen3.6-35B-A3B-UDT-MTP-GGUF-Q8_K_XL (similar to yours, 32k context window but everything else on Lemonade defaults), I was benchmarking 415 pp512/47 tp128 on ROCm, and 335 pp512/52 tp128 on Vulkan, and when I asked Qwen on ROCm to summarize MacOS development from System 4 to MacOS 9, I was getting ~330 on prompt processing and ~52 t/s on token walking speed.



That being said, the LLM (36GB downloaded) did shrink down to fit into the 32GB VRAM on the board (it probably offloaded some of the non-actve parameters? Do note that ROCm only works with the device VRAM and not the GTT) - I did expect the throughput on 2 cards be at least an improvement over a single card, so this can be a place to spend some time tuning.

If it's on the Radeon 780M iGPU on the N5 Air, the prompt/token-walk rate will scale down significantly as the hardware theoretical max is roughly 28-35% that of the V620 Pro (the compute/geometry/shader throughput is around that and the dual channel DDR5-5600 SODIMM (~80GB/sec) throughput versus the V620's VRAM bandwidth (~400GB/sec) is within that ballpark.

Edit: I pulled roughly a 205 pp512/27 tg128 on the i780M on Vulkan with the same parameters.

 

[Klee]

My little AI server is chugging along, had to go with more powerful fans for the gpu's, went to Artic 90mm 10,000 rpm fans from Artic 5000 rpm fans.

At full speed they are loud, but with the 5000 rpm fans at 100% if I ran Qwen3.8-27B Q5 running on one gpu it would start to thermal throttle.

Now I have plenty of headroom, with the 10,000 rpm fans it stays under 60c at ~7000 with that model.

Using the front fan headers off the motherboard, I do not run this 24/7, so I decide what model to run and then set the fan speed in the motherboards bios on boot up, I have run it enough to know what speed to run.

Currently its at "level 3" in the bios for the current model and its not too loud.

Will eventually go with a stand alone fan controller, I purchased two cheap ones off Amazon but decided not to use them now since the thermal probes and I would rather use a better solution.

Also I went to Ubuntu 24.04.4 from 26.04 and now have no real issues.

Currently running Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf at ~9 - 22 T/s., not super fast but definitely useable.

 

[luckylinux]

My little AI server is chugging along, had to go with more powerful fans for the gpu's, went to Artic 90mm 10,000 rpm fans from Artic 5000 rpm fans.

At full speed they are loud, but with the 5000 rpm fans at 100% if I ran Qwen3.8-27B Q5 running on one gpu it would start to thermal throttle.

Now I have plenty of headroom, with the 10,000 rpm fans it stays under 60c at ~7000 with that model.

Using the front fan headers off the motherboard, I do not run this 24/7, so I decide what model to run and then set the fan speed in the motherboards bios on boot up, I have run it enough to know what speed to run.

Currently its at "level 3" in the bios for the current model and its not too loud.

Will eventually go with a stand alone fan controller, I purchased two cheap ones off Amazon but decided not to use them now since the thermal probes and I would rather use a better solution.

Also I went to Ubuntu 24.04.4 from 26.04 and now have no real issues.

Currently running Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf at ~9 - 22 T/s., not super fast but definitely useable.

Not sure what llama.cpp Flags (or Build) you are using or your Prompt.

I tried your:

Create a 500 word essay on the MSdos operating system

System: Supermicro X10SRL-F, 256GB DDR4 RAM, SATA SSD

LXC Unprivileged Container (Fedora 44) -> Podman Rootless

llama.cpp Build b10750: build locally (localhost/llama-amd-radeon-pro-v620:debian-b10750-latest)

Containerfile:

ARG LLAMA_VERSION

# Build Environment
FROM docker.MYDOMAIN.TLD/docker.io/library/debian:trixie-slim AS builder

ARG LLAMA_VERSION
ARG DEBIAN_FRONTEND=noninteractive

RUN --mount=type=cache,mode=0777,target=/var/cache/apk,sharing=locked \
    --mount=type=cache,mode=0777,target=/var/lib/apk,sharing=locked \
    apt-get update && apt-get install -y \
    wget gnupg2 ca-certificates git build-essential cmake ninja-build

WORKDIR /build
RUN git clone https://github.com/ggml-org/llama.cpp . && \
    git fetch --tags && \
    git checkout ${LLAMA_VERSION}

# Register AMD's official ROCm repository for Debian 13
RUN mkdir --parents --mode=0755 /etc/apt/keyrings && \
    wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg | gpg --dearmor > /etc/apt/keyrings/amdrocm.gpg && \
    tee /etc/apt/sources.list.d/amdrocm-stable.sources <<EOF
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/debian13/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Install AMD's official SDK developer bundle for your specific GPU architecture (gfx1030)
RUN --mount=type=cache,mode=0777,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,mode=0777,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y amdrocm-core-dev10.0-gfx1030

# Fix for the Builder Stage asset bundling step
ENV HIPCXX=/opt/rocm/bin/hipcc

# Add this line right before running CMake to allow asset compilation binaries to link
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib

# Set the HIP compiler to clang (adjust version if your path differs)
RUN export HIPCXX=/opt/rocm/bin/hipcc && \
    cmake -B build -G Ninja \
    -DGGML_HIP=ON \
    -DGPU_TARGETS="gfx1030" \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_NATIVE=OFF \
    -DBUILD_SHARED_LIBS=OFF \
    -DCMAKE_C_COMPILER=/opt/rocm/bin/hipcc \
    -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \
    && cmake --build build --config Release --target llama-server llama-cli -j 8

# Runtime Environment
FROM docker.MYDOMAIN.TLD/docker.io/library/debian:trixie-slim AS runtime

ARG DEBIAN_FRONTEND=noninteractive

RUN --mount=type=cache,mode=0777,target=/var/cache/apk,sharing=locked \
    --mount=type=cache,mode=0777,target=/var/lib/apk,sharing=locked \
    apt-get update && \
    apt-get install -y wget gnupg2 ca-certificates libgomp1 && \
    mkdir -p /etc/apt/keyrings

# Re-register the AMD mirror to grab the slim production shared objects (.so entries)
RUN mkdir --parents --mode=0755 /etc/apt/keyrings && \
    wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg | gpg --dearmor > /etc/apt/keyrings/amdrocm.gpg && \
    tee /etc/apt/sources.list.d/amdrocm-stable.sources <<EOF
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/debian13/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Install AMD's official basic base runtime layer targeting the V620 architecture
RUN --mount=type=cache,mode=0777,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,mode=0777,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y amdrocm10.0-gfx1030

WORKDIR /app

# Copy the executables and all compiled Library Files (.so) into the same Folder
COPY --from=builder /build/build/bin/* /app/

# Establish environmental execution configurations for RDNA 2 compute handling
ENV HSA_ENABLE_SDMA=0
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib

# Refresh the system's dynamic linker cache so it recognizes the new libraries
RUN ldconfig

# Run Server
EXPOSE 8080
ENTRYPOINT ["/app/llama-server"]
CMD ["--help"]

Quadlet:

[Unit]
Description=llama Server Container

[Service]
Restart=always

[Container]
ContainerName=llama-cpp-server

Pod=llm.pod
StartWithPod=true

# Common Environment Settings
EnvironmentFile=.env

# Server Environment Settings
EnvironmentFile=.env.server

# Other Environment Settings
Environment=HIP_VISIBLE_DEVICES=0,1
# Environment=HSA_ENABLE_SDMA=0
Environment=AMD_VISIBLE_DEVICES=all

# Image
Image=localhost/llama-amd-radeon-pro-v620:debian-b10750-latest
Pull=missing

SecurityLabelDisable=true
NoNewPrivileges=true
Pull=missing

AddDevice=amd.com/gpu=all

User=root

# Forces Podman to pass the host video/render group access through the rootless user namespace
GroupAdd=keep-groups

# Volumes
Volume=/mnt/models:/models:rw,z

# Qwen3.8-Flash-Next-UD-Q4_K_XL
Exec=-m /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
     --host 0.0.0.0 \
     --port 9931 \
     -t 32 \
     -sm layer \
     -ts 1,1 \
     -ngl 32 \
     -c 32768 \
     -fa off \
     -mg 0 \
     --split-mode layer \
     --tensor-split 1,1 \
     --load-mode none \
     -lv 3

# Disable if Part of a Pod
# [Install]
# WantedBy=default.target

Granted my System is not quite as powerful, but that Token Generation Speed that you have seems quite good.

Note that my AMD Radeon Pro V620 have been power-capped to 150W each:

======================================= ROCm System Management Interface =======================================
================================================= Concise Info =================================================
Device  Node  IDs              Temp    Power  Partitions          SCLK  MCLK   Fan  Perf  PwrCap  VRAM%  GPU%
              (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                              
================================================================================================================
0       1     0x73a1,   48617  27.0°C  8.0W   N/A, N/A, 0         0Mhz  96Mhz  0%   auto  150.0W  97%    0%  
1       2     0x73a1,   47340  28.0°C  8.0W   N/A, N/A, 0         0Mhz  96Mhz  0%   auto  150.0W  90%    0%  
================================================================================================================
============================================= End of ROCm SMI Log ==============================================

Lazily adapted using ClaudeAI/ChatGPT Free Tier for Proxmox VE (Ubuntu was already in the Examples list):

GitHub - luckylinux/v620_toolbox: Melting pot of v620/gfx1030 tools and setups

Melting pot of v620/gfx1030 tools and setups. Contribute to luckylinux/v620_toolbox development by creating an account on GitHub.

github.com

EDIT 1: moved Token Performance to a dedicated Section (here):

I get approx.:

• 8 token/s with -t 8 (cpupower set to powersave)
• 4 token/s with -t 32 (cpupower set to powersave)
• 11 token/s with -t 16 (cpupower set to performance)

 

[SnJ9MX]

You should for sure ask a frontier tier AI (Opus 5.5 is exceptional at this) to profile and optimize and leave it running overnight. You could easily get a 50% speed up

 

[luckylinux]

You should for sure ask a frontier tier AI (Opus 5.5 is exceptional at this) to profile and optimize and leave it running overnight. You could easily get a 50% speed up

So I have to pay something I was trying to avoid in order to get the one I want working ?

Not sure if unsloth can do auto-tune of the LLM Engine instead of the Model.

 

[Klee]

For Qwen3.8-Flash-Next and smaller Qwen3.8 models I use this fork of llama.cpp using Vulkan compiled for gfx=1030 in its own directory.

It's maybe two weeks old so it might need to be updated but I haven't bothered to check.

GitHub - LaurentZuijdwijk/llama.cpp: llama.cpp with adaptive speculative decoding (--spec-draft-adaptive) and a Vulkan backend tuned for AMD Strix Halo. 4.7x on structured output, 1.9x mainline prefill on MoE.

EDIT
***************************************************************************************************************************************************
EDIT: I just found out that the LaurentZuijdwijk/llama.cpp build does not enable avx at all, I was checking to see if avx512 was enabled in the build log and realized it nor avx2 were enabled.

Also that version of Llama.cpp is about two months old but I will recompile it anyway just to see if it runs faster.

So I am rebuilding for, avx512 and Vulkan and Rocm.


# Build 1: ROCm + AVX-512
cmake -S . -B build-rocm \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 \
-DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON \
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON

# Build 2: Vulkan + AVX-512
cmake -S . -B build-vulkan \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_VULKAN=ON \
-DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON \
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON


**************************************************************************************************************************************************



For everything else I use standard Llama.cpp and compile it for gfx=1030 with with both ROCm and Vulkan in separate directories inside the main llama.cpp directory. Also about two or three weeks old.

This is the startup command for Qwen3.8-Flash-Next after a lot of experimentation.

" cat > run_server.sh << 'EOF'
#!/bin/bash
numactl --interleave=all ./build/bin/llama-server \
-m /home/******/ai_models/qwen3_Flash/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
--mmproj /home/*******/ai_models/qwen3_Flash/mmproj-BF16.gguf \
--device Vulkan0,Vulkan1 \
--split-mode layer \
--tensor-split 7,3 \
--n-gpu-layers 999 \
--flash-attn on \
--ctx-size 131072 \
--ubatch-size 384 \
--batch-size 768 \
--threads 40 \
--parallel 1 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-mtp \
--spec-draft-model /home/******/ai_models/qwen3_Flash/Qwen3.8-Flash-Next-MTP-Q8_0.gguf \
--spec-draft-adaptive \
--spec-draft-n-min 2 \
--spec-draft-n-max 4 \
--n-cpu-moe 21 \
--fit off \
--load-mode none \
--lazy-mode on \
--reasoning-preserve \
--jinja \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--port 8080
EOF
chmod +x run_server.sh
./run_server.sh "

Then after that if nothing has changed I just run "./run_server.sh"

I did that because if I just copy and paste the commands and flags it does not paste everything and fails.

 

[Klee]

Screenshot of the llama-server output.

Both eval and gen T/s vary a lot more than any of the smaller models, must be since so much of the model is being ran and accessed in system ram.

Right after I started to chat with Hermes.

After it is running a bit.

I did have the percentage of each gpu at 90% but I added the vision model and it bumped up dev0 to 94% but it seems fine.

The gpu's rarely run very hard with this model, temps stay cool even with the fans turned way down and I only see occasional spikes to 250 watts, most of the time both gpu's are a good bit less than 100 watts.

 

[luckylinux]

I just tried letting Claude AI play with it. Google Gemini sucks .

I recompiled against llama.cpp v0.5.0 (upstream) without any special Patches (newer llama.cpp didn't change anything on its own).

Approx. 90% VRAM Usage (same as before) but better control about what gets offloaded.

Power Limited to 150W / Card (same as before).

Exec=-m /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
     --spec-type ngram-simple \
     --spec-draft-n-max 3 \
     --host 0.0.0.0 \
     --port 9931 \
     -ot "per_layer_token_embd=CPU" \
     -ot "blk[.][0-9]*[0246][.]ffn_.*_exps[.]=CPU" \
     -t 12 \
     -ngl 99 \
     -sm layer \
     -ts 1,1 \
     -c 32768 \
     -np 1 \
     -fa on \
     -b 2048 \
     -ub 512 \
     --split-mode layer \
     --tensor-split 1,1 \
     --load-mode none \
     --temp 1.0 \
     --top-p 0.95 \
     --top-k 20 \
     --min-p 0.0 \
     -lv 3

I'm now getting approx. 19-20 token/s . Almost double as before.

 

[Klee]

I just tried letting Claude AI play with it. Google Gemini sucks .

I recompiled against llama.cpp v0.5.0 (upstream) without any special Patches (newer llama.cpp didn't change anything on its own).

Approx. 90% VRAM Usage (same as before) but better control about what gets offloaded.

Power Limited to 150W / Card (same as before).

Exec=-m /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
     --spec-type ngram-simple \
     --spec-draft-n-max 3 \
     --host 0.0.0.0 \
     --port 9931 \
     -ot "per_layer_token_embd=CPU" \
     -ot "blk[.][0-9]*[0246][.]ffn_.*_exps[.]=CPU" \
     -t 12 \
     -ngl 99 \
     -sm layer \
     -ts 1,1 \
     -c 32768 \
     -np 1 \
     -fa on \
     -b 2048 \
     -ub 512 \
     --split-mode layer \
     --tensor-split 1,1 \
     --load-mode none \
     --temp 1.0 \
     --top-p 0.95 \
     --top-k 20 \
     --min-p 0.0 \
     -lv 3

I'm now getting approx. 19-20 token/s . Almost double as before.

I just use Brave browsers Leo AI to do research on, not always right and can sometimes contradict itself. So I verify things by experamenting.

But much better than using Chrome's or other browsers AI chat.

Also you do not load a MTP model?

""-spec-type ngram-simple \"

Per Leo AI
"
--spec-type ngram-simple enables self-speculative decoding via simple n-gram pattern matching — no draft model needed.

How it works:

1. During generation, it looks back through the token history for the last occurrence of the current n-gram (a sequence of N consecutive tokens, default N=12).
2. When a match is found, it takes the next M tokens (default M=48) that followed that earlier occurrence and proposes them as a draft.
3. The target model then verifies all drafted tokens in a single batched forward pass. Accepted tokens are kept; the first mismatch is discarded.

Why it's useful:

• Zero extra VRAM — no second model, just ~16 MB for hash tables.
• Lossless — the target model verifies every token, so output quality is identical to normal generation.
• Shines on repetitive workloads — code refactoring, boilerplate, structured output where patterns repeat. On novel prose, acceptance rates drop near zero and it's a net loss."

I asked this question:
"What would that do to speed vs having a draft model, also what would be better for just general work without alot of reitition?"

Leo's reply:
"
Speed: ngram-simple vs. draft model
ngram-simple is far more limited. Its acceptance rate is highly workload-dependent — it can hit 90%+ on repetitive code, but drops to near zero on novel, varied text. Real-world tests confirm this: one user measured Qwen3-32B going from 20.4 → 20.6 tok/s with ngram-simple on general prompts (essentially no gain), while another got 15–30% on code-heavy work.

A draft model (same family, e.g. a 0.6B or 1B model paired with a 27B–70B target) maintains a 40–60% acceptance rate on general chat and reasoning, giving a consistent 1.3–2× speedup. EAGLE-3 heads push that to 60–80% acceptance and 2–3.6× in benchmarks.

Method	Typical speedup (general text)	VRAM cost
ngram-simple	~0–1.1× (near zero without repetition)	~16 MB
Draft model (same family)	1.3–2×	+1–4 GB
EAGLE-3	2–3.6×	+1–2 GB

For general, non-repetitive work
A draft model is the clear winner. Ngram-simple's entire mechanism depends on finding repeated token sequences in the history — with varied prose, creative writing, or open-ended reasoning, it simply has nothing to match and adds overhead for no gain.

A same-family draft (e.g. Qwen3-0.6B for a Qwen3-30B target) works because it learns the distribution of likely continuations, not just surface-level repetition. If you have the VRAM headroom, that's the right choice for everyday use.

Caveat: On Apple Silicon (unified memory), draft-model speculative decoding has been reported as a net loss (−11% to −24%) because both models compete for the same bandwidth. In that case, ngram-simple or MTP heads (if the model supports them) are safer bets."




So I run this MTP model: Qwen3.8-Flash-Next-MTP-Q8_0.gguf

That was the hardest thing to get working, when Qwen3.8 first came out regular llama.cpp was not working with it and that's why I went with the llama.cpp fork that I did.

But I am constantly trying new things and I document it all in a text file, lots of notes in there.

This afternoon after work I will try your llama-server flags just to see if it works.

I also probably need to try the latest version of vanilla Llama.cpp to see if there is an improvement.

 

[Klee]

Also I run on bare metal since that server is only used for playing with AI, nothing else is on it so if I have to redo everything because I hosed something important it's not a great loss.

Also I feel no need to limit the power for now and if I need to I just crank up the fan speed.

But also this server is only running when I am actually using it, in the future once I decide that I am at the place where I need to run it 24/7 i'll definitely think about power limits.

 

[luckylinux]

Also you do not load a MTP model?

I would need to use the patched version if I want to use MTP, not upstream llama.cpp .

It was barely upstreamed within the unsloth Repository and seemed to have quite a few Bugs .

EDIT 1: for Qwen3.8-27 it actually works, approx 34 token/s compared to 19 token/s without MTP

# Qwen3.8-27B-UD-Q4_K_M.gguf
Exec=-m /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
     -md /models/unsloth/Qwen3.8-27B-GGUF/MTP/mtp-Qwen3.8-27B-Q4_0.gguf \
     --spec-type draft-mtp \
     --spec-draft-n-max 3 \
     --host 0.0.0.0 \
     --port 9931 \
     -ngl 99 \
     -c 131072 \
     --cache-type-k q8_0 \
     --cache-type-v q4_0 \
     -fa on \
     -b 4096 \
     -ub 4096 \
     -t 12 \
     --threads-batch 8 \
     --split-mode layer \
     --jinja \
     --temp 1.0 \
     --top-p 0.95 \
     --top-k 20 \
     --min-p 0.0 \
     --reasoning-effort medium \
     -lv 3

Qwen3.8-Flash-Next won't work with upstream llama.cpp.