6-GPU multiplexer from K80s ‚ hot-swap between models in 0.3ms

[dcherrera]

I purchased some old bitcoin mining hardware to see if I could run old nvidia card on them. So I built a system that multiplexes 6 GPU dies through a single PCIe slot using a custom Linux kernel module. Switch between loaded models in under a millisecond.
Hardware:
- BTC-S37 mining motherboard (Picked up 6 on ebay from a total bro getting rid of his old gpu mining setup.)
- 3x NVIDIA K80 cards = 6 dies, 72GB VRAM total
- Total: ~$200 for 72GB of GPU VRAM
Results:
- 38 tok/s decode on RWKV-X 0.2B (INT8)
- 0.3ms average switch time between dies
- 10 rapid swap cycles, zero degradation
- Each die holds its own model persistently
The inference engine is pure C with zero Python dependencies. Still early but the goal is to have all 8 slots filled on the board so models can be loaded and switchable at will on dirt-cheap hardware.
Why? because I'm to broke to afford better hardware and I am capable enough to write the kernel objects needed to get it running. This mother board of the shelf cant even run one of these cards. Super fun project. Now I need to optimize and get a better models running on it.

 

[CyklonDX]

you can try llama.cpp and do a split load for llm

 

[dcherrera]

I have 5 more cards in the mail. but I have my own full c ml framework and inference engine. So I don't need llama.cpp to do that. But I'm actually thinking something different all together. I'm thinking true moe across multiple gpus.